Those working at the highest levels of abstraction inherit the most change from below.

Your application framework may be like standing on the shoulders of a giant, but that giant is a beneficiary of an entire civilization, which is nested on the ground of the Earth, which itself is locked in orbit around the Sun, which itself is orbiting around the galaxy, which itself is still accelerating through the Universe after being ejected from an epochal Big Bang, which itself was probably the result of a collision in an even higher abstraction on our path to infinite regression.

I’m not saying that you’re insignificant — quite the opposite, I’m saying you’re fortunate; as much to be alive as to be a developer with all these great tools to build on.  A few years ago it would take an army of tie-sporting programmers chanting corporate songs in lockstep to produce an end result that a single coder can do from their bedroom today.

But as commercial developers we also have a problem.  It’s so addictive to work in these high levels of abstraction to rapidly change the world for our users that we often don’t see the ground moving beneath our feet.  The reality is our platforms are in motion too, and it’s a fallacy to imagine that our currently cushy abstraction will forever insulate us from having to go back to the drawing board.  But redesign we must.

So why change?  As the platforms we’ve chosen to build on evolve, more and more of the arcane glue between our ideas and tangibility is being done for us by the platform.

Here are a few examples:

• In Firefox, the browser-based spell checker adds spell checking to every text box in a web-based application.  That removes a huge burden from web developers who should never have had to deal with server-side spell checking in the first place.  There are clear benefits for the user too: they get consistent spell checking support that they don’t have to re-train for each application.  Their last name added to the dictionary on one site is reflected on every site.
• Again in Firefox, browser-based extensions can take over even more of the heavy and redundant functionality that shouldn’t be wasting developer time at the application level.  My favorite recent example is the WriteArea extension which adds an inline HTML editor to any <textarea>.  This cuts out a large chunk of the source code footprint (FCKeditor, and ironically TinyMCE, are both hefty); it reduces the amount of client-side script you’re requiring people to cache; and it also reduces complexity, especially on Ajax-enabled sites that may be popping up asynchronous edit windows on floating DIVs.
• Memcached is a distributed shared memory cache that can take over your home-brew caching layer.
• Syndicating application content as RSS is a great way to offload notifications to tools that are better suited for it, without overloading your users’ e-mail inboxes.
• PHP5 introduced SimpleXML; which while far from perfect, I’d take any day over the event-driven nonsense we were doing in PHP4.
• PHP6 will introduce native Unicode support (I’ll wait while the Java crowd stops laughing).  Right now we’re having to hack support together with the ‘mbstring’ extension and deal with side-effects in binary strings from libraries out of our control.
• I’ve been pleasantly surprised by my forays in to OSX and iPhone development land.  While XCode itself is no Eclipse, the Interface Builder puts the UI tools I’ve used in Java (Swing or RCP) to shame.  This abstracts the relationships between the code and the interface.
• Amazon’s elastic computing (EC2) and redundant storage (S3) services are game changers. Designing your application to take advantage of elastic computing and “unlimited” storage can remove your hardware-imposed scaling limits — at least until you’re successful enough the costs of doing it yourself are less imposing.

If you haven’t gone back to the drawing board in the past 12 to 24 months, chances are you’re wasting a lot of time on things that are already solved.

Sure, that sounds great in theory, but my community will turn into a com-mutiny!

If you have a sufficiently large user community around a project, you likely have your own subgroup that gets loud every time significant change is hinted at.  “We don’t want to have to retrain our entire team to do things a different way,” they say. “Small improvements are okay, but don’t go turning us upside down.”

It’s never a great feeling when you feel your project needs to diverge from the comfort zone of some of your most loyal users — most of whom helped you grow from nothing — but it’s more important that your project still exists with at least as much contagious enthusiasm in a year from now.  Hopefully more.

So where’s the “paradox”… Or was that a sensational headline?

I’ll make this real simple.  Let’s say you’re building applications on the LAMP stack.  PHP and MySQL are going to evolve with or without you.  Someday in the fairly soon future, Linux distributions will be standardizing on PHP6 and MySQL 6.  If your application doesn’t work in that environment, your users are going to have to maintain time capsules where it does work.  If their servers break, they’re going to have to spend extra time reverting a new server to old versions of everything.  That limits what they can do with their hardware.  They’re going to miss out on the other less-directly-tangible benefits of the platform: stability, security, maintainability.

Even if your application simply works with new versions of your platform, you’re at the risk of your competition offloading grunt work to their platform and spending their time making their product and packaging better.

The paradox is that you’re going to have to change anyway, and it’s going to cost you far more energy to not change and maintain a time capsule than it’ll take you to embrace change and draw energy from your platform.

Rewriting a successful application can be daunting, but it’s also inspiring to know you’re working on something that people definitely will use because you’ve already proven the idea.  This time you’re informed by all the things that didn’t work out like planned, and you have the ability to turn the roughest spots into the improvements you’re the most proud of.

This is the “why”, but I need to sit down and write out my thoughts on the “how”.  The real dividends from a software rewrite come from sweeping away the scaffolding cruft of a first attempt and building from a stronger,cleaner, more maintainable foundation on the ideas that survived.  It’s also more important to design for changeability than around a concrete feature set.  Your first re-write shouldn’t sow the seeds of your second in another year or two.  I’ll definitely post a follow-up.