Simplicity is not so Simple

So o’er this weekend I got frustrated with the cron-o-meter development and ripped it apart and threw away the SQL database backend. It really appears, on first blush, to be they type of application born to be a classical database app. However, I couldn’t shake the feeling that using a database wasn’t doing anything other than ‘feeling like the proper way to do it’ and beyond warm software engineering fuzzies, was getting in the way more than anything. So I went back to custom file I/O.

The USDA food database in our old SQL DB took up over 30mb, and took several seconds to initialize and load. I replaced it with with 6500 XML files in a ZIP file, plus a simple index file for searching. This custom ‘database’ loads the index in a split-second, and weighs in at only 4.5mb. And it’s all implemented in about two pages of very easy to read Java code. It took weeks to design and implement the SQL database system and write all the support code to integrate it into the program. It took about 2 hours to write the simple custom solution.

In my Comp Sci. degree, the discussions of Software Engineering is always to design things as abstract and generalized as possible. It is stressed over and over again. We never covered the less obvious evils of over-abstracting and over-generalizing. You end up with a lot of complexity in a general system that handles problems you don’t need a solution for. You often end up having to conform your problems into the proper shape to fit the general solver, like round pegs in square holes. For instance, using a relational database metaphor in the program forced a lot of database specific issues and design considerations. With the SQL database gone, things can be written more naturally.

A general solution can’t make domain specific assumptions or optimizations. A specific solution will always beat the pants of off the general one. Darse has a lot of good anecdotes about this in AI. Perhaps some day he’ll regail us with tales of tic-tac-toe, or Sokoban….

As a lazy coder, it’s always so tempting to write generalized solutions to problems. If you can write something once and then you might be able to use it over and over again for many different problems. However, we get into that habit such that we try and generalize everything, often without thinking if it’s truly needed. If writing 3 specific solutions takes 2 hours each, then why spend 10 hours writing the general solution if in all likelyhood, you’ll only ever need to solve 3 or 4 instances of the problem? Well for one, it’s usually a lot more satisfying to write the general solution. It takes creativity writing highly specific solvers can be very creative too, but the redundancies of writing the same type of things over and over can become dull.

I really enjoyed trying to engineer the crap out of cronometer (and much credit is due to Chris, who is far more adept at proper “Software Engineering” than I). But this weekend I reached a turning point where I just really wanted to finish and use the tool, not just tinker with cool libraries and architectures. Now, it may be me that has the problem in feeling hindered by these constraints. But at the end of the day, the productivity of the coder is really all that matters. It doesn’t matter how ‘proper’ something is if it doesn’t actually boost productivity in practice.

I guess I’m starting to more clearly see this issue now in all coding I do, especially at work. Every day I’m making tough decisions on how to approach a problem, and often the question is to take the aesthetic route or the pragmatic route. Today, while looking over our Poker Server code, I really wanted to refactor it, wrap it in pink ribbons and make it pretty. But after discussing it with Scott, I realized that doing so was unlikely to actually help anything, other than make it look more textbook, and would take more development time to produce than it would ever save down the road.

The big realization is that Generality does not equate to Simplicity. Specific solutions can be far simpler to create, understand, maintain, and debug. If something is easier to work with and understand, then it has lowered the complexity of the project. The Simpler something is, the less chances there are that something can go wrong, or that it will be misused. Achieving simplicity in a complicated domain is far harder that achieving complexity. It is a balancing act of making intelligent choices between general and specific solutions on a case-by-base basis.

Amen.