Mark thinks he's narrowed down an argument for why good software specifications matter

If your spec isn't good enough, morons have no chance of ever getting things right. The spec ... will fail to resolve anything, and the arguments will smolder for years.

If your spec is good enough, morons have a fighting chance of getting things right the second time around, without being besieged by assholes. Meanwhile, the assholes ... eventually get bored and wander off in search of someone else to harass.

Trouble is, Mark, I think you've hit the nail on the head for why incremental development is the only sane path. You've also neglected the reality of there being Morons and Assholes in charge of writing specifications, and yes, I speak with the authority of having too many times played all four sides of this fence.

In response, I'd like to offer my own advice born of my few decades random walk through the halls of the software craft, experience that has covered military, industrial, academic and business software, games, websites, webservices and gallery artworks.

These, in a single post, are Three Unspeakable Secrets of Software Design.
how to design a tree

The root problem is very simple: All software exists in a dynamic ecology, all projects exist in time and space shared by other software projects and by the world at large. Thus the ground is always changing, the demands, the resources and the constraints ebb and flow like the wind and weather.

The first flaw with the Old School method of the Good Spec is that any dogmatic design by appeal to a specification sends a machine to do a human's job. We don't send autonomous robots into space, we send telepresence rigs, man--machine interfaces that let humans intervene to cope with calamity and opportunity.

In the early 80's I spent a lot of time on the problem of deploying geodesic and tensegrity structures. I concluded that this was ass-backwards, made complex because of and only because of our Evolved Techno Human Obsession with instant gratification. Nature does not 'build' a geodesic viral capsule or the beautiful dome of a mushroom or dandelion. Nature grows them.

In nature, the DNA 'specification' is a guide sure, but it's very largely heuristic, a set of propensities for making choices should circumstance arise. In nature, grand projects begin with a single simple cell, then two, then four, then some inter-communications to coordinate subsequent choices, always with respect to the changing environmental conditions. Aquarium fish grow to fit the tank, cliff-bound trees self-bonzai, brain-impaired humans shift facilities to other neural circuits.

Software design needs to loosen up in this regard, to lift their heads up from the spreadsheets and look out the window and realize that the software we build will exist in a real world, a world with people who use it and learn it and grow with it, a world with people who build it and learn it and grow with it.

What we need is a software engineering paradigm that does as Pete Seeger once recommended for businesses as a lesson to be learned from artists: Plan for improvisation.

It's toward doing that bit of ecology juggling that I offer these three secrets, three starting points, three genetic heuristics toward improving your craft at creating useful software.

1. the art of team design

Specifications aren't about getting it right. Specifications are about appeasing the holders of the purse-strings. End of story. It's about getting a good feeling that the financiers can be assured they have a good team who know what they are doing.

I prefer the Andy Warhol approach: Take that simple idea of what it is you want to do, then just assemble a good team, give them space and autonomy and real responsibility, and then trust them.

The greatest artworks by Andy Warhol were not on canvas, they were on boardroom charts. Andy invented what he called Business Art, creating startups as his media, people as his pigments, factory spaces as his canvases. Andy's great contribution to us all was in demonstrating how it is largely human chemistry that makes magic happen. Put together the right team and anything they do will be golden. Put together a team of morons and assholes, and ...

2. incremental micro-design

The thing is, software always looks good on paper. Unfortunately, two things happen:

1. people don't know what they want until they see what they don't want
2. as soon as you actually start to build the thing, well, reality intercedes

I hate to shock anyone, but there is no Computer Science (except low level, like chip engineering or those essential numerical algorithms texts no one reads) -- at the applications level, at the level of being useful for human consumption, it is The Art of Computer Device Design, and it's an experimental art.

It is an experimental art because, if what you are building already existed, you'd just go to BizDepot and buy it. Because it is an experimental art, it is a learning experience for everyone, for management, the investors, even the hot-shot designers and engineers. If they aren't learning, they're goofing off, eschewing their responsibility as artists.

Thus the only sane approach is to hire software craft artists, let them experiment, and then engineer the process to communicate feedback such that good choices, choices that move in the direction of the stated project goal, will promote motion in that direction. Project Management in software design should shift from pretending to be akin to Civil Engineering role-playing into being more like the Operant Conditioning used to train animals for movie roles. Got that? Good boy. Here's a cookie.

Instead of trying to visualize everything under the sun, software starts with that one basic fundamental kernel of a concept of an idea, code it, make it work, and then ask everyone, "Are we consing yet?" -- you'd be surprised how well this method works, and you'd be doubly surprised to learn how rarely this method is actually endorsed, pushed aside as childish and impractical while flatter heads prevail to pursue the fantasy of a 55 page specification.

And you'd be shocked to learn how very often those flat-heads are subversively ignored while Those Who Know Better secretly coven incremental design behind their backs. There's a lot of developers out there silently and secretly nodding their heads right now.

3. moderation in everything, including moderation

So, ok, I do concede there are situations where specifications are vital and important. Those are the situations where the specifications are possible and those are the exceptional situations, not the rule. For example, when I coded the Channel Manager for the early Mitel ISDN switch, yes, a specification was possible (industry standard protocol to implement) and I could proceed to code even though the actual device did not yet exist (nor was there any ISDN networks to test it on if there was a device).

Like anything else, this is a spectral thing, one extreme to the other, moderation is key. Every communication about a project goal can be seen as a specification; every spec should be looked at with a critical eye that says, "is this too specific?" and rip it out if there's even a hint that you're trying to tell some artist how to do their job. If you want to do off-shore subcontracting, get the artist to send them the specs, not Market Research or the VC CEO.

You might rephrase this bullet point as "Delegate specification refinement downward". A good specification might say, We wish to foster dating among cat lovers but it's up to cognitive psychologists, sociologists and veterinaries for how this might be done, and not one of them should have any say whatsoever about fonts and colours.

Unfortunately, in the name of control and responsibility, the vast majority of specs are a fool's game. Smoke and mirrors, time and money wasted, and lost time.

humanizing software design

Teams, not tasks, tight-feedback loops of an endless stream of small victories, and a federation of design that respects each expert's domain knowledge; it's a tall order, I know.

There's lots of people who won't agree with me, but I'm just telling it as I've seen it go down these past twenty some odd years. Software is a gamble, it's a prospecting game, and you can use geology and engineering to pick a good spot to stake, but the final say in every motherlode find comes from an art closer to dowsing. Regardless what they preach in MBA courses, the path to fostering good software is not servitude to Gantt charts and deliverables, it's an intractable thing, metaphysical and abstract. And very human.

Unfortunately, very few software projects allow themselves this level of humanity. Very few can step out of that 60's era Engineering School sliderule mindset of shipbuilding, accept the mythness of the manhour and just let go of the control freak within. Losing control over something they don't understand only really proves why it is they hired domain experts instead of doing it themselves, but still, no one wants to let go, to just be Andy Warhol, confident in their own ability to craft a team, perfectly content to just let it rip and see what happens.

So they negate the humanity of the process, repress the existance of teams or turn it into a football kick-butt metaphor best left for motivational psychologists; they focus instead on a Vision of the Unbuilt Thing and design by spec, and the developers, who are very often people who oddly enough actually use this technology they see the spec and laugh over the water cooler, but they code it anyway. Might as well be a moron if the asshole spec is ludicrously naive ...

one final example

So, now, bottom line: Can it work? Can we create good software without specifications, without managerial reigns of powerpoint terror and chains of Project Timelines?

All the time.

Consider the obvious examples, the free-software hits like Apache or Linux, the sort of free-software coded by large numbers of unbridled people. No spec, just a mission and a self-selecting team of talent. They proceed not to create the massive killer Hurdle-Jumper, they proceed simply with What needs doing now (a very sane sentiment for anyone) and they pause, review, revisit, rethink, redo, lather rinse and repeat.

And they'll beat the morons every time. All four of them.