Saturday, November 17, 2012

Cultured Software, Part II

OK, I'm done quietly crying bitter tears of joy as PHP intrudes upon my thoughts, so let me continue with less rant but just as much conjecture.

As a profession, software struggles to remain relevant even as the technology becomes intractably entwined with our culture. Software ecosystems can build up code detritus so rapidly that it entraps their bat-like programmers as if it were some great pile of guano sealing off the mouth of their cave. There continue to be gaps between understanding, resourcefulness in construction, discipline in deployment, and proper accountability in maintenance. Some subcultures appear to be actively trying to shrink those gaps, yet they persist and increase both in quantity and scale. Like any infrastructure - roads, fuel and water pipelines, bridges, etc - software exists in an environment and it degrades over time unless sufficient back-pressure (work) is exerted to keep it together.

There is a kind of programmer with an unhealthy avoidance, an unwillingness to reconsider coding practices and account for sufficient code churn, that is, refactoring. They don't value, count the costs of, communicate, or ethically set expectations for the effort required to make software. Such programmers may look at software as a one-shot effort, rather than an on-going concern. We have probably all seen the ill-conceived notion of job security that holds work together with spit and baling wire - through regular but unpredictable manual intervention.

Now, I wrote "unhealthy," when the reality may be that such a programmer may be completely on top of their game, making money hand over fist, and having a large gaggle of more than slightly confused impressionable clients. "Unhealthy" is more a reference to the long term consequence to the profession of the uncounted costs, and to the client of the spit and baling wire that lie behind what might be a beautiful artifice. Nor is it necessarily a "fault" of the programmer: he is reacting organically to the culture that allows him (or her) to survive and thrive.

Refactoring is a software analog of composting: it breaks down code to make the primitive knowledge entrained in the source available to feed regrowth. In composting, some substances require special handling to reduce and lots of stuff just doesn't break down at all.  Unless conditions are insanely stable, there is a finite amount of time to use compost before it weathers and living creatures scavenge its useful energy. Reworking a software system, for instance, often surfaces hidden assumptions, wrong-headed wild guesses, and punts; the mind of the programmer reveals structure in part by ripping the system apart slowly and digesting the implications.  

A few substances that make their way into compost piles can poison and turn the pile putrid (alive, but fetid), even if in small amounts they are relatively inert and innocuous. Software has such seeds of disruption too. One class of putrefying software artifact I like to think of as a fixed point, to abuse a term from algebraic theory.

Cultured Software, Part I

With few exceptions, most software is crap. You know it. That makes sense: software originated as the study of life processes, and what is life but recycled crap? 

Human languages produce a lot of crap too. Clearly this is a side-effect of being a living process, but it isn't an accident. Side-effects are central to life processes.  Every generation of organism, every shift in language, produces detritus. In biology it takes the form of excrement and body mass; in language it is seen in archaic forms, marginal slang, and jargon. If a process is alive, it generates layers of crud.

Crud builds up over time unless broken down. In biological systems the breakdown products become the source from which new growth feeds. In language, memes beget metaphors and metaphors mixed, merged and partially forgotten by successive generations become the kernel of meaning in the etymology of new words and idiomatic phrases.

If you have ever walked an old forest, you've seen how layers of crud establish ecosystems. Centuries of dead plants, bugs, animal corpses and their excrement make up your typical forest floor. Ironically, old growth forests can be threatened by invasive earthworms, which strip the built-up humus, destabilizing the habitat in their wake. In practice, language too can be eaten away by ideological fundamentalists in an effort to return to roots that never were.

Software cultures unnaturally lack the level of fecundity found in living ecosystems systems. Vendors, language designers and users have yet to hit upon the mix of value choices and whatever thresholds are necessary to establish a self-sustaining environment. It may well be that the tensions between the community and vendors inhibits or prohibits the system from reaching a sustainable state. 

PHP (and perhaps to a lesser extent Perl) is a good example of an ecosystem gone septic, succumbing to its own popular success with a sea of craptastisicm. You can find a lot of code there, but it is of a dubious, messy yet inorganic quality - like a garbage dump.  The comparison-others Ruby, Python and Node.js, present smaller yet more vibrant and cohesive cultures; there is a lot of organic crud there too, but the public code is more organic and self-referential, akin more to a compost heap than a garbage pile.

I suspect a big part of the difference is in the recognition (or lack thereof) of the problem of factoring dependencies with solutions for package management and effective abstraction constructions. PHP just doesn't present an ecosystem that values this general class of problems, and this is reflected in the unusable crud it builds up over time.