Largely, this is an engineering scale based on the needs of software businesses. It’s not about mathematical ability or the capacity to write fast algorithms. Nor is it about the depth of knowledge a programmer has about the internals of the Linux kernel. Those are important, and will usually develop in tandem with an engineer’s skill, but not the primary focus of this system. It doesn’t represent everything one needs to know in order to evaluate a software developer. It’s also not a substitute for “performance reviews” or to be used for “HR matters”. Sometimes a “2.1″ may need to be fired for personality reasons, whereas an amicable “1.2″ is an asset to a company.

Finally, this scale is somewhat context-dependent, noting the difference between skill and impact. Change the programming language or technology stack, and a developer can rise or drop by a full point. Also, a developer needs to be on a team for 6 to 12 months (in most cases) to reach a level of impact commensurate with her capability. It’s tempting to speak of levels like “1.5″ as if they were objective ratings of a person’s skill. More accurately, they describe scopes of contribution and fit between a person and job role. The size and needs of a company also have a major impact. Small technology companies often encourage their people to “experiment” 0.2-0.5 points above their level of demonstrated skill, and to thereby grow quickly, while large and bureaucratic firms usually slot people an equal amount below their skill levels (because of inherent limitations on leadership positions).

The scale I’m about to define comes from one insight about human organizations. Teams, in general, have four categories into which a person’s contribution can fall: dividers, subtracters, adders, and multipliers. Dividers are the cancerous people who have a broad-based negative effect on productivity. This usually results from problems with a person’s attitude or ethics– “benign incompetence” (except in managers, whose job descriptions allow them only to be multipliers or dividers) is rarely enough to have a “divider” effect. This is an “HR issue” (dividers must improve or be fired) but not the scope of this professional-development scale, which assumes good-faith and a wish for progress. Subtracters are people who produce less than they cost, including the time of others who must coach and supervise them. As a temporary state, there’s nothing wrong with being a subtracter– almost every software engineer starts out his career as one, and it’s common to be a subtracter in the first weeks of a new job. Adders are the workhorses: competent individual contributors who deliver most of the actual work. Finally, multipliers are those who, often in tandem with “adder” contributions, make other people more productive. In many industries, being a multiplier is thought to be the province of management alone, but in technology that couldn’t be farther from the truth, because architectural and infrastructural contributions (such as reusable code libraries) have a broad-based impact on the effectiveness of the entire company.

What this scale intends to measure is the transition from a subtracter to an adder (0.0 to 1.0), from an adder to a multiplier (1.0 to 2.0), and from a “local” to a “global” multiplier (2.0 to 3.0). I ignore issues associated with “dividers” (who may be highly competent engineers; as I alluded above, a 2.0+ engineer can be a “divider” if there are personality problems) because those are “HR issues” (improve quickly or fire) while this scale is concerned with skill and long-term professional development.

Approximately speaking, the ranges are:

0.0 to 0.4: Complete novice: this is the level of a person who is still “learning programming”, by which I mean the person is likely to have trouble getting code to compile. This person has technical limitations that make unsupervised professional-level contributions impossible. Typically, this level of development is observed only in introductory programming courses. Most college interns are already at 0.5 or above when they start programming professionally.

0.5 to 0.7: Sound grasp of fundamentals: this is the level of someone who can call herself a “programmer” but not yet an “engineer”. Can competently build small systems (up to 3000 lines of code or so) but the code is likely to be sloppy and unmaintainable. At this level, getting the code “to work” is not an issue, but the code is likely to be inefficient, and the person’s architectural skills (due to inexperience) are likely to be weak. Typical level for a college student from a strong school before her first software internship.

0.8 to 0.9: Becoming an adder: this is a person who is aware of the practical concerns (maintenance, runtime safety) associated with software development. She can deliver useful scripts and has a decent understanding of software engineering. She is capable of using Google and online resources to answer small-scoped questions about development (such as how to do File I/O in an unfamiliar language).

1.0 to 1.3: Full-fledged adder: this person has demonstrated full competence as a software engineer and can be trusted to manage a small project “full-cycle”: design, implementation, testing, and integration. Code will usually be of reasonable quality, but the engineer is not ready to be responsible for meeting deadlines, production support (being “the 3:00 am guy”), or company-wide concerns. A team of mostly 1.3 engineers can produce solid work (the average software engineer is around 1.1; note that the 50th-percentile software job is a low-autonomy “Java jockey” position) but a software company whose best engineers are in this range will struggle to produce quality software.

1.4 to 1.6: Solid adder: well above-average (top 10%) by software-industry standards, but average (at 5-10 years’ experience) in the context of the best companies (e.g. elite startups, Google, and research divisions of IBM). Can be trusted to independently solve most problems in an elegant and maintainable way. Engineer can make reasonable time estimates and communicate in both business and technical terms with management as well as other engineers. Can handle small levels of deadline/delivery responsibility and should be encouraged to participate in architectural decisions (“multiplier” concerns). Technical leadership non-urgent projects is a serious possibility.

1.7 to 1.9: Becoming a multiplier: top 5%. Engineer is on track toward full “multiplier” status. Her contributions not only solve immediate problems, but improve the state of company infrastructure. She routinely suggests architectural and process improvements, and is ready to be “tech lead” for important projects.

2.0 to 2.3: Full-fledged multiplier: engineer is an objective multiplier whose contributions add immense, demonstrated value to the team– an obvious technical leader. This represents the top 2-3% of software engineers in problem-solving, architectural, and leadership skills.

2.4 to 2.6: Becoming a global multiplier (“Fellow”): engineer’s achievements are vast and astonishing. Contributions are company-wide or extend even further (e.g. to the open-source community). Represents the top 0.25% of software engineers. Can be trusted to work on independent research (with full autonomy) and lead major initiatives.

2.7 to 3.0: Senior fellow: engineer is known within and outside the company as one of the best computer programmers alive. These are people who can design new programming languages and produce good ones.

How does this scale play out? What practical conclusions can we draw from it? Ideally, it exists to plot the trajectory that most programmers will take. Sadly, the reality of the software industry is that the average software engineer never gets far above 1.0. Since the difference between these awful programmers is not (at least, in my estimation) a problem of intellectual ability– perhaps it’s one of curiosity and drive– this is unfortunate.

Software is, by far, one of the most structurally cooperative systems in the world. What I mean by “structurally cooperative” is that our own well-being is correlated positively with the performance of others in our industry, even when they’re nominally (and transiently) “competitors”. Great programmers teach other programmers to be great and build great technologies, which are often released into the open-source community. Conversely, the badness of the average software developer causes a world of suffering for the good ones. Why are so many programming jobs (and libraries) in brain-dead languages (such as Java) rather than functional-programming languages (e.g. Scala, Ocaml, Clojure, Haskell)? Because while the Java+IDE environment makes it extremely difficult for an individual engineer to have a 1.5+ impact –there are two cases in recorded history of programmers breaking 2.0 in Java; one is Martin Odersky, who wrote Scala, and the other is Rich Hickey, who wrote Clojure– the state of the tooling makes it possible for the 0.7′s and 0.9′s to get a 0.1- to 0.3-point bump, so long as they stay within their IDEs and don’t have to work with a computer (gasp!) at the (God, no!) command line.

Sadly, the “half-life” of a software developer is about 6 years. The trajectory of an “average” software developer looks like this: he leaves a JavaSchool at a competency around 0.6 and grows by approximately 0.1 point per year– that’s not blazing along this stretch (0.1/year would be quite fast above 2.0, but it’s poky below 1.0) but a reasonable pace for a person with little mentoring and a lackluster environment. Around 1.2, he reaches a technical ceiling, convinced that “programming” has intrinsic limits on individual contribution, except for “geniuses” who have 167 IQs and 40 years of programming experience (at age 44). This is because the Java+IDE environment, and operating systems like Windows, are designed to bring up the rear (0.7 to 0.9) while holding back the best programmers, artificially imposing that “ceiling” around 1.2-1.5; these technologies are intentionally designed for a performance-middling effect– to make it possible for huge teams of “commodity” developers to produce something while holding the excellent back from producing great things.

If he had spent some time reading Hacker News and learning about this fancy “functional programming” stuff, he might have seen a credible path (for non-geniuses like this writer) to 1.5 and beyond. But he hasn’t, so he burns out on “code monkey” work and moves into management. Or he goes to law school. Or business school. Because he wrongly concluded that this “engineering” thing (as he experienced it) was just too limited and dull to do for another decade. Even after six years, he was a mediocre, bottom-90-percent programmer and so no one misses that he’s gone, but with better guidance and exposure to superior technologies, he could have become great.

This is what’s at stake in “language wars” and arguments about the configuration of the software industry. It’s not just about hating to write (much less maintain) code in unattractive languages like Java and C++, or use hideous and inelegant operating systems like Windows. Far more is involved here.

What makes this problem especially hard is that it’s not about money. In marketing terminology, it’s a problem of “reach”. Many great startups, even if they offered $200,000 per year, would simply be unable to hire 25 elite (top 5%) programmers in a year. Finding one per quarter is pretty good. Compounding this difficulty is the fact that the best software engineers’ job searches are infrequent and short. They usually find jobs through social connections and word-of-mouth before they start officially “looking”. They almost never cold-spam their CVs. Therefore, a company that’s consistently hiring top-5% programmers is probably extending offers to 1 applicant per several hundred CVs that makes its way to the hiring manager.

What is the Developer Divide? It comes down to this: it’s easier to get top developers if your product is something that top developers already use. Google was unequivocally the best search engine even in 2000, and nerds like new search engines, so it established itself as a desirable place to work. Facebook and Foursquare may be targeted toward “non-engineers” (i.e. mass market) but the products are used by enough of the people the firms want to hire as to generate name recognition that accumulates faster than the company needs to grow. That makes it very easy for them to attract so much talent they turn away more 5-percenters than they hire. But for a company whose product isn’t used already, every day, by top programmers, it becomes harder. Much harder. That’s unfortunate, because a lot of great businesses
(most, actually) need to start out doing something lower on the established-product-sexiness scale (enterprise work and products with well-defined markets, rather than speculative “social” projects) before branching out into projects of more general interest. These companies might become sexy in the future, but sometimes the first clientele has to be an unsexy but reliable one, like large corporations or suburban soccer moms.

So, how do these great companies continue to find great talent? I’m going to state one simple (but not easy) thing that any software company can do to increase its attractiveness to top engineers by at least 3 binary orders of magnitude. Here it is: ditch Java/C++ and use a decent programming language.

What’s a decent programming language, for this purpose? It’s one that a 5-percenter would use for a personal side project, or if she were calling the shots. That language might Scala, Python, or Ocaml. It might (for a project such as an operating system) be C. It would amost certainly not, in 2012, be Java or C++.

A clarification must be made, because it’s confusing to people outside of technology: C++ is not a substitute, nor  an improvement on, C. In fact, C is great for programs where low-level concerns are critical in producing a quality system: operating systems, device drivers, hard real-time, and runtime environments for the high-level garbage-collected languages we all love (such as Ocaml and Haskell). Much of the world is built on C. It’s an excellent and immensely successful mid-level language,  as opposed to C++ which is a miserably failed attempt at a C-like high-level language. So do not confuse the two.

If you ask a 5-percenter for his opinions on C and C++, he’ll probably praise the former while trashing the latter. From a non-technical business perspective, this might seem inconsistent, given that C is a proper subset of C++. “Anything you can do in C, you can do in C++.” That makes C++ “strictly better”, right? Well, no. A 5-percenter has the professional maturity to realize that he or she will not be coding in a vacuum, and that reading code is as important an act as writing it, and that therefore adding ill-considered features to a good language doesn’t improve it, but ruins it if people are stupid enough to actually use them.

I’ll stop talking about C++, because it’s not even relevant to most startups. Startups can’t afford it, because they need to accomplish big things with small teams and C++ doesn’t make it possible. C++ is mostly that skulking monster that lives in the bowels of legacy systems at banks, something you expect to fight in the 2300 AD (post-apocalyptic) world of Chrono Trigger.

Java’s problem is somewhat different and new. C++ is a bad language because it was poorly designed, but it was at least designed for good programmers (and it’s disastrous when used by inept ones). Java, in contrast, was explicitly designed to favor the interests of massive (100+) teams of mediocre developers over excellent individual contributors, whom it slows down to a small fraction of their typical speed. It came out of a failed experiment to create a home for low-skill “commodity” developers who haven’t learned a new skill since college and who need to consult the One Smart Person on the team if (God forbid) their RAM-munching IDE breaks. Java has succeeding in making commodity developers marginally effective (as opposed to negatively productive, as they are in more powerful languages) but at the expense of hobbling the best, forcing them to endure ugliness (inherent to a language designed to be used exclusively through IDEs, while 5-percenters overwhelmingly prefer the command-line interface and “classic” editors like vim and emacs) and accidental complexity. Needless to say, 5-percenters despise Java. More correctly, they despise the Java language.  (I emphasize “the language” because the Java Virtual Machine itself is a pretty powerful tool, and because there are superior languages– Scala and Clojure coming to mind– that also run on the JVM.)

If you want to build a large team of 50+ “commodity” developers, content to maintain legacy code or work on mind-numbingly boring stuff, use Java or C++. If you’re a startup, you need to build a small team of excellent developers, so use something else. If you want to hire 5-percenters now but might need to hire Java jockeys later, strongly consider Scala and Clojure, which are highly powerful languages but run in the JVM environment.

Why is this so important? It’s not just about the language. It’s about signaling. As a startup, you have to show that you get it, and that the opportunity you offer isn’t Yet Another Java Job. You can get 5-percenters to use C++ or Java (Google has made this happen, and so have many investment banks, which have huge legacy codebases in C++) but you pretty much have to be a big-name company to pull this off, and you can expect to shell out an obscene of money– a 50-100 percent markup to account for the negatives of maintaining code in a terrible language, and the career stagnation this kind of work invites. If you want a 5-percenter to work 60 hours per week for $100,000 per year, use a great language. If you want him to work 9-to-5, with two-hour lunches and personal errands deducted from the workday, for $250,000 per year, then Java and C++ are options.

More than anything else, 5-percenters want to work with other 5-percenters. This is far more important to them than prestige or money (the reason 5-percenters default to rich, prestigious companies when nothing more interesting crosses their transom is because these companies have other 5-percenters). It is, moreover, even more important than programming language selection itself. Language selection, as I’ve said, an objective mechanism of signaling. This is what creative writing calls the “show, don’t tell” principle (don’t say “Eric was honorable”; have him do something honorable). Every company says (“telling”) it has world-class talent, but language choice is an objective decision that shows that a company is, at the very least, interested in hiring 5-percenters. For that reason, there’s a good chance that it employs some.

To reiterate, because I don’t want a flame war, is it possible to find 5-percenters who will write Java and C++ on a full-time basis? Absolutely, and if you want to compete with Goldman Sachs on salary, you might be able to hire one. Will they take on the risk of working for $5,000 (pre-tax) per month at a risky, seven-person startup in these languages? Not a chance. Five-percenters tolerate C++ jobs at Google because they know that company’s existence doesn’t rely on their individual productivity. In a startup, individual productivity is an existential concern, and the Aspergerian “pathological honesty” that top programmers almost invariably have precludes them from working in low-productivity languages that they believe will retard and destroy their employer.

I’ve said enough about the awfulness of C++ and Java. What are some good languages? I’ll give a list, which is not at all inclusive, of highly-powerful languages that the best developers love. Ocaml, Haskell, Erlang, Scala, Lisp, Clojure. Less strong but still formidable are Python and Ruby. (Many 5-percenters love Python, and almost all will tolerate it, but the median Python programmer is closer to a “10-percenter”.) Why is it this way? First, great developers program and learn technology in their spare time, and a 1-person, 15-hour-per-week project must be written in a real language if it is going to amount to anything. Second, most of these languages are only used by the best employers and only taught by the best universities, so a person deeply familiar with one of them is either (a) coming from elite exposure, or (b) possessive of enough individual curiosity to indicate a high likelihood of skill and success as a computer programmer. People who want to become great programmers quickly discover languages in which it’s possible for them to be 10 times as productive as they would be in Java– and they never look back.

Are all programmers in great languages 5-percenters? The answer is no. Users of languages like Ocaml and Scala tend to fall into two categories: (a) the 5-percenters, and (b) those who are becoming 5-percenters. Not all of them are there yet, but they’re almost all improving at a rapid pace. I’ve worked for almost 4 years in JVM languages (Java, Scala, and Clojure) and what I’ve learned (perhaps astonishingly) is that, per unit time, the Scala and Clojure developers learn about Java faster than those using Java! Because they are in high-productivity languages, they can accomplish more, and because they’re achieving more, they’re learning more along the way.

From a practical standpoint: with so many great languages to choose from, which one of those awesome languages should a person pick? CTOs making this decision have some idea, but this would be an impossible decision for a non-technical CEO to make on direct experience. For a very small team, the answer is easy: whatever the best programmers want to use. I like Scala much better than Python, but if I were in a non-coding role and tasked with hiring a great programmer, and if she preferred Python, I’d have her use that, because the benefit of letting her use the language she thought was best for the job outweighs (for a small team and with no maintenance burden) any benefit conferred by using one powerful language over another. So my answer to the language-selection question to a non-programmer CEO is: ask your best developer.

For my part, I’d recommend Scala. It’s not the best language for all purposes, but it’s a great general-purpose language and (among mainstream languages) may be best choice overall for most purposes. Because it runs in the Java Virtual Machine (JVM) it has full access to all of Java’s assets. (Clojure, an excellent JVM lisp, has the same advantage, but is not as performant and does not have static typing, a feature I find invaluable.)

A person versed in economics might find my argument tenuous. If a set of “elite” languages can be used by programmers and employers as a signal of high competency, what’s to stop the less competent from “faking” this signal once they catch on to the fact that it exists? A few answers come to mind. The first is that programming in a high-power language requires an adjustment that mediocre, 9-to-5, programmers are not likely to want to make. From first principles, functional programming isn’t intellectually harder than object-oriented programming: a 120 IQ is more than enough. It’s actually simpler. (Doing object-oriented programming correctly, and rigorously understanding what one is doing with it, is much harder and much more intellectually complex than succeeding in functional programming; this is a rant for another time, but 99% of people doing “object-oriented-programming” are like crude teenagers with regard to sex– loud about it, but doing it badly.) Nonetheless, the intellectual difficulty of re-learning software on sound principles is not an easy one to make. Like the transitions from memorization to pattern recognition, and then from pattern recognition to rigorous proof in mathematics, these context-switches require a lot of work (months of serious study). Successfully learning Scala or Clojure is a sign of a very strong work ethic. (It goes without saying that your interview process should establish that the candidates actually know these languages, and aren’t playing “buzzword bingo”.)

What about mediocre businesses using language selection as a false signal? That’s even more unlikely. A recruiter (for an elite startup, currently at less than 20 people) I spoke to told me that about 70% of Clojure candidates, 40% of Python candidates, and 5% of Java candidates that he invites to an in-office, full-day interview are good enough to hire. For a startup, interviews are far more expensive (in terms of opportunity cost) than for large companies: it costs about $100 to run a technical phone screen and $1000 to conduct a full-cycle interview, because startups have to involve senior people in their interview process in order to assess quality. Put another way, this means that it costs $1429 to hire a 5-percenter in Clojure and $2500 to hire one in Python– chump change as far as recruiting expenses go. But it costs $20,000 to hire a Java developer, if you insist on the “5-percenter” standard of quality.

There’s one variable I haven’t mentioned, though, and that’s the number of CVs he gets in each language: several hundred times as many Java developers than developers in Clojure or Ocaml. Most companies need (or think they need) warm bodies in large numbers to maintain legacy horrors, not top-talent and the attitude that comes with it. Also, it’s awful that it’s this way, and I think it will change in the future, with the limiting factor being work ethic and engagement rather than innate ability, but the software world is a pyramid, with a few stars at the pinnacle and a large number of incompetents at the base. This holds for programmers and for programmer jobs, of which 90 to 95 are mind-numbingly boring. It’s also seen in the distribution of language preference (and I say “preference” because there are tens of thousands of excellent programmers writing C++ and Java right now, but very few prefer them). The result of this is that the mediocre languages have the most programmers. If a company needs to hire 200 programmers per month, it simply cannot choose Haskell as its main development language; within a couple years, it would have absorbed the entire Haskell community!

Historically, that has been a serious concern for companies when it comes to language selection. Because there are hundreds of times more Java developers than Haskell or Ocaml hackers, there are at least fives of times more half-decent ones. Thus, the worst languages paradoxically have the strongest library support. This is compounded by the fact that powerful tools (such as IDEs, which are a mixed bag of neatness and horror, but sometimes quite useful and outright required when developing in Java) must be written to compensate the shortcomings of hobbled languages. There’s no Lisp IDE because emacs does just fine, but there are a slew of Java IDEs because it’s a revolting experience to write Java without one. From a non-technical CEO’s perspective, this makes Java look better because it has the best supporting tools.

What all this means is that mediocre software shops are not going to switch over to Haskell in order to ape this signaling mechanism. They can’t, because the bottom contingent of their software staff with drop like flies, and because their leadership is unlikely to understand the language selection problem in the first place. There might be some unestablished software companies using elite languages, and of them, I make the same argument that I’d make of “15-percenters” who nonetheless show interest and competence in “elite” languages– they may not be 5-percenters now, but if they keep at it, they will be shortly!

What about the (admitted) shortcomings of elite languages? Except for Scala and Clojure, which have access to the JVM and interoperate cleanly with Java, these languages don’t have the breadth of tooling that C++ and Java do. The answer to that is almost stereotypically “hackerish”: write them! This effort is not wasted, not in the least. Writing high-quality open-source tools to support elite languages is one of the best things a software company can do to establish its reputation. This, again, is an opportunity to show, not tell.

For a small company attempting to define and establish itself, attracting top talent is hard. The best programmers are on the job market so rarely and for such short intervals that attracting them takes concerted effort. Growing companies do not have the name recognition, and cannot afford the immense salaries (over $250,000 for a senior developer) that would attract these “5-percenters” using economic means, so they must win on technical grounds. Language selection is a simple (but not easy, because it’s difficult to adopt new and dramatically more powerful tools) way to do so. The best languages (Ocaml, Scala, Haskell, Python) are “shibboleths” that elite programmers use to identify each other, so adopting one of them is a one-stop choice that instantly establishes “hacker cred” or, to use Nir Eyal’s terminology, bridges “The Developer Divide”.

“That revolting neologism was on its way down the drain,” said Santorum’s 19-year-old son, John. “I was so glad to be rid of all the ‘frothy mixture’ jokes. I thought it was all washed up. Now, Dad sneaks back out and does this. How in the fuck is a college student supposed to get laid with that association to his name?”

Santorum’s presidential run has been difficult not only for the candidate’s own children, but for the 2,319 American schoolchildren sharing that surname, which, before Santorum’s political career, held no sexual connotation. The candidate’s humiliating defeat in 2006 at the hands of Bob Casey, Jr. afforded these children a reprieve from surname-derived harassment, but one that was unfortunately short-lived. According to the American Bullying Institute, students with the afflicted surname have been the butt of 135 percent more jokes since the 2011 resurgence of Santorum’s political campaign.

Asked for comment, Rick Santorum expressed faith that his children would overcome the harassment, which he expected to “dry out overnight”. In a press conference he said, on the matter, “They’re tough little buggers.”

Not all in the family have been upset by the negative attention given to Rick Santorum’s name. Elizabeth, the candidate’s 20-year-old daughter, said, “The bright side of this ‘Santorum’ muckraking is that it distracts people from what a fascistic, mean-spirited, uncultured, stupid and downright-fucking-insane bigot he is.”

Please join me in voting out of office as many SOPA/PIPA supporters as we can. #FireSOPA, #FirePIPA.

The problem is twofold. First, natural languages don’t have the massive variations in capability observed in programming languages. French is not superior to English (or vice versa) in the way that Scala is superior to Java. Second, “switching costs” associated with acquiring a new natural language are high, because it’s hard for an adult human to learn a new natural language well: it takes years of exposure, preferably in the context of contact with native speakers. This is not as true of programming languages; learning programming is hard, but a skilled programmer can become capable in a new language in a few weeks and be proficient after months. Because of this clumsy analogy, decision-makers in the software industry often make a decision that is the obvious right one with regard to natural languages, and almost always the wrong one (because “the standard” is usually an underpowered legacy language like Java or C++) in selecting a programming language: they default to the standard.

My intention is not to discuss specific programming languages per se, but the problem inherent in the phrase “programming language”. It is a correct phrase: a programming language is a form of language, with grammar, syntax and semantics. On the other hand, it admits a certain confusion inherited from something we know about natural languages like English and Spanish: despite their evident differences, human languages are much more alike than they are different. Natural languages have their quirks and are may be more capable in specific ways– character languages are more compact, while letter languages’ smaller alphabets make it easier to convert written words into spoken form– but none is uniformly or astronomically superior to any other.

For example, there’s no natural language where to express an average English sentence (15 words) requires two hundred words, but there are languages (Scala, Python) that are 15 times more concise than Java. There’s also no language that is 100 times easier for the human brain to convert into meaningful instructions than English, but there are programming languages in which programs run 100 times faster than others. Nor is there a natural language where it’s impossible for a syntactically correct sentence (e.g. “the moon ate the moon”) to be nonsensical; this assurance, to a large degree, can be achieved in statically typed languages. While it might be offensive even to consider that natural languages might vary in “quality”, I think it quite reasonable to assert that they don’t. To my knowledge, there aren’t major, order-of-magnitude, variations in capability among natural languages. For this reason, it’s appropriate to communicate (e.g. to conduct business) in the language of which the involved parties will have the best comprehension. The right decision, in selecting a natural language, is to use “the standard”. In New York, that’s usually English. In Moscow, it’s Russian. Every locale has a small set (sometimes only one) of natural languages that are likely to be well-comprehended by most people. For this reason, it would be absurd for a Silicon Valley firm to make the decision to conduct all business in Swahili, even if Swahili were 10 or 50 percent better suited to that company’s needs.

Programming languages are different: there are order of magnitude variations. Rewrite a C program in Java, and the program becomes immune to a wide class of errors (due to Java’s automatic memory management) but one loses the ability to explicitly manage memory, and program performance becomes nondeterministic. Sometimes this is a desirable change; sometimes not. Rewrite the Java program in Python and the source code usually becomes one-tenth of its previous size, making the program easier to maintain and (in the long-run) better, but one forgoes access to the Java libraries (using Python’s, which are strong but less developed) and programs take 10 to 20 times longer to run. Rewrite the Python program in Scala and one regains access to the Java libraries and Java’s speed, but loses access to Python’s libraries and will have trouble transliterating some of Python’s (controversial, but sometimes powerful) most dynamic features.

Order-of-magnitude differences exist among programming languages, with the winners depending largely on one’s definition of “quality”. Python is expressive but the interpreter is slow. Java is fast but verbose and horrendous to read, making source code difficult to maintain and “code rot” inevitable. Scala is fast and concise, with a powerful type system, but few people understand the language or its type system well, so it’s not really possible to hire 50 Scala developers per month. C is verbose (for large-scale, complex software) and without many features, but the best or only choice for a variety of problems, such as writing device drivers, real-time programs, and operator systems. No programming language can be categorized as the uniform “best”; all have their strengths and weaknesses (except for C++, which is a bastardization of C and should never be used except to troll people).

This essay isn’t about what languages are good and which not, but it answers a question: why do most businesses use the wrong programming languages? Why are so many development shops using C++ and Java, when they could accomplish four times as much per developer-month using a more expressive language like Python or Scala? I think a major part of it comes from the fact that we call them programming languages. For an analogous question: Why do most American businesses use English? Not because English is the “best” language (if such a notion could even be defined for a natural language, and I doubt it can) but because it’s “the standard”: it’s what most other Americans use to communicate. With natural languages, there aren’t order-of-magnitude differences in capabilities, and the difficulty of learning a new natural language is high (the American “founding fathers” despised the British so much that they tried to replace English with French or Hebrew as the new country’s “official language”, and failed) so choosing the default natural language is such an obvious choice that it’s often not even a conscious decision.

Java and C++, analogously, are favored in the software industry because they’re “the standard”, like English. To a businessperson unfamiliar with technology, a proposal to implement software in Scala seems like the suggestion that all business should be conducted in Swahili– for a New York firm, ridiculous. This is why many software shops are far more skeptical of “other languages” than they would be if they understood the situation and the potential gains.

How do we dispel these notions? I think one change should be in our terminology. Often, when we debate languages, we’re actually discussing technologies built on languages. For example, sentences like “C is fast” and “Python is slow” are ridiculous as stated. A language (formally speaking) is just a set of strings of symbols, with no intrinsic concept of “speed”. What people mean, in fact, is that “compiled C programs have excellent performance due to advanced compilers and the language’s support for powerful optimizations” and “Python executables run about 10 to 20 times more slowly than analogous programs in C” (but still fast, because all computers are “fast” these days).  Speed isn’t an intrinsic trait of the language, but rather one of the technologies to compile and interpret it.  Likewise, the common justification for using Java is that “It’s has the largest set of active libraries.” Well, not exactly. There are a variety of exciting (and superior) JVM languages– most notably Scala and Clojure– that can use the bulk of these tools equally seamlessly, which means that there’s no excuse (in most cases) for preferring Java.

Programming languages are languages, but the decision to adopt one programming language over another has no similarity to such a decision over natural languages. The relevant comparison, in technology, is one of platforms, programming styles, capabilities, and supporting technologies. Referring to this decision-making process as a “language debate” is, in this context, myopic. Far more is at stake than the choice of “language” alone; the chosen programming language influences the types of programs one can write, and the variation in program quality that is staked on language selection alone is quite high.

Consider the Soviet Union. Did political and economic elites disappear when “business” was made illegal? No, not at all. Did the failings of large human organizations suddenly have less of a pernicious effect on human life? No; the opposite occurred. What was outlawed, effectively, was not the corporation (corporate power existed in the government) but small-scale entrepreneurship– a necessary social function. Certainly, elitism and favoritism didn’t go away. Instead, money (which was subject to tight controls) faded in importance in favor of blat, an intangible social commodity describing social connection as well as the peddling of influence and favors. With the money economy hamstrung by capitalism’s illegality, blat became a medium of exchange and a mechanism of bribery. People who were successful at accumulating and using social resources were called blatnoys. The blatnoy elite drove their society into corruption and, ultimately, failure. But… that’s irrelevant to American capitalism, right?

Well, no. Sadly, corporate capitalism is not run by “entrepreneurs” in any sense of the word. Being an entrepreneur is about putting capital at risk to achieve a profit. Someone who gets into an elite college because a Senator owes his parents a favor, spends four years in investment banking getting the best projects because of family contacts, gets into a top business school because his uncle knows disgusting secrets about the dean of admissions, and then is hired into a high position in a smooth-running corporation or private equity firm, is not an entrepreneur. Anything but. That’s a glorified private-sector bureaucrat at best and, at worst, a brazen, parasitic trader of illicit social resources.

There are almost no entrepreneurs in the American upper class. This claim may sound bizarre, but first we must define terms– namely, “upper class”. Rich people are not automatically upper class. Steve Jobs was a billionaire but never entered it; he remained middle-class (in social position, not wealth) his entire life. His children, if they want to enter its lower tier, have a shot. Bill Gates is lower-upper class at best, and has worked very hard to get there. Money alone won’t buy it, and entrepreneurship is (by the standards of the upper class) the least respectable way to acquire wealth. Upper class is about social connections, not wealth or income. It’s important to note that being in the upper class does not require a high income or net worth; it does, however, require the ability to secure a position of high income reliably, because the upper class lifestyle requires (at a minimum) $300,000 after tax, per person, per year.

The wealth of the upper class follows from social connection, and not the other way around. Americans frequently make the mistake of believing (especially when misled on issues related to taxation and social justice) that members of the upper class who earn seven- and eight-digit salaries are scaled-up versions of the $400,000-per-year, upper-middle-class neurosurgeon who has been working intensely since age 4. That’s not the case. The hard-working neurosurgeon and the well-connected parasite are diametric opposites, in fact. They have nothing in common and could not stand to be in the same room together. their values are at odds. The upper class views hard work as risky and therefore a bit undignified. It perpetuates itself because there is a huge amount of excess wealth that has congealed at the apex of society, and it’s relatively easy to exchange money and blat on an informal but immensely pernicious market.

Consider the fine art of politician bribery. The cash-for-votes scenario, as depicted in the movies, is actually very rare. The Bush family did have their their “100k club” when campaign contributions were limited to $1000-per-person, but entering that set required arranging for 100 people to donate the maximum amount. Social effort was required to curry favor, not merely a suitcase full of cash. Moreover, to walk into even the most corrupt politician’s office today offering to exchange $100,000 in cash for voting a certain way would be met with a nasty reception. Most scumbags don’t realize that they’re scumbags, and to make a bribe as overt as that is to call a politician a scumbag. Instead, politicians must be bribed in more subtle manners. Want to own a politician? Throw a party every year in Aspen. Invite up-and-coming journalists just dying to get “sources”. Then invite a few private-equity partners so the politician has a million-dollar “consulting” sinecure waiting if the voters wise up and fire his pasty ass. Invite deans of admissions from elite colleges if he has school-age children. This is an effective strategy for owning (eventually) nearly all of America’s decision makers; but it’s hard to pull off if you don’t own any of them. What I’ve described is the process of earning interest on blat and, if it’s done correctly and without scruples, the accrual can occur rapidly– for people with enough blat to play.

Why is such “blat bribery” so common? It makes sense in the context of the mediocrity of American society. Despite the image of upper management in large corporations as “entrepreneurial”, they’re actually not entrepreneurs at all. They’re not the excellent, the daring, the smartest, or the driven. They’re successful social climbers; that’s all. The dismal and probably terminal mediocrity of American society is a direct result of the fact that (outside of some technological sectors) it is incapable of choosing leaders, so decisions of leadership often come down to who holds the most blat. Those who thrive in corporate so-called capitalism are not entrepreneurs but the “beetle-like” men who thrived in the dystopia described in George Orwell’s 1984.

Speaking of this, what is corporate “capitalism”? It’s neither capitalism nor socialism, but a clever mechanism employed by a parasitic, socially-closed but internally-connected elite to provide the worst of both systems (the fall-flat risk and pain of capitalism, the mediocrity and procedural retardation of socialism) while providing the best (the enormous rewards of capitalism, the cushy safety of socialism) of both for themselves.

These well-fed, lily-livered, intellectually mediocre blatnoys aren’t capitalists or socialists. They’re certainly not entrepreneurs. Why, then, do they adopt the language and image of alpha-male capitalist caricatures more brazen than even Ayn Rand would write? It’s because entrepreneurship is a middle-class virtue. The middle class of the United States (for not bad reasons) still has a lot of faith in capitalism. Upper classes know that they have to seem deserving of their parasitic hyperconsumption, and to present the image of success as perceived by the populace at large. Corporate boardrooms provide the trappings they require for this. If the middle class were to suddenly swing toward communism, these boardroom blatnoys would be wearing red almost immediately.

Sadly, when one views the social and economic elite of the United States, one sees blatnoys quite clearly if one knows where to look for them. Fascists, communists, and the elites of corporate capitalism may have different stated ideologies, but (just as Stephen King expressed that The Stand‘s villain, Randall Flagg, can represent accurately any tyrant) they’re all basically the same guy.

I’ll ignore the more nefarious but low-frequency ills behind our heavy-handed justice system, such as racism and disproportionate fear. Instead, I want to focus on a more fundamental question. Why do average people, with no ill intentions, believe that negative experiences are the best medicine for criminals, despite the overwhelming amount of evidence that most people behave worst after negative experiences? I believe that there is a simple reason for this. The model that most people have for the criminal is one we’ve seen over and over: The Bully.

A topic of debate in the psychological community is whether bullies suffer from low or high self-esteem. Are they vicious because they’re miserable, or because they’re intensely arrogant to the point of psychopathy? The answer is both: there are low-self-esteem bullies and high-self-esteem bullies, and they have somewhat different profiles. Which is more common? To answer this, it’s important to make a distinction. With physical bullies, usually boys who inflict pain on people because they’ve had it done to themselves, I’d readily believe that low self-esteem is more common. Most physical bullies are exposed to physical violence either by a bigger bully or by an abusive parent. Also, physical violence is one of the most self-damaging and risky forms of bullying there is. Choosing the wrong target can put the bully in the hospital, and the consequences of being caught are severe. Most physical bullies are, on account of their coarse and risky means of expression, in the social bottom-20% of the class of bullies. On the whole, and especially when one includes adults in the set, most bullies are social bullies. Social bullies include “mean girls”, office politickers, those who commit sexual harassment, and gossips who use the threat of social exclusion to get their way. Social bullies may occasionally use threats of physical violence, usually by proxy (e.g. a threat of attack by a sibling, romantic partner, or group) but their threats generally involve the deployment of social resources to inflict humiliation or adversity on other people. In the adult world, almost all of the big-ticket bullies are social bullies.

Physical bullies are split between low- and high-self-esteem bullies. Social bullies, the only kind that most people meet in adult life, are almost always high-self-esteem bullies, and often get quite far before they are exposed and brought down. Some are earning millions of dollars per year, as successful contenders in corporate competition. Low self-esteem bullies tend to be pitied by those who understand them, which is why most of us don’t have any desire to hunt down the low self-esteem bullies who bothered us as children. It’s high self-esteem bullies that gall people the most. High self-esteem bullies never show remorse, often are excellent at concealing the damage they do, even to the point of bringing action consequences of their actions to the bullied instead of to themselves, and they generally become more effective as they get older. It’s easy to detest them; it would be unusual not to.

How is the high self-esteem bully relevant to criminal justice? At risk of being harsh, I’ll assert what most people feel regarding criminals in general, because for high-self-esteem bullies it’s actually true: the best medicine for a high self-esteem bully is an intensely negative and humiliating experience, one that associates undesirable and harmful behaviors with negative outcomes. This makes high-self-esteem bullies different from the rest of humanity. They are about 3 percent of the population, and they are improved by negative, humiliating experiences. The other 97 percent are, instead, made worse (more erratic, less capable of socially desirable behavior) by negative experiences.

The most arrogant people only respond to direct punishment, because nothing else (reward or punishment) can matter to them, coming from people who “don’t matter” in their minds. Rehabilitation is not an option, because such people would rather create the appearance of improvement (and become better at getting away with negative actions) than actually improve themselves. The only way to “matter” to such a person is to defeat him. If the high-self-esteem bully’s negative experiences are paralyzing, all the better.

Before going further, it’s important to say that I’m not advocating a massive release of extreme punishment on the bullies of the world. I’m not saying we should make a concerted effort punish them all so severely as to paralyze them. There are a few problems with that. First, it’s extremely difficult to determine, on an individual basis, a high self-esteem bully from a low-self-esteem one, and inflicting severe harm on the latter kind will make him worse. Humiliating a high-self-esteem bully punctures his narcissism and hamstrings him, but doing so to a low-self-esteem bully accelerates his self-destructive addiction to pain (for self and others) and leads to erratic, more dangerous behaviors. What comes to mind is the behavior of Carl in Fargo: he begins the film as a “nice guy” criminal but, after being savagely beaten by Shep Proudfoot, he becomes capable of murder. In practice, it’s important to know which kind of bully one is dealing with before deciding whether the best response is rehabilitation (for the low self-esteem bully) or humiliation (for the high self-esteem bully). Second, if bullying were associated with extreme punishments, the people who’d tend to be attracted to positions able to affix the “bully” label would be, in reality, the worst bullies (i.e. a witch hunt). That high self-esteem bullies are (unlike most people) improved by negative experience is a fact that I believe few doubt, but “correcting” this class of people at scale is a very hard problem, and doing so severely involves risk of morally unacceptable collateral damage.

How does this involve our criminal justice policy? Ask an average adult to name the 3 people he detests most among those he personally knows, and it’s very likely that all will be high self-esteem bullies, usually (because physical violence is rare among adults) of the social variety. This creates a template to which “the criminal” is matched. We know, as humans, what should be done to high-self-esteem bullies: separation from their social resources in an extremely humiliating way. Ten years of extremely limited freedom and serious financial consequences, followed by a lifetime of difficulty securing employment and social acceptance. For the office politicker or white-collar criminal, that works and is exactly the right thing. For the small-time drug offender or petty thief? Not so much. It’s the wrong thing.

Most caught criminals are not high self-esteem bullies. They’re drug addicts, financially desperate people, sufferers of severe mental illnesses, and sometimes people who were just very unlucky. To the extent that there are bullies in prison, they’re mostly the low-self-esteem kind– the underclass of the bullying world, because they got caught, if for no other reason. Inflicting negative experiences and humiliation on such people does not improve them. It makes them more desperate, more miserable, and more likely to commit crimes in the future.

I’ve discussed, before, why Americans so readily support the interests of the extremely wealthy. Erroneously, they believe the truly rich ($20 million net worth and up) to be scaled-up versions of the most successful members of the middle class. They conflate the $400,000-per-year neurosurgeon who has been working hard since she was 5 with the parasite who earns $3 million per year “consulting” with a private equity firm on account of his membership in a socially-closed network of highly-consumptive (and socially negative) individuals. Conservatives mistake the rich for the highly productive because, within the middle class, this correlation of economic fortune and productivity makes some sense, while it doesn’t apply at all to society’s extremes. The same is at hand in the draconian approach this country takes to criminal justice. Americans project the faces of the bullies onto the criminal, assuming society’s worst actors and most dangerous failures to be scaled-up version of the worst bullies they’ve dealt with. They’re wrong. The woman who steals $350 of food from the grocery store out of desperation is not like the jerk who stole kids’ lunch money for kicks, and the man who kills someone believing God is telling him to do so (this man will probably require lifetime separation from society, for non-punitive reasons of public safety and mental-health care) is not a scaled-up version of the playground bully.

In the U.S., the current approach isn’t working, of course, unless its purpose is to “produce” more prisoners (“repeat customers”). Few people are improved by prison, and far fewer are helped by the extreme difficulty that a felony conviction creates in the post-incarceration job search. We’ve got to stop projecting the face of The Bully onto criminals– especially nonviolent drug offenders and mentally ill people. Because right now, as far as I can tell, we are The Bully. And reviewing the conservative politics of this country’s past three decades, along with its execrable foreign policy, I think there’s more truth in that claim than most people want to admit.

Why do people like him appear only a few times in a generation? It’s not that there is a lack of talent. Not to denigrate Steve Jobs, because his talents are in any case uncommon, but I’d bet heavily (on statistical grounds) that there are at least a few hundred or thousand people like him out there, hacking away at long-shot startups, cloistered in academia, or possibly toiling in corporate obscurity. The issue is that people with his level of talent almost never succeed in human organizations such as large corporations. A keen aesthetic sense is a severe liability in most corporate jobs, as the corporate style of “professionalism” requires tolerance of ugliness rather than the pursuit of aesthetic integrity at all costs. Creative talent also becomes a negative in environments that expect years of unfulfilling dues-paying before one gets creative control of anything. People like Jobs can’t stand to waste time and so, in corporate America, they rarely make it. When they are set to boring work, the part of their brains not being used scream at them and that makes it impossible for them to climb the ladder.

That human organizations are poor at selecting leaders is well-known, and startups are an often-cited antidote to this problem. The issue there is that a herd mentality still exists in venture capital and startup press. The herd almost never likes people of high talent, and for every independent thinker in a true king-making role, there are twenty overfed bureaucrats making decisions based on “track record” and similarity to existing successes– a fact not in the favor of a 22-year-old Steve Jobs in 2011. To be honest, I don’t think such a person would have a chance in hell of acquiring venture capital funding on his own, unless born into the necessary connections. Elite incubators such as Y Combinator are solving this problem, and quite well, by connecting young talent with the necessary resources and connections. Are they doing enough? Will they succeed in changing the dynamics of startup funding and traction? I don’t have the foresight to answer that question yet; honestly, I have no idea. Time will tell.

I think a lot of people around my age (28) have spent some time thinking: How can I be more like Steve Jobs? That’s the wrong question to be asking. The perfect storm that enables even a moderately creative person (much less an out-and-out disruptive innovator like Jobs) to overcome the enormous obstacles human organizations throw at them is an event that occurs less often than a $100-million lottery windfall. The right question is closer to this: what can I do that makes people with immense creative talents, like Steve Jobs, more likely to succeed? So this, I believe, is a more reliable path to success and an indirect sort of greatness. It’s great to have dreams and work hard to achieve them, but it’s equally noble (and more often successful) to help others with great ideas realize theirs. Right now, most people with great ideas almost always linger in obscurity, with powerful people and institutions actively working to keep them there. That has been the case for nearly all of human history, but technology can change it.

How? I’m not sure. I spend a lot of time thinking about this issue. How can a world with so much creative talent in it (on account of having nearly 7 billion living people, at least a billion of whom now have the minimum resources to express creativity and at least a few million among those having the talent) be achieving so little? Why is there so much ugliness, inertia and failure? How do we empower people to change it? These questions are too big for me to answer, at least for now. I’m going to focus on one thing: the process of turning talent into greatness, the former being abundant and the latter being desperately uncommon. How do we get people with high talent to such a degree of skill that they can, as individual contributors, contribute to society substantially– so much so as to overcome the general mediocrity of human institutions?

This is an educational problem, but not one solved by traditional schooling. Greatness doesn’t come from completing assignments or acing tests, obviously. It requires personal initiative, a will to innovate, and courage. Then it requires recognition; people who show the desire to achieve great things should be given the resources to try. It doesn’t seem like it, but I’ve exposed an extremely difficult problem, one that I don’t know how to solve. Educational processes that encourage creativity make it extremely difficult to measure performance, and therefore fail to gain the social trust necessary to propel their pupils into positions of creative control in major organizations. On the other hand, those educational processes in which it’s easy to measure performance generally favor conformity and the ability to “game the system” over useful creative talents or skills. Moreover, there’s a variety of grade inflation that exists far beyond academia whose effect is socially pernicious.

Grade inflation seems like a “feel good” consequence of people being “too nice”, but from a zero-sum economic perspective, it actually reflects a society that stigmatizes failure heavily. If an average performance is rated at 2 (a C grade) on a 0-to-4 scale, then excellence in one course (A, 4.0) cancels out failure (F, 0.0) in another. On the other hand, if the average is 3.2 out of 4, then it takes four excellent grades to cancel out one failure. This makes failing a course substantially more costly. This reflects risk-aversion on the part of the educational system– the student who puts forth a mediocre performance in three courses is superior to one who excels in two but fails the third– and engenders risk-averse behavior on the part of students. That said, I’m not especially concerned with this problem in the educational system, which is far more forgiving of good-faith failure than most human organizations. A failed course can damage one’s average but rarely results in expulsion. I’m more worried about how this mentality plays out in real life.

This style of risk-aversion is reflected in human organizations such as corporations. An employee who has four great years followed by a bad one is likely to be fired for the damaged reputation acquired in that failed fifth year. People are measured according to their worst-case performance (reliability) rather than their best-case capability (talent). This is a problem for a person like Steve Jobs, obviously capable of hitting the highest of the high notes, but liable to show negative contribution (pissing people off) at his worst. It’s also a more general problem that leaves large organizations unable to tap their best people. Why? Those best people tend overwhelmingly to be “high-variance” people– those whose job performance becomes weak if they lose motivation, and who become so passionate about the quality of work that they invariably end up in personality conflicts. Low-variance individuals– generally lacking creativity but capable of sustaining a middling performance for decades, thereby showcasing “professionalism”– tend to win out in their stead. The creatively desolate world we observe in the leadership of major human organizations is a direct result of this.

In some cases, measuring performance at a person’s bottom rather than top makes sense. As Lord Acton said, “Judge talent at its best and character at its worst.” People who behave in a way that is outright unethical have proven themselves not worthy of trust, regardless of their best-case capabilities. On the other hand, a person like Steve Jobs fails in a mainstream corporate environment fails not because he is unethical but because he’s merely difficult. That is what is in error. In general, human organizations develop a toddler-like, black-and-white view in evaluation of their members and thereby lose the ability to discriminate between those who are outright criminal (and should be distrusted, terminated from employment, and possibly punished regardless of their talents) from those who have difficult personalities or who suffer a good-faith failure (a risk one must be able to afford if one wants to achieve anything).

There’s a solution to that problem, but a touchy one. In technology, programmers have taken to the open-source community as a means of building an organization-independent career. This reflects what academia has had for a long time: active encouragement for its members (graduate students and post-docs especially) to build reputations outside of their institutions. This allows people to demonstrate their best-case capabilites to the world at large. Unfortunately, there is an extremely touchy political matter here. Corporations and managers within them would generally prefer that subordinates not dedicate energy to the cultivation of an external reputation, a process that (a) distracts them from their “real work” and dues-paying, and (b) makes them more expensive to retain. Many large companies forbid employees to take consulting work or publish papers for this precise reason.

Now that I’ve laid out a few problems and ideas, I’ve noticed that both time (9:08 am, and I haven’t yet left for work) and word count (1683 and rising) are encouraging me to finish up. For a closing thought, I’ll admit that I don’t have many answers to the “big picture” problems here. I don’t know what it will take to fix the problems of large human organizations that lead to their pervasive mediocrity. I don’t even know if it can be done. Where to focus? Something smaller and tractable, something grass-roots.

Brilliant people, like Steve Jobs, aren’t born fully-fledged like Venus from the sea. Jobs became what he was due to a thousand influences– his friendship with Steve Wozniak, his early successes, later failures, and yet-later successes. That much is obvious. He was always talented but he became great on account of the opportunities he had– the successes and failures that refined his aesthetic sense until (in his later adulthood) it was a finished product. I also believe that there are thousands of people much like him in existence, their talents unexpressed. I don’t think we need to work hard to “find” them. These people are loud and obnoxious enough that this is an easy task. What we need to do, as much as we can, is enable such people to overcome the hurdles imposed by large human organizations more interested in protecting entrenched mediocrity than in pursuing excellence. We need to fight that battle as much as we can. And yet, we must accept that we aren’t likely to get very far. There’s more we need to do.

We need to rethink education. I’m not just talking about schooling. Instead, I’m talking about technology and business and culture. We need to remove from ourselves the notion that education is a “product” to be “consumed” by those rendered economically useless by their youth and inexperience. Education needs to be an ongoing process. It needs to pervade everything we do. Instead of writing code, managing people, or running businesses we need to focus on teaching people what we can, and on learning from them reciprocally. We need to reinvent corporate and organizational cultures outright so that talent is naturally turned into greatness, and so that excellence isn’t a weird, socially counterproductive personality trait but something that all strive toward.

This is a poignant reminder to all of us that we may have less time than we think we do. Society, with this expressed most prominently in the early rounds of the career game, expects young people to deny their mortality– to delay creative expression and fulfilling work to a distant, prosperous future that might never come. Jobs never denied his mortality. He did what he loved. This is the only reasonable approach because, in the end, mortality refuses to be denied.

Before reading further, I ask the reader to peruse Jeff Atwood’s excellent blog post, a header for what I’m about to write. In my mind, the most important point he makes is this one: We should probably have our best developers doing software maintenance, not whoever draws the shortest straw. 

Code maintenance, as usually practiced, is dull and lacks creativity. It’s also usually a career tarpit, because new invention is generally superior (from a developer’s perspective) in terms of visibility, education and advancement. Maintenance work rarely delivers on any of these. Maintaining one’s own code is a crucial educational experience, in which the developer sees first-hand the consequences of her architectural decisions. Maintaining the monstrosities created by others? It’s not much of a learning experience after the first time it’s done, and it’s rarely visible either. This incentive structure is perverse, because a working, healthy system that fulfills known, existing needs (i.e. the outcome of a successful maintenance project) is of more value than a new one that may or may not prove to be useful.

Maintenance is extremely important work; it’s the upkeep of one of a software company’s most important assets. Unfortunately, certain systemic aspects of organizational cultures make it unlikely to be rewarded in accord with its importance. The consequence is that maintenance work tends to “roll downhill” until it reaches junior programmers who lack the leverage to work on anything else. The result of this is that maintenance projects are usually done by people who don’t have the organizational knowledge, level of skill, or, most importantly, the political pull to do the job properly.

Inept or even average-case software maintenance fails to improve the general health of the code. A few bugs may be fixed or reactive features added, but the overall health of the software tends to decline. Methods become longer, dependencies are added, junk parameters are added to existing functions, and the architectural intentions behind the original system are buried under thousands of hacks. Code maintenance as practiced tends to beget an even greater maintenance burden in the future.

What needs to change? There’s a simple answer. Give maintainers power. The reason why the vast majority of software maintenance projects fail to improve code health whatsoever is that the developers in a maintenance role (usually junior programmers with no clout) lack the political pull to do their jobs properly. Telling a more senior developer, who was “promoted away” from maintenance duties years ago, to drop everything and document prior work is generally not an option for the maintenance programmer. Nor is removing ill-advised functionality from interfaces (someone more senior is guaranteed to “need” that function) or declaring a project in need of a rewrite. This is the sort of clout that a maintainer needs to have in order to have any shot whatsoever at succeeding.

There’s a flip side of this. If code maintenance work implies power, it absolutely shouldn’t be delegated to junior developers who can’t yet be trusted with that kind of power.

Giving software maintainers power will remove most of the stigma associated with such projects, seeing as they are hated because of the lack of power they often entail. It’s taxation without representation. Often, the maintainer’s job is specified in uninspiring and untenable terms like, “Make it work exactly like the old system, but faster”. Bug-for-bug replications, as well as faithful reproductions of any software without a spec, are doomed to failure. Improving code health necessitates imposing change on client developers. Rarely are such costs considered tolerable for the sake of “mere” upkeep, and maintenance projects usually, for that reason, fail to provide lasting improvement. The low rate of success observed on such projects makes them undesirable, and therefore they are avoided by the most skilled developers.

On the other hand, if maintenance responsibilities were coupled with sufficient power to restore the health of the codebase, such projects could become attractive enough that those with skill and seniority would want to do them. People who work on a daily basis with software generally develop a deep interest in the system’s health, and this alone can be motivation to commit time to its upkeep. Software maintenance has the reputation of grunt work, but it really isn’t intrinsically unpleasant for all developers; there are people who would enjoy such projects immensely if they were properly structured.

On structure, it’s critical that a company of any size separate the “software engineer” job description (which currently entails creative, new invention as well as maintenance) into two separate job descriptions. One (I’ll call it the “software architect” track) exists for the creative engineers who tend to fail at maintenance efforts because of their inability to tolerate ugliness. The other (I’ll call it the “maintenance engineer” track) exists for the “Sherlock Holmes” sort who enjoy deciphering ineptly- or even hostilely-written code. Since qualified maintenance engineers are uncommon, and since most developers find that work undesirable, maintenance engineers should be paid highly. I would argue that a 30% premium is not unreasonable. Moreover, they must be given social status and clout, since their job is to be pains in the asses of developers who are potentially more senior than they are. If maintenance projects are structured and allocated in this way, it’s very unlikely that they will dominate engineering time as they do now.

One thing I would argue against, however, is any suggestion that the maintenance engineer’s existence should absolve developers of the responsibility for maintaining their own code. It doesn’t. Maintenance engineers may be given a supervisory role in code and design review, but maintenance of one’s own work should remain each developer’s own responsibility; otherwise, “moral hazard” kicks in. I will also note that it is a terrible practice for developers to be “promoted away” from maintenance responsibilities, those being delegated to less important folk, as a reward for a successful launch. If the launch’s success came at the expense of enormous “technical debt” (the interest on which is usurious) then a maintenance time-bomb has been built.

I’ve addressed the social changes required in order to allow successful software maintenance. What will this entail in terms of project structure? In-place maintenance of defective systems– any software module that requires full-time maintenance effort fits the bill for “defective”– will become rarer, with outright replacement more common. That’s a good thing, because software either enervates or empowers and the former kind can rarely be upgraded to the latter through incremental changes. Note here that I’m advocating replacement, not “complete rewrites”. In my opinion, total rewrites (unless specific efforts are made to avoid prior mistakes) are often as inadvisable as in-place maintenance; the latter keeps defective software on life support, but the former practice often generates new defects– often more flaws, on account of the time pressure that complete-rewrite projects usually face.

What is gained in doing all this? First of all, job satisfaction among software engineers improves dramatically. Instead of the industry being characterized by a gloomy, never-ending maintenance slog, the proper allocation of such work ensures that it’s done properly and quickly rather than being passed from one hapless junior developer to another. Second, engineer productivity goes through the roof, because engineers with creative and architectural talents can unlock them instead of being forced to contend with legacy misery. Third, employee retention is improved not only for individual software companies but for the industry as a whole. Instead of architecturally-inclined engineers being burned out in droves (I’ll note that people with strong architectural talents tend to be notoriously intolerant of ugliness, to a point that can be emotionally and intellectually crippling in software maintenance) by the legacy burden that is the current reality of the industry, they’re encouraged to make a career of it, the result being better software architecture across the board. The result of these improvements in architectural know-how will be better products– more comprehensible, more attractive, easier to improve and keep up, and more profitable.