The trajectory of a software engineer… and where it all goes wrong.

I’m going to do something difficult and controversial and, since this is a first cut at it, I probably won’t do it perfectly. I’m going to develop a system for evaluating the skill and impact of a software engineer or, at the least, outline a trajectory for the idealized growth of a software programmer. The scale runs from 0.0 to 3.0, with each whole number representing a threshold in professional development.

Largely, this is an engineering scale based on the needs of software businesses. It’s not about mathematical ability or the capacity to write fast algorithms. Nor is it about the depth of knowledge a programmer has about the internals of the Linux kernel. Those are important, and will usually develop in tandem with an engineer’s skill, but not the primary focus of this system. It doesn’t represent everything one needs to know in order to evaluate a software developer. It’s also not a substitute for “performance reviews” or to be used for “HR matters”. Sometimes a “2.1″ may need to be fired for personality reasons, whereas an amicable “1.2″ is an asset to a company.

Finally, this scale is somewhat context-dependent, noting the difference between skill and impact. Change the programming language or technology stack, and a developer can rise or drop by a full point. Also, a developer needs to be on a team for 6 to 12 months (in most cases) to reach a level of impact commensurate with her capability. It’s tempting to speak of levels like “1.5″ as if they were objective ratings of a person’s skill. More accurately, they describe scopes of contribution and fit between a person and job role. The size and needs of a company also have a major impact. Small technology companies often encourage their people to “experiment” 0.2-0.5 points above their level of demonstrated skill, and to thereby grow quickly, while large and bureaucratic firms usually slot people an equal amount below their skill levels (because of inherent limitations on leadership positions).

The scale I’m about to define comes from one insight about human organizations. Teams, in general, have four categories into which a person’s contribution can fall: dividers, subtracters, adders, and multipliers. Dividers are the cancerous people who have a broad-based negative effect on productivity. This usually results from problems with a person’s attitude or ethics– “benign incompetence” (except in managers, whose job descriptions allow them only to be multipliers or dividers) is rarely enough to have a “divider” effect. This is an “HR issue” (dividers must improve or be fired) but not the scope of this professional-development scale, which assumes good-faith and a wish for progress. Subtracters are people who produce less than they cost, including the time of others who must coach and supervise them. As a temporary state, there’s nothing wrong with being a subtracter– almost every software engineer starts out his career as one, and it’s common to be a subtracter in the first weeks of a new job. Adders are the workhorses: competent individual contributors who deliver most of the actual work. Finally, multipliers are those who, often in tandem with “adder” contributions, make other people more productive. In many industries, being a multiplier is thought to be the province of management alone, but in technology that couldn’t be farther from the truth, because architectural and infrastructural contributions (such as reusable code libraries) have a broad-based impact on the effectiveness of the entire company.

What this scale intends to measure is the transition from a subtracter to an adder (0.0 to 1.0), from an adder to a multiplier (1.0 to 2.0), and from a “local” to a “global” multiplier (2.0 to 3.0). I ignore issues associated with “dividers” (who may be highly competent engineers; as I alluded above, a 2.0+ engineer can be a “divider” if there are personality problems) because those are “HR issues” (improve quickly or fire) while this scale is concerned with skill and long-term professional development.

Approximately speaking, the ranges are:

0.0 to 0.4: Complete novice: this is the level of a person who is still “learning programming”, by which I mean the person is likely to have trouble getting code to compile. This person has technical limitations that make unsupervised professional-level contributions impossible. Typically, this level of development is observed only in introductory programming courses. Most college interns are already at 0.5 or above when they start programming professionally.

0.5 to 0.7: Sound grasp of fundamentals: this is the level of someone who can call herself a “programmer” but not yet an “engineer”. Can competently build small systems (up to 3000 lines of code or so) but the code is likely to be sloppy and unmaintainable. At this level, getting the code “to work” is not an issue, but the code is likely to be inefficient, and the person’s architectural skills (due to inexperience) are likely to be weak. Typical level for a college student from a strong school before her first software internship.

0.8 to 0.9: Becoming an adder: this is a person who is aware of the practical concerns (maintenance, runtime safety) associated with software development. She can deliver useful scripts and has a decent understanding of software engineering. She is capable of using Google and online resources to answer small-scoped questions about development (such as how to do File I/O in an unfamiliar language).

1.0 to 1.3: Full-fledged adder: this person has demonstrated full competence as a software engineer and can be trusted to manage a small project “full-cycle”: design, implementation, testing, and integration. Code will usually be of reasonable quality, but the engineer is not ready to be responsible for meeting deadlines, production support (being “the 3:00 am guy”), or company-wide concerns. A team of mostly 1.3 engineers can produce solid work (the average software engineer is around 1.1; note that the 50th-percentile software job is a low-autonomy “Java jockey” position) but a software company whose best engineers are in this range will struggle to produce quality software.

1.4 to 1.6: Solid adder: well above-average (top 10%) by software-industry standards, but average (at 5-10 years’ experience) in the context of the best companies (e.g. elite startups, Google, and research divisions of IBM). Can be trusted to independently solve most problems in an elegant and maintainable way. Engineer can make reasonable time estimates and communicate in both business and technical terms with management as well as other engineers. Can handle small levels of deadline/delivery responsibility and should be encouraged to participate in architectural decisions (“multiplier” concerns). Technical leadership non-urgent projects is a serious possibility.

1.7 to 1.9: Becoming a multiplier: top 5%. Engineer is on track toward full “multiplier” status. Her contributions not only solve immediate problems, but improve the state of company infrastructure. She routinely suggests architectural and process improvements, and is ready to be “tech lead” for important projects.

2.0 to 2.3: Full-fledged multiplier: engineer is an objective multiplier whose contributions add immense, demonstrated value to the team– an obvious technical leader. This represents the top 2-3% of software engineers in problem-solving, architectural, and leadership skills.

2.4 to 2.6: Becoming a global multiplier (“Fellow”): engineer’s achievements are vast and astonishing. Contributions are company-wide or extend even further (e.g. to the open-source community). Represents the top 0.25% of software engineers. Can be trusted to work on independent research (with full autonomy) and lead major initiatives.

2.7 to 3.0: Senior fellow: engineer is known within and outside the company as one of the best computer programmers alive. These are people who can design new programming languages and produce good ones.

How does this scale play out? What practical conclusions can we draw from it? Ideally, it exists to plot the trajectory that most programmers will take. Sadly, the reality of the software industry is that the average software engineer never gets far above 1.0. Since the difference between these awful programmers is not (at least, in my estimation) a problem of intellectual ability– perhaps it’s one of curiosity and drive– this is unfortunate.

Software is, by far, one of the most structurally cooperative systems in the world. What I mean by “structurally cooperative” is that our own well-being is correlated positively with the performance of others in our industry, even when they’re nominally (and transiently) “competitors”. Great programmers teach other programmers to be great and build great technologies, which are often released into the open-source community. Conversely, the badness of the average software developer causes a world of suffering for the good ones. Why are so many programming jobs (and libraries) in brain-dead languages (such as Java) rather than functional-programming languages (e.g. Scala, Ocaml, Clojure, Haskell)? Because while the Java+IDE environment makes it extremely difficult for an individual engineer to have a 1.5+ impact –there are two cases in recorded history of programmers breaking 2.0 in Java; one is Martin Odersky, who wrote Scala, and the other is Rich Hickey, who wrote Clojure– the state of the tooling makes it possible for the 0.7′s and 0.9′s to get a 0.1- to 0.3-point bump, so long as they stay within their IDEs and don’t have to work with a computer (gasp!) at the (God, no!) command line.

Sadly, the “half-life” of a software developer is about 6 years. The trajectory of an “average” software developer looks like this: he leaves a JavaSchool at a competency around 0.6 and grows by approximately 0.1 point per year– that’s not blazing along this stretch (0.1/year would be quite fast above 2.0, but it’s poky below 1.0) but a reasonable pace for a person with little mentoring and a lackluster environment. Around 1.2, he reaches a technical ceiling, convinced that “programming” has intrinsic limits on individual contribution, except for “geniuses” who have 167 IQs and 40 years of programming experience (at age 44). This is because the Java+IDE environment, and operating systems like Windows, are designed to bring up the rear (0.7 to 0.9) while holding back the best programmers, artificially imposing that “ceiling” around 1.2-1.5; these technologies are intentionally designed for a performance-middling effect– to make it possible for huge teams of “commodity” developers to produce something while holding the excellent back from producing great things.

If he had spent some time reading Hacker News and learning about this fancy “functional programming” stuff, he might have seen a credible path (for non-geniuses like this writer) to 1.5 and beyond. But he hasn’t, so he burns out on “code monkey” work and moves into management. Or he goes to law school. Or business school. Because he wrongly concluded that this “engineering” thing (as he experienced it) was just too limited and dull to do for another decade. Even after six years, he was a mediocre, bottom-90-percent programmer and so no one misses that he’s gone, but with better guidance and exposure to superior technologies, he could have become great.

This is what’s at stake in “language wars” and arguments about the configuration of the software industry. It’s not just about hating to write (much less maintain) code in unattractive languages like Java and C++, or use hideous and inelegant operating systems like Windows. Far more is involved here.

How any software company can cross “The Developer Divide”

Nir Eyal wrote a blog post, The Developer Divide: When Great Companies Can’t Hire, on this conundrum: there are a lot of excellent technology companies that haven’t managed to attract the brightest (and generally pickiest) engineers in sufficient numbers to hire as fast as they grow, a problem that forces a company either to lower its hiring standards or curtail growth, neither of which is desirable.

What makes this problem especially hard is that it’s not about money. In marketing terminology, it’s a problem of “reach”. Many great startups, even if they offered $200,000 per year, would simply be unable to hire 25 elite (top 5%) programmers in a year. Finding one per quarter is pretty good. Compounding this difficulty is the fact that the best software engineers’ job searches are infrequent and short. They usually find jobs through social connections and word-of-mouth before they start officially “looking”. They almost never cold-spam their CVs. Therefore, a company that’s consistently hiring top-5% programmers is probably extending offers to 1 applicant per several hundred CVs that makes its way to the hiring manager.

What is the Developer Divide? It comes down to this: it’s easier to get top developers if your product is something that top developers already use. Google was unequivocally the best search engine even in 2000, and nerds like new search engines, so it established itself as a desirable place to work. Facebook and Foursquare may be targeted toward “non-engineers” (i.e. mass market) but the products are used by enough of the people the firms want to hire as to generate name recognition that accumulates faster than the company needs to grow. That makes it very easy for them to attract so much talent they turn away more 5-percenters than they hire. But for a company whose product isn’t used already, every day, by top programmers, it becomes harder. Much harder. That’s unfortunate, because a lot of great businesses
(most, actually) need to start out doing something lower on the established-product-sexiness scale (enterprise work and products with well-defined markets, rather than speculative “social” projects) before branching out into projects of more general interest. These companies might become sexy in the future, but sometimes the first clientele has to be an unsexy but reliable one, like large corporations or suburban soccer moms.

So, how do these great companies continue to find great talent? I’m going to state one simple (but not easy) thing that any software company can do to increase its attractiveness to top engineers by at least 3 binary orders of magnitude. Here it is: ditch Java/C++ and use a decent programming language.

What’s a decent programming language, for this purpose? It’s one that a 5-percenter would use for a personal side project, or if she were calling the shots. That language might Scala, Python, or Ocaml. It might (for a project such as an operating system) be C. It would amost certainly not, in 2012, be Java or C++.

A clarification must be made, because it’s confusing to people outside of technology: C++ is not a substitute, nor  an improvement on, C. In fact, C is great for programs where low-level concerns are critical in producing a quality system: operating systems, device drivers, hard real-time, and runtime environments for the high-level garbage-collected languages we all love (such as Ocaml and Haskell). Much of the world is built on C. It’s an excellent and immensely successful mid-level language,  as opposed to C++ which is a miserably failed attempt at a C-like high-level language. So do not confuse the two.

If you ask a 5-percenter for his opinions on C and C++, he’ll probably praise the former while trashing the latter. From a non-technical business perspective, this might seem inconsistent, given that C is a proper subset of C++. “Anything you can do in C, you can do in C++.” That makes C++ “strictly better”, right? Well, no. A 5-percenter has the professional maturity to realize that he or she will not be coding in a vacuum, and that reading code is as important an act as writing it, and that therefore adding ill-considered features to a good language doesn’t improve it, but ruins it if people are stupid enough to actually use them.

I’ll stop talking about C++, because it’s not even relevant to most startups. Startups can’t afford it, because they need to accomplish big things with small teams and C++ doesn’t make it possible. C++ is mostly that skulking monster that lives in the bowels of legacy systems at banks, something you expect to fight in the 2300 AD (post-apocalyptic) world of Chrono Trigger.

Java’s problem is somewhat different and new. C++ is a bad language because it was poorly designed, but it was at least designed for good programmers (and it’s disastrous when used by inept ones). Java, in contrast, was explicitly designed to favor the interests of massive (100+) teams of mediocre developers over excellent individual contributors, whom it slows down to a small fraction of their typical speed. It came out of a failed experiment to create a home for low-skill “commodity” developers who haven’t learned a new skill since college and who need to consult the One Smart Person on the team if (God forbid) their RAM-munching IDE breaks. Java has succeeding in making commodity developers marginally effective (as opposed to negatively productive, as they are in more powerful languages) but at the expense of hobbling the best, forcing them to endure ugliness (inherent to a language designed to be used exclusively through IDEs, while 5-percenters overwhelmingly prefer the command-line interface and “classic” editors like vim and emacs) and accidental complexity. Needless to say, 5-percenters despise Java. More correctly, they despise the Java language.  (I emphasize “the language” because the Java Virtual Machine itself is a pretty powerful tool, and because there are superior languages– Scala and Clojure coming to mind– that also run on the JVM.)

If you want to build a large team of 50+ “commodity” developers, content to maintain legacy code or work on mind-numbingly boring stuff, use Java or C++. If you’re a startup, you need to build a small team of excellent developers, so use something else. If you want to hire 5-percenters now but might need to hire Java jockeys later, strongly consider Scala and Clojure, which are highly powerful languages but run in the JVM environment.

Why is this so important? It’s not just about the language. It’s about signaling. As a startup, you have to show that you get it, and that the opportunity you offer isn’t Yet Another Java Job. You can get 5-percenters to use C++ or Java (Google has made this happen, and so have many investment banks, which have huge legacy codebases in C++) but you pretty much have to be a big-name company to pull this off, and you can expect to shell out an obscene of money– a 50-100 percent markup to account for the negatives of maintaining code in a terrible language, and the career stagnation this kind of work invites. If you want a 5-percenter to work 60 hours per week for $100,000 per year, use a great language. If you want him to work 9-to-5, with two-hour lunches and personal errands deducted from the workday, for $250,000 per year, then Java and C++ are options.

More than anything else, 5-percenters want to work with other 5-percenters. This is far more important to them than prestige or money (the reason 5-percenters default to rich, prestigious companies when nothing more interesting crosses their transom is because these companies have other 5-percenters). It is, moreover, even more important than programming language selection itself. Language selection, as I’ve said, an objective mechanism of signaling. This is what creative writing calls the “show, don’t tell” principle (don’t say “Eric was honorable”; have him do something honorable). Every company says (“telling”) it has world-class talent, but language choice is an objective decision that shows that a company is, at the very least, interested in hiring 5-percenters. For that reason, there’s a good chance that it employs some.

To reiterate, because I don’t want a flame war, is it possible to find 5-percenters who will write Java and C++ on a full-time basis? Absolutely, and if you want to compete with Goldman Sachs on salary, you might be able to hire one. Will they take on the risk of working for $5,000 (pre-tax) per month at a risky, seven-person startup in these languages? Not a chance. Five-percenters tolerate C++ jobs at Google because they know that company’s existence doesn’t rely on their individual productivity. In a startup, individual productivity is an existential concern, and the Aspergerian “pathological honesty” that top programmers almost invariably have precludes them from working in low-productivity languages that they believe will retard and destroy their employer.

I’ve said enough about the awfulness of C++ and Java. What are some good languages? I’ll give a list, which is not at all inclusive, of highly-powerful languages that the best developers love. Ocaml, Haskell, Erlang, Scala, Lisp, Clojure. Less strong but still formidable are Python and Ruby. (Many 5-percenters love Python, and almost all will tolerate it, but the median Python programmer is closer to a “10-percenter”.) Why is it this way? First, great developers program and learn technology in their spare time, and a 1-person, 15-hour-per-week project must be written in a real language if it is going to amount to anything. Second, most of these languages are only used by the best employers and only taught by the best universities, so a person deeply familiar with one of them is either (a) coming from elite exposure, or (b) possessive of enough individual curiosity to indicate a high likelihood of skill and success as a computer programmer. People who want to become great programmers quickly discover languages in which it’s possible for them to be 10 times as productive as they would be in Java– and they never look back.

Are all programmers in great languages 5-percenters? The answer is no. Users of languages like Ocaml and Scala tend to fall into two categories: (a) the 5-percenters, and (b) those who are becoming 5-percenters. Not all of them are there yet, but they’re almost all improving at a rapid pace. I’ve worked for almost 4 years in JVM languages (Java, Scala, and Clojure) and what I’ve learned (perhaps astonishingly) is that, per unit time, the Scala and Clojure developers learn about Java faster than those using Java! Because they are in high-productivity languages, they can accomplish more, and because they’re achieving more, they’re learning more along the way.

From a practical standpoint: with so many great languages to choose from, which one of those awesome languages should a person pick? CTOs making this decision have some idea, but this would be an impossible decision for a non-technical CEO to make on direct experience. For a very small team, the answer is easy: whatever the best programmers want to use. I like Scala much better than Python, but if I were in a non-coding role and tasked with hiring a great programmer, and if she preferred Python, I’d have her use that, because the benefit of letting her use the language she thought was best for the job outweighs (for a small team and with no maintenance burden) any benefit conferred by using one powerful language over another. So my answer to the language-selection question to a non-programmer CEO is: ask your best developer.

For my part, I’d recommend Scala. It’s not the best language for all purposes, but it’s a great general-purpose language and (among mainstream languages) may be best choice overall for most purposes. Because it runs in the Java Virtual Machine (JVM) it has full access to all of Java’s assets. (Clojure, an excellent JVM lisp, has the same advantage, but is not as performant and does not have static typing, a feature I find invaluable.)

A person versed in economics might find my argument tenuous. If a set of “elite” languages can be used by programmers and employers as a signal of high competency, what’s to stop the less competent from “faking” this signal once they catch on to the fact that it exists? A few answers come to mind. The first is that programming in a high-power language requires an adjustment that mediocre, 9-to-5, programmers are not likely to want to make. From first principles, functional programming isn’t intellectually harder than object-oriented programming: a 120 IQ is more than enough. It’s actually simpler. (Doing object-oriented programming correctly, and rigorously understanding what one is doing with it, is much harder and much more intellectually complex than succeeding in functional programming; this is a rant for another time, but 99% of people doing “object-oriented-programming” are like crude teenagers with regard to sex– loud about it, but doing it badly.) Nonetheless, the intellectual difficulty of re-learning software on sound principles is not an easy one to make. Like the transitions from memorization to pattern recognition, and then from pattern recognition to rigorous proof in mathematics, these context-switches require a lot of work (months of serious study). Successfully learning Scala or Clojure is a sign of a very strong work ethic. (It goes without saying that your interview process should establish that the candidates actually know these languages, and aren’t playing “buzzword bingo”.)

What about mediocre businesses using language selection as a false signal? That’s even more unlikely. A recruiter (for an elite startup, currently at less than 20 people) I spoke to told me that about 70% of Clojure candidates, 40% of Python candidates, and 5% of Java candidates that he invites to an in-office, full-day interview are good enough to hire. For a startup, interviews are far more expensive (in terms of opportunity cost) than for large companies: it costs about $100 to run a technical phone screen and $1000 to conduct a full-cycle interview, because startups have to involve senior people in their interview process in order to assess quality. Put another way, this means that it costs $1429 to hire a 5-percenter in Clojure and $2500 to hire one in Python– chump change as far as recruiting expenses go. But it costs $20,000 to hire a Java developer, if you insist on the “5-percenter” standard of quality.

There’s one variable I haven’t mentioned, though, and that’s the number of CVs he gets in each language: several hundred times as many Java developers than developers in Clojure or Ocaml. Most companies need (or think they need) warm bodies in large numbers to maintain legacy horrors, not top-talent and the attitude that comes with it. Also, it’s awful that it’s this way, and I think it will change in the future, with the limiting factor being work ethic and engagement rather than innate ability, but the software world is a pyramid, with a few stars at the pinnacle and a large number of incompetents at the base. This holds for programmers and for programmer jobs, of which 90 to 95 are mind-numbingly boring. It’s also seen in the distribution of language preference (and I say “preference” because there are tens of thousands of excellent programmers writing C++ and Java right now, but very few prefer them). The result of this is that the mediocre languages have the most programmers. If a company needs to hire 200 programmers per month, it simply cannot choose Haskell as its main development language; within a couple years, it would have absorbed the entire Haskell community!

Historically, that has been a serious concern for companies when it comes to language selection. Because there are hundreds of times more Java developers than Haskell or Ocaml hackers, there are at least fives of times more half-decent ones. Thus, the worst languages paradoxically have the strongest library support. This is compounded by the fact that powerful tools (such as IDEs, which are a mixed bag of neatness and horror, but sometimes quite useful and outright required when developing in Java) must be written to compensate the shortcomings of hobbled languages. There’s no Lisp IDE because emacs does just fine, but there are a slew of Java IDEs because it’s a revolting experience to write Java without one. From a non-technical CEO’s perspective, this makes Java look better because it has the best supporting tools.

What all this means is that mediocre software shops are not going to switch over to Haskell in order to ape this signaling mechanism. They can’t, because the bottom contingent of their software staff with drop like flies, and because their leadership is unlikely to understand the language selection problem in the first place. There might be some unestablished software companies using elite languages, and of them, I make the same argument that I’d make of “15-percenters” who nonetheless show interest and competence in “elite” languages– they may not be 5-percenters now, but if they keep at it, they will be shortly!

What about the (admitted) shortcomings of elite languages? Except for Scala and Clojure, which have access to the JVM and interoperate cleanly with Java, these languages don’t have the breadth of tooling that C++ and Java do. The answer to that is almost stereotypically “hackerish”: write them! This effort is not wasted, not in the least. Writing high-quality open-source tools to support elite languages is one of the best things a software company can do to establish its reputation. This, again, is an opportunity to show, not tell.

For a small company attempting to define and establish itself, attracting top talent is hard. The best programmers are on the job market so rarely and for such short intervals that attracting them takes concerted effort. Growing companies do not have the name recognition, and cannot afford the immense salaries (over $250,000 for a senior developer) that would attract these “5-percenters” using economic means, so they must win on technical grounds. Language selection is a simple (but not easy, because it’s difficult to adopt new and dramatically more powerful tools) way to do so. The best languages (Ocaml, Scala, Haskell, Python) are “shibboleths” that elite programmers use to identify each other, so adopting one of them is a one-stop choice that instantly establishes “hacker cred” or, to use Nir Eyal’s terminology, bridges “The Developer Divide”.

Santorum Children Beg Father to Plug Presidential Campaign

Penn Hills, Pennsylvania– It’s typical for national political campaigns to be hard on candidates’ families, but Rick Santorum’s six children, all of whom are currently in school and have been smeared by the negative associations with the Santorum name, have reached a unanimous decision: their father should end his runs for President. Late last night, the Santorum children held an intervention in which they entreated their father to plug up the campaign.

“That revolting neologism was on its way down the drain,” said Santorum’s 19-year-old son, John. “I was so glad to be rid of all the ‘frothy mixture’ jokes. I thought it was all washed up. Now, Dad sneaks back out and does this. How in the fuck is a college student supposed to get laid with that association to his name?”

Santorum’s presidential run has been difficult not only for the candidate’s own children, but for the 2,319 American schoolchildren sharing that surname, which, before Santorum’s political career, held no sexual connotation. The candidate’s humiliating defeat in 2006 at the hands of Bob Casey, Jr. afforded these children a reprieve from surname-derived harassment, but one that was unfortunately short-lived. According to the American Bullying Institute, students with the afflicted surname have been the butt of 135 percent more jokes since the 2011 resurgence of Santorum’s political campaign.

Asked for comment, Rick Santorum expressed faith that his children would overcome the harassment, which he expected to “dry out overnight”. In a press conference he said, on the matter, “They’re tough little buggers.”

Not all in the family have been upset by the negative attention given to Rick Santorum’s name. Elizabeth, the candidate’s 20-year-old daughter, said, “The bright side of this ‘Santorum’ muckraking is that it distracts people from what a fascistic, mean-spirited, uncultured, stupid and downright-fucking-insane bigot he is.”

#FireSOPA #FirePIPA — Vote the bastards out.

I, Michael O. Church, hereby state that in the election of 2012, I will vote against any Senatorial or House candidate who supported or voted for SOPA and PIPA during this term. Even if the opponent is a conservative Republican religious nutcase (and I assure you that I am not one) I will be voting for that person in order to oust the incumbent. No exceptions.

Please join me in voting out of office as many SOPA/PIPA supporters as we can. #FireSOPA, #FirePIPA.

A problem with the term, programming “language”

I’m going to make a radical and perhaps offensive assertion: most of the time, when programmers use the phrase “programming language”, they are using it inappropriately, and creating a dangerously fallacious analogy, one for which few programmers but many decision-makers in business fall: that programming languages can be compared and assessed as one would for natural languages.

The problem is twofold. First, natural languages don’t have the massive variations in capability observed in programming languages. French is not superior to English (or vice versa) in the way that Scala is superior to Java. Second, “switching costs” associated with acquiring a new natural language are high, because it’s hard for an adult human to learn a new natural language well: it takes years of exposure, preferably in the context of contact with native speakers. This is not as true of programming languages; learning programming is hard, but a skilled programmer can become capable in a new language in a few weeks and be proficient after months. Because of this clumsy analogy, decision-makers in the software industry often make a decision that is the obvious right one with regard to natural languages, and almost always the wrong one (because “the standard” is usually an underpowered legacy language like Java or C++) in selecting a programming language: they default to the standard.

My intention is not to discuss specific programming languages per se, but the problem inherent in the phrase “programming language”. It is a correct phrase: a programming language is a form of language, with grammar, syntax and semantics. On the other hand, it admits a certain confusion inherited from something we know about natural languages like English and Spanish: despite their evident differences, human languages are much more alike than they are different. Natural languages have their quirks and are may be more capable in specific ways– character languages are more compact, while letter languages’ smaller alphabets make it easier to convert written words into spoken form– but none is uniformly or astronomically superior to any other.

For example, there’s no natural language where to express an average English sentence (15 words) requires two hundred words, but there are languages (Scala, Python) that are 15 times more concise than Java. There’s also no language that is 100 times easier for the human brain to convert into meaningful instructions than English, but there are programming languages in which programs run 100 times faster than others. Nor is there a natural language where it’s impossible for a syntactically correct sentence (e.g. “the moon ate the moon”) to be nonsensical; this assurance, to a large degree, can be achieved in statically typed languages. While it might be offensive even to consider that natural languages might vary in “quality”, I think it quite reasonable to assert that they don’t. To my knowledge, there aren’t major, order-of-magnitude, variations in capability among natural languages. For this reason, it’s appropriate to communicate (e.g. to conduct business) in the language of which the involved parties will have the best comprehension. The right decision, in selecting a natural language, is to use “the standard”. In New York, that’s usually English. In Moscow, it’s Russian. Every locale has a small set (sometimes only one) of natural languages that are likely to be well-comprehended by most people. For this reason, it would be absurd for a Silicon Valley firm to make the decision to conduct all business in Swahili, even if Swahili were 10 or 50 percent better suited to that company’s needs.

Programming languages are different: there are order of magnitude variations. Rewrite a C program in Java, and the program becomes immune to a wide class of errors (due to Java’s automatic memory management) but one loses the ability to explicitly manage memory, and program performance becomes nondeterministic. Sometimes this is a desirable change; sometimes not. Rewrite the Java program in Python and the source code usually becomes one-tenth of its previous size, making the program easier to maintain and (in the long-run) better, but one forgoes access to the Java libraries (using Python’s, which are strong but less developed) and programs take 10 to 20 times longer to run. Rewrite the Python program in Scala and one regains access to the Java libraries and Java’s speed, but loses access to Python’s libraries and will have trouble transliterating some of Python’s (controversial, but sometimes powerful) most dynamic features.

Order-of-magnitude differences exist among programming languages, with the winners depending largely on one’s definition of “quality”. Python is expressive but the interpreter is slow. Java is fast but verbose and horrendous to read, making source code difficult to maintain and “code rot” inevitable. Scala is fast and concise, with a powerful type system, but few people understand the language or its type system well, so it’s not really possible to hire 50 Scala developers per month. C is verbose (for large-scale, complex software) and without many features, but the best or only choice for a variety of problems, such as writing device drivers, real-time programs, and operator systems. No programming language can be categorized as the uniform “best”; all have their strengths and weaknesses (except for C++, which is a bastardization of C and should never be used except to troll people).

This essay isn’t about what languages are good and which not, but it answers a question: why do most businesses use the wrong programming languages? Why are so many development shops using C++ and Java, when they could accomplish four times as much per developer-month using a more expressive language like Python or Scala? I think a major part of it comes from the fact that we call them programming languages. For an analogous question: Why do most American businesses use English? Not because English is the “best” language (if such a notion could even be defined for a natural language, and I doubt it can) but because it’s “the standard”: it’s what most other Americans use to communicate. With natural languages, there aren’t order-of-magnitude differences in capabilities, and the difficulty of learning a new natural language is high (the American “founding fathers” despised the British so much that they tried to replace English with French or Hebrew as the new country’s “official language”, and failed) so choosing the default natural language is such an obvious choice that it’s often not even a conscious decision.

Java and C++, analogously, are favored in the software industry because they’re “the standard”, like English. To a businessperson unfamiliar with technology, a proposal to implement software in Scala seems like the suggestion that all business should be conducted in Swahili– for a New York firm, ridiculous. This is why many software shops are far more skeptical of “other languages” than they would be if they understood the situation and the potential gains.

How do we dispel these notions? I think one change should be in our terminology. Often, when we debate languages, we’re actually discussing technologies built on languages. For example, sentences like “C is fast” and “Python is slow” are ridiculous as stated. A language (formally speaking) is just a set of strings of symbols, with no intrinsic concept of “speed”. What people mean, in fact, is that “compiled C programs have excellent performance due to advanced compilers and the language’s support for powerful optimizations” and “Python executables run about 10 to 20 times more slowly than analogous programs in C” (but still fast, because all computers are “fast” these days).  Speed isn’t an intrinsic trait of the language, but rather one of the technologies to compile and interpret it.  Likewise, the common justification for using Java is that “It’s has the largest set of active libraries.” Well, not exactly. There are a variety of exciting (and superior) JVM languages– most notably Scala and Clojure– that can use the bulk of these tools equally seamlessly, which means that there’s no excuse (in most cases) for preferring Java.

Programming languages are languages, but the decision to adopt one programming language over another has no similarity to such a decision over natural languages. The relevant comparison, in technology, is one of platforms, programming styles, capabilities, and supporting technologies. Referring to this decision-making process as a “language debate” is, in this context, myopic. Far more is at stake than the choice of “language” alone; the chosen programming language influences the types of programs one can write, and the variation in program quality that is staked on language selection alone is quite high.