Blub vs. engineer empowerment

No, I’m not quitting the Gervais / MacLeod Series. Part 23, which will actually be the final one because I want to get back to technology in how I spend spare time, is half-done. However, I am going to take a break in it to write about something else. 

I’ve written about my distaste for language and framework wars, at least when held for their own sake. I’m not fading from my position on that. If you tell go off and tell someone that her favorite language is a U+1F4A9 because it’s (statically|dynamically) typed, then you’re just being a jerk. There are a few terrible languages out there (especially most corporate internal DSLs) but C, Python, Scala, Lisp and Haskell were all designed by very smart people and they all have their places. I’ve seen enough to know that. There isn’t one language to rule them all. Trust me.

Yet, I contend that there is a problem of Blub in our industry. What’s Blub? Well, it’s often used as an epithet for an inferior language, coined in this essay by Paul Graham. As tiring as language wars are, Blubness is real. I contend, however, that it’s not only about the language. There’s much more to Blub.

Let’s start with the original essay and use Graham’s description of Blub:

Programmers get very attached to their favorite languages, and I don’t want to hurt anyone’s feelings, so to explain this point I’m going to use a hypothetical language called Blub. Blub falls right in the middle of the abstractness continuum. It is not the most powerful language, but it is more powerful than Cobol or machine language.

And in fact, our hypothetical Blub programmer wouldn’t use either of them. Of course he wouldn’t program in machine language. That’s what compilers are for. And as for Cobol, he doesn’t know how anyone can get anything done with it. It doesn’t even have x (Blub feature of your choice).

As long as our hypothetical Blub programmer is looking down the power continuum, he knows he’s looking down. Languages less powerful than Blub are obviously less powerful, because they’re missing some feature he’s used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn’t realize he’s looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.

When we switch to the point of view of a programmer using any of the languages higher up the power continuum, however, we find that he in turn looks down upon Blub. How can you get anything done in Blub? It doesn’t even have y.

By induction, the only programmers in a position to see all the differences in power between the various languages are those who understand the most powerful one. (This is probably what Eric Raymond meant about Lisp making you a better programmer.) You can’t trust the opinions of the others, because of the Blub paradox: they’re satisfied with whatever language they happen to use, because it dictates the way they think about programs.

So what is Blub? Well, some might read that description and say that it sounds like Java (has garbage collection, but not lambdas). So is Java Blub? Well, not quite. Sometimes (although rarely) Java is the right language to use. As a general-purpose language, Java is a terrible choice; but for high-performance Android development, Java’s the best. It is not James Gosling’s fault that it became the go-to language for clueless corporate managers and a tool-of-choice for mediocre “commodity developers”. That fact may or may not be related to weaknesses of the language, but it doesn’t make the language itself inferior.

Paul Graham looks at languages from a language-designer’s viewpoint, and also with an emphasis on aesthetics. As an amateur painter whose original passion was art, that shouldn’t surprise us. And in my opinion, Lisp is the closest thing out there to an aesthetically beautiful language. (You get used to the parentheses. Trust me. You start to like them because they are invisible when you don’t want to see them, but highlight structure when you do.) Does this mean that it’s right for everything? Of course not. If nothing else, there are cases when you don’t want to be working in a garbage-collected language, or when performance requirements make C the only game in town. Paul Graham seems to be focused on level of abstraction, and equating the middle territory (Java and C# would take that ground, today) with mediocrity. Is that a fair view?

Well, the low and high ends of the language-power spectrum tend to harbor a lot of great programmers, while the mediocre developers tend to be Java (or C#, or VB) monoglots. Good engineers are not afraid to go close to the metal, or far away from it into design-your-own-language land, if the problem calls for it. They’re comfortable in the whole space, so you’re more likely to find great people at the fringes. Those guys who write low-latency trading algorithms that run on GPUs have no time to hear about “POJOs“, and the gals who blow your mind with elegant Lisp macros have no taste for SingletonVisitorFactories. That said, great programmers will also operate at middling levels of abstraction when that is the right thing to do.

The problem of Blubness isn’t about a single language or level of abstraction. Sometimes, the C++/Java level of abstraction sometimes is the right one to work at. So there certainly are good programmers using those languages. Quite a large number of them, in fact. I worked at Google, so I met plenty of good programming using these generally unloved languages.

IDEs are another hot topic in the 10xers-versus-commodity-engineers flamewar. I have mixed feelings about them. When I see a 22-year-old settling in to his first corporate job and having to use the mouse, that “how the other half programs” instinct flares up and I feel compelled to tell him that, yes, you can still write code using emacs and the command line. My honest appraisal of IDEs? They’re a useful tool, sometimes. With the right configuration, they can be pretty neat. My issue with them is that they tend to be symptomatic. IDEs really shine when you have to read large amounts of other peoples’ poorly-written code. Now, I would rather have an IDE to do than not have one (trust me; I’ve gone both ways on that) but I would really prefer a job that didn’t involve trudging though bad legacy code on a daily basis. When someone tells me that “you have to use an IDE around here” I take it as a bad sign, because it means the code quality is devastatingly bad, and the IDE’s benefit will be to reduce Bad Code’s consumption of my time from 98% to 90%– still unacceptable.

What do IDEs have to do with Blub? Well, IDEs seem to be used often to support Blubby development practices. They make XML and Maven slightly less hideous, and code navigation (a valuable feature, no disagreement) can compensate, for a little while, for bad management practices that result in low code quality. I don’t think that IDEs are inherently bad, but I’ve seen them take the most hold in environments of damaged legacy code and low engineer empowerment.

I’ve thought a lot about language design and languages. I’ve used several. I’ve been in a number of corporate environments. I’ve seen good languages turn bad and bad languages become almost tolerable. I’ve seen the whole spectrum of code quality. I’ve concluded that it’s not generally useful to yell at people about their choices of languages. You won’t change, nor will they, and I’d rather work with good code in less-favored languages than bad code in any language. Let’s focus on what’s really at stake. Blub is not a specific language, but it is a common enemy: engineer disempowerment.

As technologists, we’re inclined toward hyperrationality, so we often ignore people problems and mask them up as technical ones. Instead of admitting that our company hired a bunch of terrible programmers who refuse to improve, we blame Java, as if the language itself (rather than years of terrible management, shitty projects, and nonexistent mentorship) somehow jammed their brains. Well, that doesn’t make sense because not every Java programmer is brain damaged. When something goes to shit in production, people jump to the conclusion that it wouldn’t have happened in a statically-typed language. Sorry, but that’s not true. Things break in horrible ways in all kinds of languages. Or, alternatively, when development is so slow that every top-25% engineer quits, people argue that it wouldn’t have happened in a fast-prototyping, dynamically-typed language. Wrong again. Bad management is the problem, not Scala or Python or even Java.

Even terrible code isn’t deserving of the anger that’s directed at it. Hell, I’ve written terrible code, especially early in my career. Who hasn’t? That anger should be directed against the manager who is making the engineer use shitty code (because the person who wrote it is the manager’s favorite) and not at the code itself. Terrible romance novels are written every day, but they don’t anger me because I never read them. But if I were forced to read Danielle Steele novels for 8 hours per day, I would fucking explode.

Ok, that’s enough negativity for a while…

I had a bit of a crisis recently. I enjoy computer science and I love solving hard problems. I enjoy programming. That said, the software industry has been wearing me down, this past couple of years. The bad code, low autonomy, and lack of respect for what we do is appalling. We have the potential to add millions of dollars per year in economic value, but we tend to get stuck with fourth quadrant work that we lack the power to refuse. I’ve seen enough of startups to know that most of them aren’t any better. The majority of those so-called “tech startups” are marketing experiments that happen to involve technology because, in the 21st century, everything does. I recently got to a point where I was considering leaving software for good. Computer science is fine and I have no problem with coding, but the corporate shit (again, just as bad in many startups) fries the brain and weakens the soul.

For some positivity, I went to the New York Clojure Meetup last night. I’ve been to a lot of technology Meetups, but there was a distinct feel at that one. The energy was more positive than what I’ve seen in many technical gatherings. The crowd was very strong, but that’s true of many technical meetups. Here, there was a flavor of “cleaner burning” in addition to the high intelligence that is always the case at technology meetups. People weren’t touting one corporate technology at the expense of another, and there was real code– good code, in fact– in a couple of the presentations. The quality of discussion was high, in addition to the quality of the people.

I’d had this observation, before, about certain language communities and how the differences of those are much greater than differences in language. People who intend to be lifelong programmers aren’t happy having New Java Despondency Infarction Framework X thrown at them every two years by some process-touting manager. They want more. They want a language that actually improves understanding of deep principles pertaining to how humans solve problems. It’s not that functional programming is inherently and universally superior. Pure functional programming has strong merits, and is often the right approach (and sometimes not) but most of what makes FP great is the community it has generated. It’s a community of engineers who want to be lifelong programmers or scientists, and who are used to firing up a REPL and trying out a new library. It’s a community of people who still use the command line and who still believe that to program is a virtue. The object-oriented world is one in which every programmer wants to be a manager, because object-orientation is how “big picture guys” think.

I’m very impressed with Clojure as a language, and that community has made phenomenally good decisions over the past few years. I started using it in 2008, and the evolution has been very positive. It’s not that I find Clojure (or Lisp) to be inerrant, but the community (and some others, like Haskell’s) stands in stark contrast against the anti-intellectualism of corporate software development. And I admire that immensely. It’s a real sacrifice that we 1.5+ engineers make on an ongoing basis when we demand that we keep learning, do things right, and build on sound principles. It doesn’t come easy. It can demand unusual hours, costs us jobs, and can put us in the ghetto, but there it is.

In the mean time, though, I don’t think it’s useful to mistake language choice as the prevailing or most important issue. If we do that, we’re just as guilty of cargo cultism as the stereotypical Java-happy IT managers. No, the real issue that matters is engineer empowerment, and we need to keep up our culture around that.

Six languages to master.

Eric Raymond, in “How to Become a Hacker“, recommended five languages: C, Java, Python, Perl, and Lisp. Each he recommended for different reasons: Python and Java as introductory languages, C to understand and hack Unix, Perl because of its use in scripting, and Lisp for, to use his words which are so much better than anything I can come up with, the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot.

It’s 2012. Many languages have come to the fore that didn’t exist when this essay was written. Others, like Perl, have faded somewhat. What is today’s five-language list? I won’t pretend that my answer is necessarily the best; it’s biased based on what I know. That said, I’d think the 5 highest-return languages for people who want to become good engineers are the following, and in this order: Python, C, ML, Clojure, and Scala.

Why these 5? Python I include because it’s easy to learn and, in-the-small, extremely legible. I’d rather not use it for a large system, but people who are just now learning to program are not going to be writing huge systems. They’re not going to be pushing major programs into production. At least, they shouldn’t be. What they should be able to do is scrape a webpage or build a game or investigate an applied math problem and say, “Shit, that’s cool.” Python gets people there quickly. That will motivate them to get deeper into programming. Python is also a language that is not great at many things, but good at one hell of a lot of them. It’s quick to write, legible in the small, and expressive. It allows imperative and functional styles. It has great libraries, and it has strong C bindings, for when performance is needed.

People who are getting started in programming want to do things that are macroscopically interesting from a beginner’s perspective. They don’t just want to learn about algorithms and how compilers work, because none of that’s interesting to them until they learn more of the computer science that motivates the understanding of why these things are important. Compilers aren’t interesting until you’ve written programs in compiled languages. At the start, people want to be able to write games, scrape webpages, and do simple systems tasks that come up. Python is good because it’s relatively easy to do most programming tasks in it.

After Python, C is a good next choice, and not because of its performance. That’s largely irrelevant to whether it will make someone a better programmer (although the confidence, with regard to understanding performance, that can come with knowing C is quite valuable). C is crucial because there’s a lot of computer science that becomes inaccessible if one sticks to higher-level languages (and virtual machines) like Java, C#, and Python. Garbage collection is great, but what is the garbage collector written in? C. As is Unix, notably. For all this, I think C is a better choice than C++ because there’s another important thing about C: C++ is a mess and it’s not clear whether it’s a good language for more than 1% of the purposes to which it’s put, but C, on the other hand, has utterly dominated the mid-level language category. For all its flaws, C is (like SQL for database query languages) a smashing, undisputed success, and for good reasons. The high-level language space is still unsettled, with no clear set of winners, but the mid-level language used to write the runtimes and garbage collectors of those high level languages is usually C, and will be for some time.

Python and C give a person coverage of the mid- and very-high levels of language abstraction. I’m avoiding including low-level (i.e. assembly) languages because I don’t think any of them have the generalist’s interest that would justify top-5 placement. Familiarity with assembly language and how it basically works is a must, but I don’t think mastery of x86 intricacies is necessary for most programmers.

Once a programmer’s fluent in Python and C, we’re talking about someone who can solve most coding problems, but improvement shouldn’t end there. Taste is extremely important, and it’s lack of taste rather than lack of intellectual ability that has created the abundance of terrible code in existence. Languages can’t inherently force people to learn taste, but a good starting point in this direction is ML: SML or OCaml with the “O” mostly not used.

ML has been described as a “functional C” for its elegance. It’s fast, and it’s a simple language, but its strong functional programming support makes it extremely powerful. It also forces people to program from the bottom up. Instead of creating vague “objects” that might be hacked into bloated nonsense over the lifespan of a codebase, they create datatypes (mostly, records and discriminated unions, with parameterized types available for polymorphism) out of simpler ones, and use referentially transparent functions as the basic building blocks of most of their programs. This bottom-up structure forces people to build programs on sound principles (rather than the vague, squishy abstractions of badly-written object-oriented code) but ML’s high-level capability brings people into the awareness that one can write complex software using a bottom-up philosophy. Python and C teach computer science at higher and lower levels, but ML forces a programmer to learn how to write good code.

There’s also something philosophical that Python, C, and Ocaml tend to share that C++ and Java don’t: small-program philosophy, which is generally superior. I’ve written at length about the perils of the alternative. In these languages, it’s much more uncommon to drag in the rats’ nest of dependencies associated with large Java projects. For an added bonus, you never have to look at those fucking ugly singleton directories called “com”. Once a person has used these three languages to a significant extent, one gets a strong sense of how small-program development works and why immodular, large-project orientation is generally a bad thing.

When you write C or Ocaml or Python, you get used to writing whole programs that accomplish something. There’s a problem, you solve it, and you’re done. Now you have a script, or a library, or a long-running executable. You may come back to it to improve it, but in general, you move on to something else, while the solution you’ve created adds to the total stored value of your code repository. That’s what’s great about small-program development: problems are actually solved and work is actually “done” rather than recursively leading to more work without any introspection on whether the features being piled on the work queue make sense. Developers who only experience large-program development– working on huge, immodular Java projects in IDEs a million metaphorical miles from where the code actually runs for real– never get this experience of actually finishing a whole program.

Once a person has grasped ML, we’re talking about a seriously capable programmer, even though ML isn’t a complex language. Learned in the middle of one’s ML career is a point to which I’ll return soon, but for now leave hanging: types are interesting. One of the most important things to learn from ML is how to use the type system to enforce program correctness: it generates a massive suite of implicit unit tests that (a) never have to be written, and (b) don’t contribute to codebase size. (Any decent programmer knows that “lines of code” represent expenditure, not accomplishment.)

The fourth language to learn is Clojure, a Lisp that happens to run on the JVM. The JVM has its warts, but it’s powerful and there are a lot of good reasons to learn that ecosystem, and Clojure’s a great entry point. A lot of exciting work is being done in the JVM ecosystem, and languages like Clojure and Scala keep some excellent programmers interested in it. Clojure is an excellent Lisp, but with its interactive “repl” (read-eval-print-loop) and extremely strong expressivity, it is (ironically)  arguably the best way to learn Java. It has an outstanding community, a strong feature set, and some excellent code in the open-source world.

Lisp is also of strong fundamental importance, because its macro system is unlike anything else in any other language and will fundamentally alter how an engineer thinks about software, and because Lisp encourages people to use a very expressive style. It’s also an extremely productive language: large amounts of functionality can be delivered in a small amount of time. Lisp is a great language for learning the fundamentals of computing, and that’s one reason why Scheme has been traditionally used in education. (However, I’d probably advocate starting with Python because it’s easier to get to “real stuff” quickly in it. Structure and Interpretation of Computer Programs and Scheme should be presented when people know they’re actually interested in computing itself.)

When one’s writing large systems, Lisp isn’t the best choice, because interfaces matter at that point, and there’s a danger that people will play fast-and-loose with interfaces (passing nested maps and lists and expecting the other side to understand the encoding) in a way that can be toxic. Lisp is great if you trust the developers working on the project, but (sadly) I don’t think many companies remain in such a state as they grow to scale.

Also, static typing is a feature, not a drawback. Used correctly, static typing can make code more clear (by specifying interfaces) and more robust, in addition to the faster performance usually available in compiled, statically typed languages. ML and Haskell (which I didn’t list, but it’s a great language in its own right) can teach a person how to use static typing well.

So after Lisp, the 5th language to master is Scala. Why Scala, after learning all those others, and having more than enough tools to program in interesting ways? First of all, it has an incredible amount of depth in its type system, which attempts to unify the philosophies of ML and Java and (in my opinion) does a damn impressive job. The first half of Types and Programming Languages is, roughly speaking, the theoretic substrate for ML. But ML doesn’t have a lot of the finer features. It doesn’t have subtyping, for example. Also, the uniqueness constraint on record and discriminated union labels (necessary for full Hindley-Milner inference, but still painful) can have a negative effect on the way people write code. The second half of TAPL, which vanilla ML doesn’t really support, is realized in Scala. Second, I think Scala is the language that will salvage the 5 percent of object-oriented programming that is actually useful and interesting, while providing such powerful functional features that the remaining 95% can be sloughed away. The salvage project in which a generation of elite programmers selects what works from a variety of programming styles– functional, object-oriented, actor-driven, imperative– and discards what doesn’t work, is going to happen in Scala. So this is a great opportunity to see first-hand what works in language design and what doesn’t.

Scala’s a great language that also requires taste and care, because it’s so powerful. I don’t agree with the detractors who claim it’s at risk of turning into C++, but it definitely provides enough rope for a person to hang himself by the monads.

What’s most impressive about Clojure and Scala is their communities. An enormous amount of innovation, not only in libraries but also in language design, is coming out of these two languages. There is a slight danger of Java-culture creep in them, and Scala best practices (expected by the leading build environments) do, to my chagrin, involve directories called “src” and “main” and even seem to encourage singleton directories called “com”, but I’m willing to call this a superficial loss and, otherwise, the right side seems to be winning. There’s an incredible amount of innovation happening in these two languages that have now absorbed the bulk of the top Java developers.

Now… I mentioned “six languages” in this post’s title but named five. The sixth is one that very few programmers are willing to use in source code: English. (Or, I should say, the scientifically favored natural language of one’s locale.) Specifically, technical English, which requires rigor as well as clarity and taste. Written communication. This is more important than all of the others. By far. For that, I’m not complaining that software engineers are bad at writing. Competence is not a problem. Anyone smart enough to learn C++ or the finer points of Lisp is more than intelligent enough to communicate in a reasonable way. I’m not asking people to write prose that would make Faulkner cry; I’m asking them to explain the technical assets they’ve created at, at the least, the level of depth and rigor expected in a B+ undergraduate paper. The lack of writing in software isn’t an issue of capability, though, but of laziness.

Here’s one you hear sometimes: “The code is self-documenting.” Bullshit. It’s great when code can be self-documenting, making comments unnecessary, but it’s pretty damn rare to be solving a problem so simple that the code responsible for solving it is actually self-explanatory. Most problems are custom problems that require documentation of what is being solved, why, and how. People need to know, when they read code, what they’re looking at; otherwise, they’re going to waste a massive amount of time focusing on details that aren’t relevant. Documentation should not be made a crutch– you should also do the other important things like avoiding long functions and huge classes– but it is essential to write about what you’re doing. People need to stop thinking about software as machinery that “explains itself” and start thinking of it as writing a paper, with instructions for humans about what is happening alongside the code actually doing the work.

One of the biggest errors I encounter with regard to commenting is the tendency to comment minutiae while ignoring the big picture. There might be a 900-line program with a couple comments saying, “I’m doing this it’s O(n) instead of O(n^2)” or “TODO: remove hard-coded filename”, but nothing that actually explains why these 900 lines of code exist. Who does that help? These types of comments are useless to people who don’t understand what’s happening at all, which they generally won’t in the face of inadequate documentation. Code is much easier to read when one knows what one is looking at, and microcomments on tiny details that seemed important when the code was written are not helpful.

Comments are like static typing: under-regarded if not ill-appreciated because so few people use them properly, but very powerful (if used with taste) in making code and systems actually legible and reusable. Most real-world code, unfortunately, isn’t this way. My experience is that about 5 to 10 percent of code in a typical codebase is legible, and quite possibly only 1 percent is enjoyable to read (which good code truly is). The purpose of a comment should not be only to explain minutiae or justify weird-looking code. Comments should also ensure that people always know what they’re actually looking at.

The fallacy that leads to a lack of respect for documentation is that writing code is like building a car or some other well-understood mechanical system. Cars don’t come with a bunch of labels on all the pieces, because cars are fairly similar under the hood and a decent mechanic can figure out what is what. With software, it’s different. Software exists to solve a new problem; if it were solving an old problem, old software could be used. Thus, no two software solutions are going to be the same. In fact, programs tend to be radically different from one another. Software needs to be documented because every software project is inherently different, at least in some respects, from all the others.

There’s another problem, and it’s deep. The 1990s saw an effort, starting with Microsoft’s visual studio, to commoditize programmers. The vision was that, instead of programming being a province of highly-paid, elite specialists with a history of not working well with authority, software could be built by bolting together huge teams of mediocre, “commodity” developers, and directing them using traditional (i.e. pre-Cambrian) management techniques. This has begun to fail, but not before hijacking object-oriented programming, turning Java’s culture poisonous, and creating some of the most horrendous spaghetti code (MudballVisitorFactoryFactory) the world has ever seen. Incidentally, Microsoft is now doing a penance by having its elite research division investigate functional programming in a major way, the results being F# and a much-improved C#. Microsoft, on the whole, may be doomed to mediocrity, but they clearly have a research division that “gets it” in an impressive way. Still, that strikes me as too little, too late. The damage has be done, and the legacy of the commodity-developer apocalypse still sticks around.

The result of the commodity-programmer world is the write-only code culture that is the major flaw of siloized, large-program development. That, I think, is the fundamental problem with Java-the-culture, IDE-reliance, and the general lack of curiosity observed (and encouraged) among the bottom 80 percent of programmers. To improve as programmers, people need to read code and understand it, in order to get a sense of what good and bad code even are, but almost no one actually reads code anymore. IDEs take care of that. I’m not going to bash IDEs too hard, because they’re pretty much essential if you’re going to read a typical Java codebase, but IDE culture is, on the whole, a major fail that makes borderline-employable programmers out of people who never should have gotten in in the first place.

Another problem with IDE culture is that the environment becomes extremely high maintenance, between plugins that often don’t work well together, build system idiosyncracies that accumulate over time, and the various menu-navigation chores necessary to keep the environment sane (as opposed to command-line chores, which are easily automated). Yes, IDEs do the job: bad code becomes navigable, and commodity developers (who are terrified of the command line and would prefer not to know what “build systems” or “version control” even are) can crank out a few thousand lines of code per year. However, the high-maintenance environment requires a lot of setup work, and I think this is culturally poisonous. Why? For a contrast, in the command-line world, you solve your own problems. You figure out how to download software (at the command line using wget, not clicking a button) and install it. Maybe it takes a day to figure out how to set up your environment, but once you’ve suffered through this, you actually know a few things (and you usually learn cool things orthogonal to the problem you were originally trying to solve). When a task gets repetitive, you figure out how to automate it. You write a script. That’s great. People actually learn about the systems they’re using. On the other hand, in IDE-culture, you don’t solve your own problems because you can’t, because there it would take too long. In the big-program world, software too complex for people to solve their own problems is allowed to exist. Instead of figuring it out on your own, you flag someone down who understands the damn thing, or you take a screenshot of the indecipherable error box that popped up and send it to your support team. This is probably economically efficient from a corporate perspective, but it doesn’t help people become better programmers over time.

IDE culture also creates a class of programmers who don’t work with technology outside of the office– the archetypal “5:01 developers”– because they get the idea that writing code requires an IDE (worse yet, an IDE tuned exactly in line with the customs of their work environment). If you’re IDE-dependent, you can’t write code outside of a corporate environment, because when you go home, you don’t have a huge support team to set the damn thing up in a way that you’re used to and fix things when the 22 plugins and dependencies that you’ve installed interact badly.

There are a lot of things wrong with IDE culture, and I’ve only scratched the surface, but the enabling of write-only code creation is a major sticking point. I won’t pretend that bad code began with IDEs because that’s almost certainly not true. I will say that the software industry is in a vicious cycle, which the commodity-developer initiative exacerbated. Because most codebases are terrible, people don’t read them. Because “no one reads code anymore”, the bulk of engineers never get better, and continue to write bad code.

Software has gone through a few phases of what it means for code to actually be “turned in” as acceptable work. Phase 1 is when a company decides that it’s no longer acceptable to horde personal codebases (that might not even be backed up!) and mandates that people check their work into version control. Thankfully, almost all companies have reached that stage of development. Version control is no longer seen as “subversive” by typical corporate upper management. It’s now typical. The second is when a company mandates that code have unit tests before it can be relied upon, and that a coding project isn’t done until it has tests. Companies are reaching this conclusion. The third milestone for code-civilizational development, which very few companies have reached, is that the code isn’t done until you’ve taught users how to use it (and how to interact with it, i.e. instantiate the program and run it or send messages to it, in a read-eval-print-loop appropriate to the language). That teaching can be supplied at a higher level in wikis, codelabs, and courses… but it also needs to be included with the source code. Otherwise, it’s code out-of-context, which becomes illegible after a hundred thousand lines or so. Even if the code is otherwise good, out-of-context code without clear entry-points and big-picture documentation becomes incomprehensible around this point.

What do I not recommend? There’s no language that I’d say is categorically not worth learning, but I do not recommend becoming immersed in Java (except well enough to understand the innards of Clojure and Scala). The language is inexpressive, but the problem isn’t the language, and in fact I’d say that it’s unambiguously a good thing for an engineer to learn how the JVM works. It’s that Java-the-Culture (VisitorSelectionFactories, pointless premature abstraction, singleton directories called “com” that betray dripping contempt for the command line and the Unix philosophy, and build environments so borked that it’s impossible not to rely on an IDE) that is the problem; it’s so toxic that it reduces an engineer’s IQ by 2 points per month.

For each of these five programming languages, I’d say that a year of exposure is ideal and probably, for getting a generalist knowledge, enough– although it takes more than a year to actually master any of these. Use and improvement of written communication, on the other hand, deserves more. That’s a lifelong process, and far too important for a person not to start early on. Learning new programming languages and, through this, new ways of solving problems, is important; but the ability to communicate what problem one has solved is paramount.

Why you can’t hire good Java developers.

Before I begin, the title of my essay deserves explanation. I am not saying, “There are no good Java developers.” That would be inflammatory and false. Nor am I saying it’s impossible to hire one, or even three, strong Java developers for an especially compelling project. What I will say is this: in the long or even medium term, if the house language is Java, it becomes nearly impossible to establish a hiring process that reliably pulls in strong developers with the very-low false-positive rates (ideally, below 5%) that technology companies require.

What I won’t discuss (at least, not at length) are the difficulties in attracting good developers for Java positions, although those are significant, because most skilled software developers have been exposed to a number of programming languages and Java rarely emerges as the favorite. That’s an issue for another post. In spite of that problem, Google and the leading investment banks have the resources necessary to bring top talent to work with uninspiring tools, and one willing to compete with them on compensation will find this difficulty surmountable. Nor will I discuss why top developers find Java uninspiring and tedious; that also deserves its own post (or five). So I’ll assume, for simplicity, that attracting top developers is not a problem for the reader, and focus on the difficulties Java creates in selecting them.

In building a technology team, false positives (in hiring) are considered almost intolerable. If 1 unit represents the contribution of a median engineer, the productivity of the best engineers is 5 to 20 units, and that of the worst can be -10 to -50 (in part, because the frankly incompetent absorb the time and morale of the best developers). In computer programming, making a bad hire (and I mean a merely incompetent one, not even a malicious or unethical person) isn’t a minor mistake as it is in most fields. Rather, a bad hire can derail a project and, for small businesses, sink a company. For this reason, technical interviews at leading companies tend to be very intensive. A typical technology company will use a phone screen as a filter (a surprising percentage of people with impressive CVs can’t think mathematically or solve problems in code, and phone screens shut them out) followed by a code sample, and, after this, an all-day in-office interview involving design questions, assessment of “fit” and personality, and quick problem-solving questions. “White board” coding questions may be used, but those are generally less intensive (due to time constraints) than even the smallest “real world” coding tasks. Those tend to fall closer to the general-intelligence/”on-your-feet” problem-solving questions than to coding challenges.

For this reason, a code sample is essential in a software company’s hiring process. It can come from an open-source effort, a personal “side project”, or even a (contrived) engineering challenge. It will generally be between 100 and 500 lines of code (any more than 500 can’t be read in one sitting by most people).  The code’s greater purpose is irrelevant– but the scope of the sample must be sufficient to determine whether the person writes quality code “in the large” as well as for small projects. Does the person have architectural sense, or use brute-force inelegant solutions that will be impossible for others to maintain? Without the code sample, a non-negligible false-positive rate (somewhere around 5 to 10%, in my experience) is inevitable.

This is where Java fails: the code sample. With 200 lines of Python or Scala code, it’s generally quite easy to tell how skilled a developer is and to get a general sense of his architectural ability, because 200 lines of code in these languages can express substantial functionality. With Java, that’s not the case: a 200-line code sample (barely enough to solve a “toy” problem) provides absolutely no information about whether a job candidate will solve problems in an infrastructurally sound way, or will instead create the next generation’s legacy horrors. The reasons for this are as follows. First, Java is tediously verbose, which means that 200 lines of code in it contain as much information as 20-50 lines of code in a more expressive language. There just isn’t much there there. Second, in Java, bad and good code look pretty much the same: one actually has to read an implementation of “the Visitor pattern” for detail to know if it was used correctly and soundly. Third, Java’s “everything is a class” ideology means that people don’t write programs but classes, and that even mid-sized Java programs are, in fact, domain-specific languages (DSLs)– usually promiscuously strewn about the file system because of Java’s congruence requirements around class and package names. Most Java developers solve larger problems by creating utterly terrible DSLs, but this breakdown behavior simply doesn’t show up on the scale of a typical code sample (at most, 500 lines of code).

The result of all this is that it’s economically infeasible to separate good and bad Java developers based on their code. White-board problems? Code samples? Not enough signal, if the language is Java. CVs? Even less signal there. The result is that any Java shop is going to have to filter on something other than coding ability (usually, the learned skill of passing interviews). In finance, that filter is general intelligence as measured by “brainteaser” interviews. The problem here is that general intelligence, although important, does not guarantee that someone can write decent software. So that approach works for financial employers because they have uses (trading and “quant” research) for high-IQ people who can’t code, but not for typical technology companies that rely on a uniformly high quality in the software they create.

Java’s verbosity makes the most critical aspect of software hiring– reading the candidates’ code not only for correctness (which can be checked automatically) but architectural quality– impossible unless one is willing to dedicate immense and precious resources (the time of the best engineers) to the problem, and to request very large (1000+ lines of code) code samples. So for Java positions, this just isn’t done– it can’t be done. This is to the advantage of incompetent Java developers, who with practice at “white-boarding” can sneak into elite software companies, but to the deep disadvantage of companies that use the language.

Of course, strong Java engineers exist, and it’s possible to hire a few. One might even get lucky and hire seven or eight great Java engineers before bringing on the first dud. Stranger things have happened. But establishing a robust and reliable hiring process requires that candidate code be read for quality before a decision is made. In a verbose language like Java, it’s not economical (few companies can afford to dedicate 25+ percent of engineering time to reading job candidates’ code samples) and therefore, it rarely happens. This makes an uncomfortably high false-positive rate, in the long term, inevitable when hiring for Java positions.

How any software company can cross “The Developer Divide”

Nir Eyal wrote a blog post, The Developer Divide: When Great Companies Can’t Hire, on this conundrum: there are a lot of excellent technology companies that haven’t managed to attract the brightest (and generally pickiest) engineers in sufficient numbers to hire as fast as they grow, a problem that forces a company either to lower its hiring standards or curtail growth, neither of which is desirable.

What makes this problem especially hard is that it’s not about money. In marketing terminology, it’s a problem of “reach”. Many great startups, even if they offered $200,000 per year, would simply be unable to hire 25 elite (top 5%) programmers in a year. Finding one per quarter is pretty good. Compounding this difficulty is the fact that the best software engineers’ job searches are infrequent and short. They usually find jobs through social connections and word-of-mouth before they start officially “looking”. They almost never cold-spam their CVs. Therefore, a company that’s consistently hiring top-5% programmers is probably extending offers to 1 applicant per several hundred CVs that makes its way to the hiring manager.

What is the Developer Divide? It comes down to this: it’s easier to get top developers if your product is something that top developers already use. Google was unequivocally the best search engine even in 2000, and nerds like new search engines, so it established itself as a desirable place to work. Facebook and Foursquare may be targeted toward “non-engineers” (i.e. mass market) but the products are used by enough of the people the firms want to hire as to generate name recognition that accumulates faster than the company needs to grow. That makes it very easy for them to attract so much talent they turn away more 5-percenters than they hire. But for a company whose product isn’t used already, every day, by top programmers, it becomes harder. Much harder. That’s unfortunate, because a lot of great businesses
(most, actually) need to start out doing something lower on the established-product-sexiness scale (enterprise work and products with well-defined markets, rather than speculative “social” projects) before branching out into projects of more general interest. These companies might become sexy in the future, but sometimes the first clientele has to be an unsexy but reliable one, like large corporations or suburban soccer moms.

So, how do these great companies continue to find great talent? I’m going to state one simple (but not easy) thing that any software company can do to increase its attractiveness to top engineers by at least 3 binary orders of magnitude. Here it is: ditch Java/C++ and use a decent programming language.

What’s a decent programming language, for this purpose? It’s one that a 5-percenter would use for a personal side project, or if she were calling the shots. That language might Scala, Python, or Ocaml. It might (for a project such as an operating system) be C. It would amost certainly not, in 2012, be Java or C++.

A clarification must be made, because it’s confusing to people outside of technology: C++ is not a substitute, nor  an improvement on, C. In fact, C is great for programs where low-level concerns are critical in producing a quality system: operating systems, device drivers, hard real-time, and runtime environments for the high-level garbage-collected languages we all love (such as Ocaml and Haskell). Much of the world is built on C. It’s an excellent and immensely successful mid-level language,  as opposed to C++ which is a miserably failed attempt at a C-like high-level language. So do not confuse the two.

If you ask a 5-percenter for his opinions on C and C++, he’ll probably praise the former while trashing the latter. From a non-technical business perspective, this might seem inconsistent, given that C is a proper subset of C++. “Anything you can do in C, you can do in C++.” That makes C++ “strictly better”, right? Well, no. A 5-percenter has the professional maturity to realize that he or she will not be coding in a vacuum, and that reading code is as important an act as writing it, and that therefore adding ill-considered features to a good language doesn’t improve it, but ruins it if people are stupid enough to actually use them.

I’ll stop talking about C++, because it’s not even relevant to most startups. Startups can’t afford it, because they need to accomplish big things with small teams and C++ doesn’t make it possible. C++ is mostly that skulking monster that lives in the bowels of legacy systems at banks, something you expect to fight in the 2300 AD (post-apocalyptic) world of Chrono Trigger.

Java’s problem is somewhat different and new. C++ is a bad language because it was poorly designed, but it was at least designed for good programmers (and it’s disastrous when used by inept ones). Java, in contrast, was explicitly designed to favor the interests of massive (100+) teams of mediocre developers over excellent individual contributors, whom it slows down to a small fraction of their typical speed. It came out of a failed experiment to create a home for low-skill “commodity” developers who haven’t learned a new skill since college and who need to consult the One Smart Person on the team if (God forbid) their RAM-munching IDE breaks. Java has succeeding in making commodity developers marginally effective (as opposed to negatively productive, as they are in more powerful languages) but at the expense of hobbling the best, forcing them to endure ugliness (inherent to a language designed to be used exclusively through IDEs, while 5-percenters overwhelmingly prefer the command-line interface and “classic” editors like vim and emacs) and accidental complexity. Needless to say, 5-percenters despise Java. More correctly, they despise the Java language.  (I emphasize “the language” because the Java Virtual Machine itself is a pretty powerful tool, and because there are superior languages– Scala and Clojure coming to mind– that also run on the JVM.)

If you want to build a large team of 50+ “commodity” developers, content to maintain legacy code or work on mind-numbingly boring stuff, use Java or C++. If you’re a startup, you need to build a small team of excellent developers, so use something else. If you want to hire 5-percenters now but might need to hire Java jockeys later, strongly consider Scala and Clojure, which are highly powerful languages but run in the JVM environment.

Why is this so important? It’s not just about the language. It’s about signaling. As a startup, you have to show that you get it, and that the opportunity you offer isn’t Yet Another Java Job. You can get 5-percenters to use C++ or Java (Google has made this happen, and so have many investment banks, which have huge legacy codebases in C++) but you pretty much have to be a big-name company to pull this off, and you can expect to shell out an obscene of money– a 50-100 percent markup to account for the negatives of maintaining code in a terrible language, and the career stagnation this kind of work invites. If you want a 5-percenter to work 60 hours per week for $100,000 per year, use a great language. If you want him to work 9-to-5, with two-hour lunches and personal errands deducted from the workday, for $250,000 per year, then Java and C++ are options.

More than anything else, 5-percenters want to work with other 5-percenters. This is far more important to them than prestige or money (the reason 5-percenters default to rich, prestigious companies when nothing more interesting crosses their transom is because these companies have other 5-percenters). It is, moreover, even more important than programming language selection itself. Language selection, as I’ve said, an objective mechanism of signaling. This is what creative writing calls the “show, don’t tell” principle (don’t say “Eric was honorable”; have him do something honorable). Every company says (“telling”) it has world-class talent, but language choice is an objective decision that shows that a company is, at the very least, interested in hiring 5-percenters. For that reason, there’s a good chance that it employs some.

To reiterate, because I don’t want a flame war, is it possible to find 5-percenters who will write Java and C++ on a full-time basis? Absolutely, and if you want to compete with Goldman Sachs on salary, you might be able to hire one. Will they take on the risk of working for $5,000 (pre-tax) per month at a risky, seven-person startup in these languages? Not a chance. Five-percenters tolerate C++ jobs at Google because they know that company’s existence doesn’t rely on their individual productivity. In a startup, individual productivity is an existential concern, and the Aspergerian “pathological honesty” that top programmers almost invariably have precludes them from working in low-productivity languages that they believe will retard and destroy their employer.

I’ve said enough about the awfulness of C++ and Java. What are some good languages? I’ll give a list, which is not at all inclusive, of highly-powerful languages that the best developers love. Ocaml, Haskell, Erlang, Scala, Lisp, Clojure. Less strong but still formidable are Python and Ruby. (Many 5-percenters love Python, and almost all will tolerate it, but the median Python programmer is closer to a “10-percenter”.) Why is it this way? First, great developers program and learn technology in their spare time, and a 1-person, 15-hour-per-week project must be written in a real language if it is going to amount to anything. Second, most of these languages are only used by the best employers and only taught by the best universities, so a person deeply familiar with one of them is either (a) coming from elite exposure, or (b) possessive of enough individual curiosity to indicate a high likelihood of skill and success as a computer programmer. People who want to become great programmers quickly discover languages in which it’s possible for them to be 10 times as productive as they would be in Java– and they never look back.

Are all programmers in great languages 5-percenters? The answer is no. Users of languages like Ocaml and Scala tend to fall into two categories: (a) the 5-percenters, and (b) those who are becoming 5-percenters. Not all of them are there yet, but they’re almost all improving at a rapid pace. I’ve worked for almost 4 years in JVM languages (Java, Scala, and Clojure) and what I’ve learned (perhaps astonishingly) is that, per unit time, the Scala and Clojure developers learn about Java faster than those using Java! Because they are in high-productivity languages, they can accomplish more, and because they’re achieving more, they’re learning more along the way.

From a practical standpoint: with so many great languages to choose from, which one of those awesome languages should a person pick? CTOs making this decision have some idea, but this would be an impossible decision for a non-technical CEO to make on direct experience. For a very small team, the answer is easy: whatever the best programmers want to use. I like Scala much better than Python, but if I were in a non-coding role and tasked with hiring a great programmer, and if she preferred Python, I’d have her use that, because the benefit of letting her use the language she thought was best for the job outweighs (for a small team and with no maintenance burden) any benefit conferred by using one powerful language over another. So my answer to the language-selection question to a non-programmer CEO is: ask your best developer.

For my part, I’d recommend Scala. It’s not the best language for all purposes, but it’s a great general-purpose language and (among mainstream languages) may be best choice overall for most purposes. Because it runs in the Java Virtual Machine (JVM) it has full access to all of Java’s assets. (Clojure, an excellent JVM lisp, has the same advantage, but is not as performant and does not have static typing, a feature I find invaluable.)

A person versed in economics might find my argument tenuous. If a set of “elite” languages can be used by programmers and employers as a signal of high competency, what’s to stop the less competent from “faking” this signal once they catch on to the fact that it exists? A few answers come to mind. The first is that programming in a high-power language requires an adjustment that mediocre, 9-to-5, programmers are not likely to want to make. From first principles, functional programming isn’t intellectually harder than object-oriented programming: a 120 IQ is more than enough. It’s actually simpler. (Doing object-oriented programming correctly, and rigorously understanding what one is doing with it, is much harder and much more intellectually complex than succeeding in functional programming; this is a rant for another time, but 99% of people doing “object-oriented-programming” are like crude teenagers with regard to sex– loud about it, but doing it badly.) Nonetheless, the intellectual difficulty of re-learning software on sound principles is not an easy one to make. Like the transitions from memorization to pattern recognition, and then from pattern recognition to rigorous proof in mathematics, these context-switches require a lot of work (months of serious study). Successfully learning Scala or Clojure is a sign of a very strong work ethic. (It goes without saying that your interview process should establish that the candidates actually know these languages, and aren’t playing “buzzword bingo”.)

What about mediocre businesses using language selection as a false signal? That’s even more unlikely. A recruiter (for an elite startup, currently at less than 20 people) I spoke to told me that about 70% of Clojure candidates, 40% of Python candidates, and 5% of Java candidates that he invites to an in-office, full-day interview are good enough to hire. For a startup, interviews are far more expensive (in terms of opportunity cost) than for large companies: it costs about $100 to run a technical phone screen and $1000 to conduct a full-cycle interview, because startups have to involve senior people in their interview process in order to assess quality. Put another way, this means that it costs $1429 to hire a 5-percenter in Clojure and $2500 to hire one in Python– chump change as far as recruiting expenses go. But it costs $20,000 to hire a Java developer, if you insist on the “5-percenter” standard of quality.

There’s one variable I haven’t mentioned, though, and that’s the number of CVs he gets in each language: several hundred times as many Java developers than developers in Clojure or Ocaml. Most companies need (or think they need) warm bodies in large numbers to maintain legacy horrors, not top-talent and the attitude that comes with it. Also, it’s awful that it’s this way, and I think it will change in the future, with the limiting factor being work ethic and engagement rather than innate ability, but the software world is a pyramid, with a few stars at the pinnacle and a large number of incompetents at the base. This holds for programmers and for programmer jobs, of which 90 to 95 are mind-numbingly boring. It’s also seen in the distribution of language preference (and I say “preference” because there are tens of thousands of excellent programmers writing C++ and Java right now, but very few prefer them). The result of this is that the mediocre languages have the most programmers. If a company needs to hire 200 programmers per month, it simply cannot choose Haskell as its main development language; within a couple years, it would have absorbed the entire Haskell community!

Historically, that has been a serious concern for companies when it comes to language selection. Because there are hundreds of times more Java developers than Haskell or Ocaml hackers, there are at least fives of times more half-decent ones. Thus, the worst languages paradoxically have the strongest library support. This is compounded by the fact that powerful tools (such as IDEs, which are a mixed bag of neatness and horror, but sometimes quite useful and outright required when developing in Java) must be written to compensate the shortcomings of hobbled languages. There’s no Lisp IDE because emacs does just fine, but there are a slew of Java IDEs because it’s a revolting experience to write Java without one. From a non-technical CEO’s perspective, this makes Java look better because it has the best supporting tools.

What all this means is that mediocre software shops are not going to switch over to Haskell in order to ape this signaling mechanism. They can’t, because the bottom contingent of their software staff with drop like flies, and because their leadership is unlikely to understand the language selection problem in the first place. There might be some unestablished software companies using elite languages, and of them, I make the same argument that I’d make of “15-percenters” who nonetheless show interest and competence in “elite” languages– they may not be 5-percenters now, but if they keep at it, they will be shortly!

What about the (admitted) shortcomings of elite languages? Except for Scala and Clojure, which have access to the JVM and interoperate cleanly with Java, these languages don’t have the breadth of tooling that C++ and Java do. The answer to that is almost stereotypically “hackerish”: write them! This effort is not wasted, not in the least. Writing high-quality open-source tools to support elite languages is one of the best things a software company can do to establish its reputation. This, again, is an opportunity to show, not tell.

For a small company attempting to define and establish itself, attracting top talent is hard. The best programmers are on the job market so rarely and for such short intervals that attracting them takes concerted effort. Growing companies do not have the name recognition, and cannot afford the immense salaries (over $250,000 for a senior developer) that would attract these “5-percenters” using economic means, so they must win on technical grounds. Language selection is a simple (but not easy, because it’s difficult to adopt new and dramatically more powerful tools) way to do so. The best languages (Ocaml, Scala, Haskell, Python) are “shibboleths” that elite programmers use to identify each other, so adopting one of them is a one-stop choice that instantly establishes “hacker cred” or, to use Nir Eyal’s terminology, bridges “The Developer Divide”.