Why programmers can’t make any money: dimensionality and the Eternal Haskell Tax

To start this discussion, I’m going to pull down a rather dismal tweet from Chris Allen (@bitemyapp):

For those who don’t know, Haskell is a highly productive, powerful language that enables programmers (at least, the talented ones) to write correct code quickly: at 2 to 5 times the development speed of Java, with similar performance characteristics, and fewer bugs. Chris is also right on the observation that, in general, Haskell jobs don’t pay as well. If you insist on doing functional programming, you’ll make less money than people who sling C++ at banks with 30-year-old codebases. This is perverse. Why would programmers be economically penalized for using more powerful tools? Programmers are unusual, compared to rent-seeking executives, in actually wanting to do their best work. Why is this impulse penalized?

One might call this penalty “the Haskell Tax” and, for now, that’s what I’ll call it. I don’t think it exists because companies that use Haskell are necessarily cheaper or greedier than others. That’s not the case. I think the issue is endemic in the industry. Junior programmer salaries are quite high in times like 2014, but the increases for mid-level and senior programmers fall short of matching their increased value to the business, or even the costs (e.g. housing, education) associated with getting older. The only way a programmer can make money is to develop enough of a national reputation that he or she can create a bidding war. That’s harder to do for one who is strongly invested in a particular language. It’s not Haskell’s fault. There’s almost certainly a Clojure Tax and an Erlang Tax and a Scala Tax.

Beyond languages, this applies to any career-positive factor of a job. Most software jobs are career-killing, talent-wasting graveyards and employers know this, so when there’s a position that involves something interesting like machine learning, green-field projects, and the latest tools, they pay less. This might elicit a “well, duh” response, insofar as it shouldn’t be surprising that unpleasant jobs pay well. The reason this is such a disaster is because of its long-term effect, both on programmers’ careers and on the industry. Market signals are supposed to steer people toward profitable investment, but in software, it seems to fall the other way. Work that helps a programmer’s career is usually underpaid and, under the typical awfulness of closed allocation, jealously guarded, politically allocated, and usually won through unreasonable sacrifice.

Why is the Haskell Tax so damning?

As I said, the Haskell Tax doesn’t apply only to Haskell. It applies to almost all software work that isn’t fish-frying. It demolishes upper-tier salaries. One doesn’t, after all, get to be an expert in one’s field by drifting. It takes focus, determination, and hard work. It requires specialization, almost invariably. With five years of solid experience, a person can add 3 to 50 times more value than the entry-level grunt. Is she paid for that? Almost never. Her need to defend her specialty (and refuse work that is too far away from it) weakens her position. If she wants to continue in her field, there are a very small number of available jobs, so she won’t have leverage, and she won’t make any money. On the other hand, if she changes specialty, she’ll lose a great deal of her seniority and leverage, she’ll be competing with junior grunts, and so she won’t make any money either. It’s a Catch-22.

This puts an economic weight behind the brutality and incivility of closed allocation. It deprives businesses of a great deal of value that their employees would otherwise freely add. However, it also makes people less mobile, because they can’t move on to another job unless a pre-defined role exists matching their specialties. In the long run, the effect of this is to provide an incentive against expertise, to cause the skills of talented programmers to rot, and to bring the industry as a whole into mediocrity.

Code for the classes and live with the masses. Code for the masses and live… with the masses.

Artists and writers have a saying: sell to the masses, and live with the classes; sell to the classes, and live with the masses. That’s not really a statement about social class as about the low economic returns of high-end work. Poets don’t make as much money as people writing trashy romance novels. We might see the Haskell Tax as an extension of this principle. Programmers who insist on doing only the high-end work (“coding for the classes”) are likely to find themselves either often out of work, or selling themselves at a discount.

Does this mean that every programmer should just learn what is learned in 2 years at a typical Java job, and be done with it? Is that the economically optimal path? The “sell to the masses” strategy is to do boring, line-of-business, grunt work. Programmers who take that tack still live with the masses. That kind of programming (parochial business logic) doesn’t scale. There’s as much work, for the author, in writing a novel for 10 people as 10 million; but programmers don’t have that kind of scalability, and the projects where there are opportunities for scaling, growth, and multiplier-type contributions are the “for the classes” projects that every programmer wants to do (we already discussed why those don’t pay). So, programming for the masses is just as much of a dead end, unless they can scale up politically– that is, become a manager. At that point, they can sell code, but they don’t get to create it. They become ex-technical, and ex-technical management (with strongly held opinions, once right but now out of date) can be just as suboptimal as non-technical management.

In other words, the “masses” versus “classes” problem looks like this, for the programmer: one can do high-end work and be at the mercy of employers because there’s so little of it to go around, or to low-end commodity work that doesn’t really scale. Neither path is going to enable her to buy a house in San Francisco.

Dimensionality

One of the exciting things about being a programmer is that the job always changes. New technologies emerge, and programmers are expected to keep abreast of them even when their employers (operating under risk aversion and anti-intellectualism) won’t budget the time. What does it mean to be a good programmer? Thirty years ago, it was enough to know C and how to structure a program logically. Five years ago, a software engineer was expected to know a bit about a Linux, MySQL, a few languages (Python, Java, C++, Shell) and the tradeoffs among them. In 2014, the definition of “full stack” has grown to the point that almost no one can know all of it. Andy Shora (author of the afore-linked essay) puts it beautifully, on the obnoxiousness of the macho know-it-all programmer:

I feel the problem for companies desperate to hire these guys and girls, is that the real multi-skilled developers are often lost in a sea of douchebags, claiming they know it all.

Thirty years ago, there was a reasonable approximation of a linear ordering on programmer skill. If you could write a C compiler, understood numerical stability, and could figure out how to program in a new language or for a new platform by reading the manuals, you were a great programmer. If you needed some assistance and often wrote inefficient algorithms, you were either a junior or mediocre. In 2014, it’s not like that at all; there’s just too much to learn and know! I don’t know the first thing, for example, about how to build a visually appealing casual game. I don’t expect that I’d struggle as much with graphics as many do, because I’m comfortable with linear algebra, and I would probably kill it when it comes to AI and game logic, but the final polish– the difference between Candy Crush and an equivalent but less “tasty” game– would require someone with years of UI/UX experience.

The question of, “What is a good programmer?”, has lost any sense of linear ordering. The field is just too vast. It’s now an N-dimensional space. This is one of the things that makes programming especially hostile to newcomers, to women, and non-bullshitters of all stripes. The question of which of those dimensions matter and which don’t is political, subjective, and under constant change. One year, you’re a loser if you don’t know a scripting language. The next, you’re a total fuckup if you can’t explain what’s going on inside the JVM. The standards change at every company and frequently, leaving most people not only at a loss regarding whether they are good programmers, but completely without guidance about how to get there. This also explains the horrific politics for which software engineering is (or, at least, ought to be) notorious. Most of the “work” in a software company is effort spent trying to change the in-house definition of a good programmer (and, to that end, fighting incessantly over tool choices).

I don’t think that dimensionality is a bad thing. On the contrary, it’s a testament to the maturity and diversity of the field. The problem is that we’ve let anti-intellectual, non-technical businessmen walk in and take ownership of our industry. They demand a linear ordering of competence (mostly, for their own exploitative purposes). It’s the interaction between crass commercialism and dimensionality that causes so much pain.

Related to this is the Fundamental Hypocrisy of Employers, a factor that makes it damn hard for a programmer to navigate this career landscape. Technology employers demand specialization in hiring. If you don’t have a well-defined specialty and unbroken career progress toward expertise in that field, they don’t want to talk to you. At the same time, they refuse to respect specialties once they’ve hired people, and people who insist on protecting their specialties (which they had to do to get where they are) are downgraded as “not a team player”. Ten years of machine learning experience? Doesn’t matter, we need you to fix this legacy Rails codebase. It’s ridiculous, but most companies demand an astronomically higher quality of work experience than they give out. The result of this is that the game is won by political favorites and self-selling douchebags, and most people in either of those categories can’t really code.

The Eternal Haskell Tax

The Haskell Tax really isn’t about Haskell. Any programmer who wishes to defend a specialty has a smaller pool of possible jobs and will generally squeeze less money out of the industry. As programming becomes more specialized and dimensional, the Haskell Tax problem affects more people. The Business is now defining silos like “DevOps” and “data science” which, although those movements began with good intentions, effectively represent the intentions of our anti-intellectual colonizers to divide us against each other into separate camps. The idea (which is fully correct, by the way) that a good Haskell programmer can also be a good data scientist or operations engineer is threatening to them. They don’t want a fluid labor market. Our enemies in The Business dislike specialization when we protect our specialties (they want to make us interchangeable, “full stack” generalists) but, nonetheless, want to keep intact the confusion and siloization that dimensionality creates. If the assholes in charge can artificially disqualify 90 percent of senior programmers from 90 percent of senior programming jobs based on superficial differences in technologies, it means they can control us– especially if they control the assignment of projects, under the pogrom that is closed allocation– and (more importantly) pay us less.

The result of this is that we live under an Eternal Haskell Tax. When the market favors it, junior engineers can be well-paid. But the artificial scarcities of closed allocation and employer hypocrisy force us into unreasonable specialization and division, making it difficult for senior engineers to advance. Engineers who add 10 times as much business value as their juniors are lucky to earn 25 percent more; they, as The Business argues, should consider themselves fortunate in that they “were given” real projects!

If we want to fix this, we need to step up and manage our own affairs. We need to call “bullshit” on the hypocrisy of The Business, which demands specialization in hiring but refuses to respect it internally. We need to inflict hard-core Nordic Indignation on closed allocation and, in general, artificial scarcity. Dimensionality and specialization are not bad things at all (on the contrary, they’re great) but we need to make sure that they’re properly managed. We can’t trust this to the anti-intellectual colonial authorities who currently run the software industry, who’ve played against us at every opportunity. We have to do it ourselves.

Status checks and the stink-when-breaking problem

Most software engineers, being rational people and averse to legacy, hate management– a set of legacy processes that were necessary over the concave, commodity work that existed for 200 years, but counterproductive over the convex stuff we do as software engineers. No, we don’t hate managers. They’re just people. They have this job we dislike because it seems to require them to get in our way. We only covet their jobs insofar as we wish we had more control over tools and the division of labor– we have this impractical dream of being managers-who-code (i.e. exceptionalism) meaning that we’re full-time programmers who have managerial control, even though the real world doesn’t work that way and most actual managers find their time spent mostly in meetings– but mostly, we think what they do is unnecessary. That’s not 100% true. It’s about 97% true. Resolving personnel issues, mentoring new hires, building teams and looking out for the health of the group– all of these are important tasks. The “do what I say or I’ll fire you” extortionist management that exists when bosses prioritize “managing up” and stop giving a shit about genuine leadership and motivation is the problem. I’ve said enough on that to make my opinions well-known.

A typical inefficiency that we perceive, as engineers, is the fact that most of the work that “needs to be done” doesn’t. It’s fourth-quadrant work that exists only because people lack the ability to refuse it. We’re rationalists and careerists, so we doubly hate this. We realize that it’s inefficient, but, more to our dislike, that it’s individually bad for us to work on low-yield projects. Why is there so much nonsensical dues-paying? And why does Corporate America feel more like the Stanford Prison Experiment than a group of people working together to achieve a common goal? I believe I can answer that. We need to understand what “business” is to non-rational people. It’s emotional. Judgments are made quickly, then back-rationalized. Decisions are made without data, or on sparse and poorly audited data, which is why an error in an Excel model can be such a big deal. So let me get to the one question that non-technical “businessmen” ask about people when evaluating them. It is…

Do they stink when they break?

Some people drop to 0.9 (that is, they lose 10% of peak productivity) under managerial adversity. (A few might go up, with a 1-in-1-million effective-asshole manager like Steve Jobs. So rare I’m ignoring it.) Some drop to zero. Some go to -10 as they cause morale problems. Managers want to see what happens ahead of time so they can promote the first category, mark the second as non-advancers, and fire the third. People are sorted, then, based on the rightward side of their decline curve. The reason this is fucked-up and morally wrong is that, if management is at all decent, that extreme territory is never seen. So a lot of what management tries to do is probe people for those tendencies in little ways that don’t disrupt operations.

Rationalists like engineers don’t like this. We look at the affair and say, “Why the fuck are you trying to break people in the first place? That’s fucking abusive and wrong.” We try to build systems and processes that fail harmlessly, do their jobs as well as possible, and certainly never break people. We like things that “just work”. Some of us are glory whores when it comes to macroscopics and “launches”, and we have to be that way for career reasons but, on the whole, the best endorsement of a product is for people to use it and not even know they are using it, because it works so damn well. When we look for weak points in a system, we work for technical issues and try to solve those. Non-rationalists don’t think that way. They look for weak people and, when there are none, they make one up to justify their continued high position (and as an easy blame eater).

When you deal with a non-rationalist, the first think you need to know about him is that he’s trying to figure out how bad it smells when you break. If you’re loaded up with 80 hours per week of undesirable work for two months on end, will you bear it? Or will you quit? Underperform? Make mistakes? Whine and cause morale problems? Grind to a halt, get fired, and demand severance not to sue? Sabotage systems? Non-rationalists want to find peoples’ failure modes; engineers want to build systems that are tolerant of human failure.

I think that this stink-when-breaking matter is one of the fundamentals of negotiation, as well. When you stink, you lose all negotiatory power. You whine, you make messes, you get people pissed off, and you end up in a position of weakness. You prove that you’ve been broken. You might have worse kinds of stink up your sleeve, but you’ve put your managerial adversary into damage-control mode and lost your element of surprise. The best way to extract value from a non-rationalist is, instead, to leave him not knowing whether you’ve broken yet, which means you have to minimize your smell (of defeat, resentment, hatred, or adversity). Are you intact, or is your stink very mild? You just can’t stink, if you want to negotiate from a position of strength, no matter how badly you’ve been broken. Breaks will heal; stink is permanent.

Okay, we now have an understanding that allows us to understand the non-rationalists that we either answer to, or answer to people who answer to. That’s a start. Now I’m going to talk about the bane of programmers: impromptu status checks. I’m not talking about necessary communication, as would occur during a production crisis, nor about planned standups (which are painful when poorly run, but at least scheduled). I’m talking about the unplanned and involuntary games of 52-Card Pickup that managers inflict on us when they’re bored. Mention status pings to a software engineer, and it’s like you just uttered a racial slur. We fucking hate hate hate hate hate those with a +8 modifier to Anger. I call it “email syndrome”. Email can be checked 74 times per day with no degradation to performance. Making the same assumption about a person is a terrible idea. For an engineer, one impromptu status ping per day is an order of magnitude too much (during normal circumstances). If you really need daily status information (and you shouldn’t; you should trust people to do their jobs) then run a scheduled standup for which people can prepare. It’s just disrespectful to put people on trial, constantly, without their having the right to prepare.

I used to take a Hanlon-esque approach to status pings. (Hanlon’s Razor is “never attribute to malice what can be explained by benign incompetence.”) Now, my attitude is more cynical. Why? Because the first thing you learn in Technology Management 101 is that engineers can’t fucking stand impromptu status pings. Our feeling on this is simple: if you really don’t trust us with hours and days of our own fucking time, then don’t hire us. It’s that simple and, not only that, but most technology managers know that programmers hate these utterly pointless status checks. If we wanted to be interrupted to feed babies, we’d have our own.

So what’s the real point of an impromptu status check? You know that Carlin-esque phenomenon where you look at your watch but don’t read it, then have to look again because you didn’t catch what time it was? That’s exactly how managers’ impromptu status checks work. It’s not actually about project status. No one has any clue how long things “should” take anyway; on that, the world is split between people who know they don’t know and people who don’t know that they don’t know. It’s about making a manager feel secure. You have two jobs. One is to make the manager feel important; the whole reason he is interrupting you is because to establish that he can, and your job is that of an actor. The other is to make him feel like you are not slacking, and that aspect of it (lack of trust) is goddamn insulting. Again, I’m not talking about a production crisis that might actually merit such a fast audit-cycle. I’m talking about 2-20 status pings per week during peace time. It’s painful and wrong and the context-switches drop productivity to zero.

Why don’t managers intuitively understand this? I’m going to paint management in the best possible light. They’re salesmen. That’s not a pejorative. Far from it. That’s a valuable function, especially because while trust sparsity is horrible as a matter of policy, it tends to be somewhat inevitable, and salesmen are the best people at breaking down trust sparsity. They sell their superiors on their ability to deliver a project. They (if they’re good) sell their reports on the value of the project to their careers. All day, they’re selling someone on something. So they’re in Sales Mode constantly, and that is their flow. For a manager, a status ping is a welcome break: a chance to have someone else do the selling (i.e. prove that he hasn’t wasted the previous 3.37 hours of working time) to them. For the programmer, though, it’s a sudden and unexpected jerk into Sales Mode, which is a different state of consciousness from Building Mode. Both kinds of flow have their charms and can be a lot of fun, but mixing the two is both ineffective and miserable, especially when not by one’s choice.

If these status pings are so horrible, then why are there so many programming environments in which they’re common? Well, most managers are non-rationalists, and non-rationalists are all about emotion, perception, and superficial loyalty. Again, remember that question, and that test: does this guy stink when he breaks? Is his attitude (that he’s feeding an impulsive toddler) evident? Does his face show a look of enmity or defeat? Does he make eye contact with the boss or with the computer screen? What’s his posture? What’s his tone of voice?

Of course, it’s these status checks have negative value as an assessment of project progress, because the signal is going to be opposite of the truth. Employees pass these impromptu status checks when they’re prepared, and that means they aren’t doing any real work. They fail (because they’re pissed off about the interruption) when they’re actually getting something accomplished. That, however, is not what they’re really about. The manager will forget everything you told him, 15 minutes later. (That’s part of why status checks are so fucking infuriating; they’re about social status, not project progress.) They’re about not stinking when you break. It doesn’t matter what you got done in the past few hours, but you need to show an ability to communicate progress without getting pissed off about the indignity of a tight audit cycle and hope, for fuck’s sake, that either he’ll start to trust you more or that you’ll be able to get a different manager.

What’s the solution? Well, there are two kinds of status checks. The first kind is the simple kind that doesn’t involve follow-on questions. I think the solution is obvious. Write down a sentence about what you’ve accomplished, every 30 minutes or so. Maybe this could be integrated with the Pomodoro technique. Now you have a story that can be read by rote and don’t have to you don’t have to deal with the social and emotional overhead of a sudden switch into Sales Mode. You’re prepared. That will work as long as there aren’t follow-on questions; if there are, then it’s the second kind of status check.The second kind is the cavity search that comes from a manager who just has nothing else to do. I think the only solution in that case (unless it’s rare) is to change companies.

Tech Wars

Any creative field undergoes periods of divergence and convergence, and technology is no exception. Right now, we’re in the late swing of a divergent phase, with a plethora of new languages, frameworks, paradigms and methodologies so vast that it’s impossible, at this point, even to know a respectable fraction of them. There are “back-end programmers” who know nothing of Javascript, and “front-end guys” who’ve never used a profiler. That will end, as people grow tired of having to learn a new technology stack with every job change, and also as the dimensionality of the bilateral matching problem between engineers and employers becomes intolerable. As painful as convergence can be, such a time will come out of necessity.

For a single company, convergence is often enforced internally, with “approved tools” lists and style guides. Often the communication that drives these decisions takes a familiar form: flame wars.

These partisan fights are common in technology. There’s one set of programmers who believe Java is the only language that has a right to exist and that, if you’re not using an IDE, you’re using “stone tools”. There’s another contingent that thinks everything should be written in Ruby, because that’s what Rails is in and why do we need another goddamn language? There’s a third that’s fixated on C++ on the basis that, if you can’t manage memory, how can you make sure an application performs? For now, I don’t care who’s right in this– really, that depends on the problem, because there clearly isn’t “one language to rule them all”– and language or style comparisons are not what I care to discuss today. I think we can all agree these flame wars are stupid and counterproductive. Do tabs/spaces debates really require long email chains? No, they don’t.

This, if anything, is the thing I dislike second-most about programmers. The worst thing about programmers is that most of them (although this is a largely a product of a corporate environment that rewards mediocrity) lack taste and curiosity, and therefore never become competent. The second-worst is that, among those who have taste, most of the heated debates come down to superficial nonsense. Yes, code quality matters. It’s critically important; it can make or wreck a company. Cosmetic differences, on the other hand, are just not that important or interesting. Tooling choices are important, but rarely as critical as opinionated engineers make them out to be, in the sense that they’re rarely existential matters for the company. Closed allocation is cultural death, and unethical management will ruin your startup, but the difference between Python and Scala is unlikely to be lethal. Can you write performant systems using mostly Python? Of course. Can high-quality, attractive code be written using an inelegant language like C++? Yes, it can. I have strong preferences regarding languages, but I’ve learned over the years not to take them too seriously. More dangerous than picking a suboptimal language is not getting any work done because of partisan bickering.

Language and tool wars are an outgrowth of a careerist idiocy that one would expect software engineers to be above. Engineers engage in this hyperbolic mudslinging because their career needs, in a typical corporate power struggle, often necessitate some control over the technical environment– and that’s often a zero-sum squabble. The software industry isn’t a real meritocracy, but it insists on looking like one, so the object of the game is to convince the team to use the tools with which one is most familiar, in order to become (because of tool fit, not intrinsic superiority) the most productive. This is the problem with technology in the late-autumn phase of a divergent spell: an engineer’s job performance is going to be more of a function of his match with the work environment and tools than of his overall competence, so there’s an enormous incentive to change the game if one can. There’s more individual gain in changing existing technical choices than grinding away at regular work. Most software engineers have learned this, and I can’t count on one hand the number of times I’ve seen a company go through a scorched-earth code rewrite because some douchebag managerial favorite only knew one language, and (of course) that language became the only possible option for the firm. This is terrible for the company that has to reinvent all of its technical assets, but beneficial to the favorite who naturally becomes the master of the new technical universe.

In a world of tight deadlines, short employment stints, and extremely detailed job requirements, this is only getting worse. It’s no longer acceptable to spend months in a “ramp up” period on a new job. It won’t get a person fired, but you’re unlikely to get the best projects if you don’t blow someone away in your first 3 months. So you need to find or carve out a niche where you can use tools you already know, in order to get some macroscopic achievement (a “launch”) as soon as possible. This forces companies into one of two undesirable alternatives. The first (sprawling divergence) is to let everyone use the tools they like, and the predictable result is “siloization” as engineers stick to the tools they already know how to use, rather than take the time to understand what other people are doing. Taking these systems into production and supporting them becomes a nightmare, because they often have 7 different NoSQL databases to support what’s essentially a CRUD app. The other (enforced convergence) is to require certain uniformity in development. The company becomes an “X Shop” for some specific language (or database, or methodology) X. That’s when the flame wars begin, because everyone is going to have an enormous stake in the choice of X. Now, it’s not enough for X to be the best tool for your project; you have to shove X down other peoples’ throats in the company, or you won’t be able to use it at all. This also falls down as necessity requires exceptions to the uniformity requirements, and the decision of who can get an exception becomes immensely political.

In truth, we as programmers are, in many companies, behaving like executives, spending more time arguing about how to do work than doing the actual work.

This, I think, is the fall-down calamity of the software industry. It’s why 50 percent of people who are professional programmers won’t be, seven years from now. Software engineering is all about improvement– building things that didn’t exist before, automating tedious processes– and it’s only natural that we want to improve ourselves. In order to get better, one needs a reliable stream of increasingly high-quality projects. What makes great software engineers extremely rare (and they are) isn’t a lack of talent alone; the limiting factor is the paucity of quality work. At a certain point, one reaches a level where it’s difficult to get quality work, even for a qualified (or overqualified) engineer. You end up spending more time convincing managers to give you good projects (playing politics, acquiring control of the division of labor) than you actually get to spend on the work. People get enervated and drop out, or they become managers and lose touch with the day-to-day process of writing code. Tooling wars are one component of this nasty slog that programmers have to enter in order to have a shot at the best work.

What’s the solution? I mentioned the alternative to uniformity, which is the “everyone uses whatever they want” divergent approach that often generates an intolerable support burden. It can work, but it will fail in the typical hierarchical corporation. When you have this proliferation of creativity and ideas, you need two things. First, there has to be a pruning process, because some of what is produced won’t be very good. Managers never get this right. Their decisions on what modules to keep (and, thereby, force engineers to maintain and use) and which ones to burn will be based on political factors rather than code or product quality. Second, it requires extremely strong cross-hierarchical communication, because it requires that people begin to make technical choices based on the long-term benefit of the group rather than short-term “productivity” in an attempt to acquire status. People who are building technical assets will need to actually give a damn about making their clients succeed, as opposed to the selfish, parochial goal of pleasing managers. You won’t get this kind of communication and pay-it-forward attitude– not as prevailing behaviors, rather than occasional serendipities endowed by the comfortable– in a closed-allocation company.

The open-allocation sociology is one in which there is cross-hierarchical collaboration because the conceptual hierarchy necessitated by the problem doesn’t congeal into a rigid hierarchy of people. Because there’s so much internal mobility, other teams are potential future teammates, and people will generally treat them well. What this means is that, while there will be divergence in the name of exploration and creative license, people will also take care of the convergence tasks, such as integrating their work with the rest of the company and teaching other teams how to use the assets they generate.

For a contrast, the closed-allocation sociology is one in which people strive, in the short term, for the image of superior productivity, in order to advance rapidly into a role where they control the division of labor instead of being controlled by it. This encourages people to diverge carelessly, for the twin purposes of (a) appearing highly productive and useful, and (b) creating personal security by generating technical assets that, although management can be convinced of their importance through political means, become so opaque to use as to leave other teams beholden to their creator. Of course, this productivity is illusory, the reality being that costs are externalized to the rest of the company. The heavy-handed, managerial antidote is to mandate convergence, but that inducing battles regarding which technologies and approaches live and which die. The result are hyperbolic, counterproductive and fear-driven arguments that devolve into the technical flame wars we all know and loathe.

As long as we have hierarchical corporations and closed-allocation regimes where one must either control the division of labor or be controlled by it, we will have endless flame wars over even the smallest of technical choices. The stakes are too high for them not to exist.

The unbearable B-ness of software

I’m not Jack Welch’s biggest fan. For one thing, he invented the “rank-and-yank” HR policies that literally decimate companies. I don’t disagree with the idea that companies would improve their health by letting go 5-10 percent of their people per year, but I think the discovery process involved is impossible and often politically toxic. There is, however, one thing he’s said that I think has a lot of value: “A players hire A players; B players hire C players“. His point is that if you have mediocre management, you’ll actually end up with terrible employees. I would say it’s not limited to hiring only. A players make more A players. They teach each other how to be better. Not only that, but they raise the potential for what an A player can do. B players don’t have the foresight or “ownership” mentality to mentor others, and produce non-productive C players.

The insight I had recently is that this applies to software as well. “B” architectural decisions or tooling choices, which seem mediocre but tolerable at the time, create “C” (ugly, defective, or unusable) software. Software contributions often have, not always intentionally, a multiplicative (or divisive) effect across a company, in which case a single programmer can become “manager-equivalent” in impact. This is something that most companies, especially outside of technology, fail to realize about software. They miss this fact to their peril.

I’m increasingly convinced that it’s hard to be a great programmer and not be, at least, a half-decent leader. This doesn’t mean that one needs to be a “people manager” or even have any interest in taking that direction. However, a great software engineer is:

  • a decision-maker, because engineers frequently have to choose technologies and make infrastructural choices under conditions of incomplete information.
  • a creative problem-solver, because for the hardest problems there is no “canned” solution. In fact, for some problems (as in machine learning) the best solution may not be known, and simply designing the experiments is a non-trivial problem.
  • a designer, because the ability to create attractive, robust and maintainable software systems is uncommon, and the work is non-trivial.
  • a teacher, because good engineers don’t just leave a wad of undocumented code laying around, but take responsibility for making sure that other people can use them.

How is all of this, when done right, not a leadership role?

Of course, software engineers are not in general treated like leaders in any large company that I know of, and a fair proportion of the people who are hired into software positions just aren’t capable of being leaders. Still, there’s an overwhelming and self-perpetuating culture of B-ness in software, with engineers not trusted to pick their projects and choose their own tools. This culture of mediocrity is one which what I called “Java Shop Politics” emerges. I regret the name, however. It’s not fair to single out Java, especially when it was Microsoft, with Visual Basic and the first scratches at IDE-culture, that first attempted to create the commodity programmer world. A better name would be “Big Software Politics”.

I would replace Welch’s A-, B-, and C-player language with a four-class system of dividers, subtracters, adders, and multipliers. I’ve separated the “C” category between the ones who are truly toxic and hurt others’ productivity (dividers) from the more harmless people who just don’t get much done (subtracters). Dividers, I think, should be fired quickly if they don’t improve. The only crime of subtracters is to draw more in salary than they produce, but it’s worse (for morale) to fire them, so they should be mentored into adders and (eventually) multipliers whenever it is possible, and gently let go if it seems not to be. Ultimately, no company should retain an employee who doesn’t have the drive and capability to become a multiplier, but it takes time for a person to get there and firing is an extremely blunt instrument. In general, I wouldn’t fire anyone but a divider.

“B-player” and “adder” seem to correspond neatly, as do “A-player” and “multiplier”. The first category can crank out CRUD apps just fine, and write software to spec, but lack the architectural or design skill to build much on their own. Adders are the workhorses who are capable of implementing others’ decisions but unready to make their own, while multipliers deliver growth by making others (and themselves) more productive through their (often far-reaching) contributions.

Management is especially challenging and dangerous because it becomes impossible, almost by definition, for a manager to be a mere adder. A manager’s job is to alter the way people work, and as with stock traders, “random” moves have negative expectancy. The percentage of people who have multiplier-level knowledge, talent, or skill is small– maybe 10 to 15 percent, in a typical company. Managers who don’t have that capacity become dividers who add noise and entropy.

Programming, the art of managing machines, is much the same way. There are always a few junior-level, self-contained additive projects in every company, but the core infrastructural work that will be used by a large number of people is multiplicative– if done well. Done poorly, it reduces capability and has a dividing effect. How is it that software– typically construed as an asset– can have such a divisive effect? The problem is management. When people don’t freely choose what tools they use, and what software they rewrite as opposed to what they deal with “as-is”, low-quality software combined with managerial blessing will lead to unproductive and unhappy programmers.

At the beginning of this year, I developed a scale for assessing the capability of a software engineer, and I’d formalize it a bit more with a model that first separates software work into three levels:

Level 1: Additive work, such as scripts to generate reports or CRUD business apps. This will typically be written once and read only by the original code author. Relevant question: can you code?

Level 2: Multiplicative work, such as tool development and critical production infrastructure in which performance, scalability, design and code quality matter, because large numbers of people rely on the work. Most of the “sexy” problems fit in this category. Relevant questions: does your work make others more productive? Can they use it? Do they enjoy using it?

Level 3: Globally multiplicative work, such as the design of new general-purpose languages. A level-3 accomplishment needs to be “best in class”, in some way, on a worldwide basis because its purpose is to push forward the state of the entire industry. Linux and Clojure are some examples of level-3 achievements. Most of this work is R&D that few companies are willing to pay for, these days. Relevant question: are you doing original work that increases capability globally?

As with any model, this is chock full of simplifications. In reality, there are hard L1 tasks that might be rated above 1.5, and easy L2 jobs as well that might be “only” 1.7 in difficulty, but for simplicity I’ll assume that tasks can neatly be “bucketed” into one of these three categories. The going assumption is that a programmer’s level represents the level at which she will make the right calls 95% of the time. For a level-2 task, the 2.0 programmer will succeed 95% of the time, the 1.5 programmer will get 50%, and the 1.0 will get 5%, with an “S-shaped” logistic interpolation giving meaning to the fractional levels (e.g. 1.1, 1.9). In practice, these concepts are too difficult to define for formal measurement (making it useless even to attempt to measure beyond one decimal place) and the bulk of professional software engineers are between 1.0 and 2.0. While it’s difficult to apply percentiles to software engineering, the population being ill-defined, I’d estimate that:

  • the median full-time professional software engineer is about 1.1. Many senior (20+ years experience) engineers never crack 1.2.
  • graduates of elite computer science programs are about 1.3.
  • about 1 in 10 professional software engineers are at 1.5 or above.
  • about 1 in 100 software engineers are 2.0 or above.

In the corporate world, level-3 software considerations are generally irrelevant. Such efforts tend to have an R&D flavor, and there’s rarely the budget (the limiting resource being the time of extremely high-power people) or risk-tolerance for a company to attempt them, so levels 1 and 2 are where the action is. You could safely say that level-1 work can usually be done by an adder or “B player”, while level-2 projects require an “A player”, “10X-er”, or multiplier.

Companies and software managers know, from extremely painful (read: expensive) experience, that level-2 work is hard and risky, and that most professional software engineers lack the competence to do it well, and even fewer are capable of supervising such work. The result is that they try to minimize the amount of such work, and the degree to which they’re willing to rely on it. If one “unit” of level-2 work can be replaced with four “units” of level-1 work, that seems like the prudent choice, because it’s astronomically easier to hire 1.0-1.3 programmers than to vie for 1.8-2.1, who can only be detected and assessed by other great programmers. This is the essence of the “commodity programmer” culture: create a world in which as much of the work as is possible is level 1, and allocate the level-2 work only to people with a “track record”, an assessment that often has more to do with politics and social position than capability.

What goes wrong? Well, the first problem with commodity developer culture is that the bulk of engineers living within it never improve. They stop learning, because there’s no need for them to progress. When companies staff people on subordinate, bite-sized work below their level of capability, they get bored and often decline as time goes on. If you don’t have 1.5+ level work, you’re not going to have many 1.5+ engineers, and you won’t keep any of them for very long. If what high-level work you have is jealously guarded and allocated politically, the best engineers won’t stick around for years to prove themselves adequate for it. The ones without other options will.

The second problem is that projects often become level-2 by accident, and also that level-2 needs tend to emerge once the complexity load of all the level-1 work reaches a critical mass. This is akin to Greenspun’s Tenth Rule, which essentially states that when low-level languages are applied to complex problems that require a more “high-level” approach, people implement the features that already exist in high-level languages like Lisp. This shouldn’t actually be taken as a slight against C: for low-level (here, “low-level” pertains to the degree of abstraction that is used, and is non-pejorative) problems where memory management is critical (yes, plenty of those still exist) it is often the best language, but you wouldn’t want to write a complex, modern web application entirely in C. In any case, what this “Rule” is really about is the emergence of complexity, driven by need. Lisp is a well-thought-out high-level language (a level-3 accomplishment) and almost guaranteed to be superior to the domain-specific language (DSL) that a typical corporate programmer, constrained by deadlines and managers and low autonomy, would “inject” into a C project to add high-level functionality (on an ad-hoc basis) to it.

I think Greenspun’s Tenth Rule can be generalized. At some point, the complexity load induced by level-1 requirements and work requires level-2 insights and capability. The problem with the commodity developer world is that, because level-2 work is effectively not allowed, the stuff that does happen at such level occurs accidentally in an ad-hoc way. A manager might see a neat script (a level-1 accomplishment) written by a junior developer and say, “This whole company should have that. By next Friday.” However, the developer lacks both the engineering skills and the political clout to recognize bad requirements and reject them, the result being an overblown, late, and unmaintainable system that serves many needs poorly instead of doing a few things well. Such are the good intentions that pave the road to hell.

All of this, I think, explains the sorry state of the software industry today. The business world understands software’s problems at a high level– most software is shitty, late, and over budget– and (correctly) concludes that the majority of software developers lack the skills to attack problems at the multiplicative (level-2) scope, while most executives are incapable of assessing software talent at the individual level. The result is a bureaucratic, creativity-killing culture that is tailored to software engineers in the 1.1-1.3 range, a world designed to protect itself against B players. The long-term problem is that this makes it nearly impossible for most software engineers to become 1.5+ A-players. One almost certainly won’t achieve it during one’s day job. Mediocrity self-perpetuates. What companies really need to do is give software engineers a creative “sandbox” (an R&D-like environment) in which they can attempt 1.5+ projects, and if they fail, no one gets hurt. However, I wouldn’t hold my breath for this, because most managers are going to see this as “not doing real work”.

The result is a culture of “B”-ness in software. That would be acceptable, but the reality is that level-2, “A player” needs emerge anyway as a result of the need to manage (and, preferably, reduce) the accumulating complexity that additive software generates. Unfortunately, this commodity developer culture is utterly unprepared to approach such problems, and fails miserably at them. The result of this is the slew of “C” (failed) software projects that will litter any corporate environment given enough time.

The software career and the Second Design Paradox

I’ll cold-open with a somewhat pointed, but valid question: why are there so few great software engineers? The answer: the software industry makes it extremely difficult to become good at engineering, unless a person is economically autonomous (read: wealthy, or well-connected enough to secure venture funding). Here are some of the obstacles that engineers face if they want to become great.

  • Low expectations of individual contributors. Non-managerial programmers are rarely expected, in most work environments, to have multiplier effects that make the team more productive, and therefore are rarely able to have the kinds of infrastructural contributions that would bring them feedback on what works, architecturally speaking, and what doesn’t. Instead, they’re often put to work on parochial business problems of low general interest or application, and often their contributions are to maintain some small piece of a massive system, so they rarely get a sense of large-scale engineering. 
  • Rarity of high-quality work. The kind of work that engineers need in order to become great is uncommon and requires substantial concentration. It’s almost impossible to get it without managerial buy-in (because one needs freedom from interruption and low-quality distractions) and, for the tiny amount of such work that companies allow to exist, it is allocated politically, rather than based on any sensible principles.
  • Obscene program size. When the program-to-programmer relationship is many-to-one, one gets sensible and small programs and highly productive engineers, but individual efforts are hard to track and measure. Consequently, many managers prefer a world where the program-to-programmer relationship is one-to-many and “headcount” can be tracked at a “per-program” level. These massive single-program teams tend to end up in a state where one person (the “blazer”) is highly productive and the others (the “trailers”) spend the bulk of their time keeping up with the changes, so individual productivity is low. Why is this low productivity tolerable? Because software engineers are massively underpaid compared to the business value they actually provide, so it’s acceptable (from a corporate perspective) to have them running at 25% efficiency.
  • Political distractions. When it takes as much time to prove that one is qualified to do interesting work as can be spent actually doing it, it becomes hard to maintain a high level of competence, much less improve upon it. It takes three to four years of campaigning in most companies to be considered credible enough to get the best projects, and by this point, a person’s technical skills have declined.

All of these issues exist, but there’s an obvious question. For high-potential software engineers, why don’t the political problems “solve themselves”? If they’re so good, why aren’t their managers falling over themselves to clear obstacles and let them do what they do best? Businesses would benefit immensely if they could discover and tap the top talent, so why don’t they? The answers are inherent in the concept of “The Design Paradox”, which I’ve split into two related “paradoxes”, one psychological and the other political.

First Design Paradox: people who lack design sense (“taste”) cannot assess it in others.

People with taste but not vision can recognize tasteful visionaries apart from charlatans, but people without taste are hopeless. This is a serious problem in software, because what differentiates a good programmer from a bad one is not how fast code can be written or how many lines are produced per day, but having the taste necessary to design usable, maintainable systems. Are those technical assets actually usable, or will they be tomorrow’s despised legacy horrors? This specifically technological variety of taste needs to be refined with experience, and that experience is hard to come by, so the taste itself is rare. People who don’t interact with software at a code level on a regular basis will have no intuition for what beautiful software is. You’re not likely to find this taste in a person off the street, and if there isn’t a strong reason to believe you have it, you almost certainly don’t.

This requires that technological taste be present at the very top of the company. The CEO will generally not be reading source code, but if the CEO lacks taste, then getting a tasteful CTO or VP/Eng is a crapshoot. If the CTO lacks taste, he won’t be able to tell which of his reports are capable and which are charlatans. Charlatans tend to be more skilled at office politics– they have more experience, due to a lifelong need to compensate for their own mediocrity– so there’s adverse selection involved.

What should a CEO do if he’s self-aware to understand his lack of taste in software architecture? Something very few managers like to do: give back some control. I think one of the reasons why Valve’s open allocation works so well is that it’s an admission of a reality that exists in every complex company: that executives don’t have all the answers. Workers are trusted to vote with their feet and throw their efforts behind the most tasteful projects– and they don’t have to wrangle with a rusty, corrupt permission system in order to do so.

Most discussions of “The Design Paradox” pertain to the first one, but I’m more interested in the second.

Second Design Paradox: people with taste will perform worse in environments with a history of poor taste, giving tasteless people a competitive advantage.

By “perform worse”, I mean that they will be worse at their jobs, and politically less successful. This is actually somewhat counter-intuitive. It’s akin to saying that the best writers are worse than average people at reading poorly-constructed prose– probably not true, for that particular example. In programming, however, it is true. Under a certain definition of “worst” (which I’ll explain, below) the best writers of code are the worst readers of it. That’s right. A critical skill for a corporate software engineer (reading other people’s code) is, under a certain set of circumstances, strongly negatively correlated to the ability to design good systems. What is that set of circumstances? It’s when the systems that must be understood are themselves incoherent or badly designed. Everyone struggles with it, but the best programmers have even more of a problem with bad code. Good designers will learn good technologies quickly and struggle immensely with bad ones.

I know this from painful experience. When I read code, I’m trying to connect what I am reading with how I would solve the problem. Usually, there are discrepancies, and if it’s good code, these are informative. Whenever I encounter a difference between my mental model of the problem and how it is actually solved, it helps me understand it in a deeper and more detailed way. So I enjoy reading good code, just as I enjoy reading a well-written mathematical paper, because there’s a convergence that occurs as I really learn something. The problem with bad code is that there’s such a stark and irreducible difference between how I would solve it and the way it is done, because it is done so tastelessly, and also because the only thing learned from it is parochial garbage pertaining to how ineffective or poorly-trained programmers think. You don’t learn the fundamentals of computer science from an AbstractProxyVisitorSingletonFactory. I can’t find convergence with a pile of FactoryFactory garbage because I would never write FactoryFactory garbage. There is simply no problem or set of circumstances (except a desire to severely punish an employer) that could bring me to write that kind of code, ever.

The Second Design Paradox is an aesthetic cousin of the Peter Principle, which is that “employees tend to rise to their level of incompetence”. There are two ways one can approach this. The first, which is wrong, is the argument that higher-level positions in a company are more difficult to perform, and that people therefore ascend as long as they are competent but reach a point where the work is too difficult for them. The reason this is wrong is that, although it is abstractly more difficult to do an executive job well, the managerial ranks of almost every company are so self-protecting and political that merit and promotion divorce early, and it becomes easier to succeed politically as one ascends. The standards of personal integrity and intelligence are lower once one has “made it”. In most companies, it’s hard to get into the managerial or executive “club”, but relatively easy, once in, to keep rising. In fact, the tribal self-protection of managers is at such an extreme that the obviously incompetent executives get what are known as “exit promotions”, which make their titles more attractive so they can “fail up” into another company and become someone else’s problem.

The second interpretation of the Peter Principle, which is the right one, is that companies tend to tap people for leadership roles based on their success as subordinates, while good leaders make shitty subordinates and good subordinates make ineffective leaders, because the ones who subordinate well are the people who just don’t care about doing things right.

Now I have to reconcile two statements that seem to contradict each other. One is the polite maxim that “Good leaders know when to follow, and do it well”. That’s true. The other is “Good leaders make shitty subordinates”, which is equally true. How can this be? There is a fundamental difference between following and subordinating. One who follows is choosing to do so, because she perceives it to be better for the group and, thus, also herself, to let a more informed or impartial person make calls. That’s the hardest thing about being a leader: knowing when to trust your own judgment, when to defer to someone else’s, and when to step down outright. What I like about Valve’s model, with its absence of entitled, permanent leaders called “managers”, is the recognition of this fact. On the other hand, a person who subordinates is accepting authority outright. Subordination is unconditional following and obedience that comes out of a preternatural acceptance of rank. When you follow, you have a choice. When you subordinate, there is no choice. You are surrendering to rank.

Most corporate managers aren’t leaders, but an internal police force trained to enforce personnel policies that come from a “Theory X” belief that most workers are ethically somewhere between depravity and juvenility and will steal from the company (slacking and prioritization of one’s own career goals both included as “stealing”) if not closely supervised. The problem is that indoctrination into the managerial religion makes it harder to lead, because it embraces such a negative view of the people being managed. True leaders never show contempt for those who are following them. They recognize that they have a transient functional superiority, not an innate moral one, and that if it is better for the group for them to take an inferior (following) role, they should do so.

Design has a natural need for leadership, because of the need for simplicity in the finished product. Many should be encouraged to suggest ideas and avenues for exploration (divergent creativity) but one person must sit in the center with the task of pruning, simplifying and prioritizing (convergent creativity). It’s rare that you get a good design without exactly one person who takes on the “killing” role. If no one takes on that nasty job, conceptual integrity vanishes as requirements and boondoggles are added. If two or more people take it on, all with some share of veto power, then the political machinations that ensue often create a product whose shape reflects the parochial political environment in which it was formed (Conway’s Law) rather than any universal insight about a problem. Decentralized power is good for government because the objectives are global concerns like justice and sustainability that are too large for one party to manage, but for the finished aesthetics of a local entity like a new product, you actually need a single leader who will accept input from many (thereby following, at the right scale, when appropriate) but prune when it is required to preserve the integrity of the whole product.

The question, then, is whether human organizations are effective at bringing design leaders into leadership roles, and I would say, based on experience, that the answer is “no”. Why? Because these organizations tend to promote people based on their ability to subordinate, resulting in a low quality of leadership. They don’t have the design sense, at the top, to recognize emerging design leaders, but it’s obvious to them who are the best subordinates and who are the grumblers.

In software, anyone who takes a maintenance role is expected to follow, by definition. When you maintain code, you can’t just rip it to shreds because you think it’s badly designed, because you’ll break others’ work. You have to modify the software in accord with the contract (often unspecified) that it has set with its users. Even if the CTO of a company takes on a maintenance project (and sometimes this is exactly the right thing for a leader to do) she is taking on a role where she must, to some extent, follow. That’s not a bad thing.

However, in most large companies, maintenance isn’t done by people who choose it for the good of the company, but by people who get stuck on it because they’re junior and don’t have the clout to do anything better. They don’t have the power to do maintenance right. They chug along at 20 lines of code per month, fixing minor bugs and adding low-priority features, working below their frontier of ability, and waiting to be tapped for a role with more substance. They’re not following, because they don’t have a choice. If they voice their desire to work on something better, or assert their creativity in any substantial way, they’ll probably be fired within a year. They’re subordinate. People with design sense tend to be turned off by this sort of thing, so they rarely pay their dues and climb the ranks. The ones who do are those who never bothered to learn design sense, and who think “functional programming” is some academic mumbo-jumbo with no place in the real world.

In the long run, the result of this is that most corporate software is of very low design quality, which aggravates the problem. Left unchecked, it creates an arena where the lack of taste becomes a political advantage, because the people who lack taste are better equipped to adapt to the bad architectural decisions the group becomes stuck with. They rise into positions of authority and then make more bad design decisions, utterly inadvertently because bad designs are the only thing they’ve been exposed to. At this point, the organization isn’t just producing an occasional piece of bad software. It becomes a BadSoftwareFactory.

Programmer autonomy is a $1 trillion issue.

Sometimes, I think it’s useful to focus on trillion-dollar problems. While it’s extremely difficult to capture $1 trillion worth of value, as made evident by the nonexistence of trillion-dollar companies, innovations that create that much are not uncommon, and these trillion-dollar problems often have not one good idea, but a multitude of them to explore. So a thought experiment that I sometimes run is, “What is the easiest way to generate $1 trillion in value?” Where is the lowest-hanging trillion dollars of fruit? What’s the simplest, most obvious thing that can be done to add $1 trillion of value to the economy?

I’ve written at length about the obsolescence of traditional management, due to the convex input-output relationship in modern, creative work, making variance-reductive management counter-productive. I’ve also shared a number of thoughts about Valve’s policy of open allocation, and the need for software engineers to demand the respect accorded to traditional professions: sovereignty, autonomy, and the right to work directly for the profession without risk of the indirection imposed by traditional hierarchical management. So, for this exercise, I’m going to focus on the autonomy of software engineers. How much improvement will it take to add $1 trillion in value to the economy?

This is an open-ended question, so I make it more specific: how many software engineers would we have to give a high degree (open allocation) of autonomy in order to add $1 trillion? Obviously, the answer is going to depend on the assumptions that are made, so I’ll have to answer some basic questions. Because there’s uncertainty in all of these numbers, the conclusion should be treated as an estimate. However, when possible, I’ve attempted to make my assumptions as conservative as possible. Therefore, it’s quite plausible that the required number of engineers is actually less than half the number that I’ll compute here.

Question 1: How many software engineers are there? I’m going to restrict this analysis to the developed world, because it’s hard to enforce cultural changes globally. Data varies, but most surveys indicate that there are about 1.5 million software engineers in the U.S. If we further assume the U.S. to be one-fifth of “the developed world”, we get 7.5 million full-time software engineers in the world. This number could probably be increased by a factor of two by loosening the definition of “software engineer” or “developed world”, but I think it’s a good working number. The total number of software engineers is probably two to three times that.

Question 2: What is the distribution of talent among software engineers? First, here’s the scale that I developed for assessing the skill and technical maturity of a software engineer. Defining specific talent percentiles requires specifying a population, but I think the software engineer community can be decomposed into the following clusters, differing according to the managerial and cultural influences on their ability and incentive to gain competence.

Cluster A: Managed full-time (75%). These are full-time software engineers who, at least nominally, spend their working time either coding or on software-related problems (such as site reliability). If they were asked to define their jobs, they’d call themselves programmers or system administrators: not meteorologists or actuaries who happen to program software. Engineers in this cluster typically work on large corporate codebases and are placed in a typical hierarchical corporate structure. They develop an expertise within their defined job scope, but often learn very little outside of it. Excellence and creativity generally aren’t rewarded in their world, and they tend to evolve into the stereotypical “5:01 developers”. They tend to plateau around 1.2-1.3, because the more talented ones are usually tapped for management before they would reach 1.5. Multiplier-level contributions for engineers tend to be impossible to achieve in their world, due to bureaucracy, limited environments, project requirements developed by non-engineers, and an assumption that anyone decent will become a manager. I’ve assigned Cluster A a mean competence of 1.10 and a standard deviation of 0.25, meaning that 95 percent of them are between 0.6 and 1.6.

Cluster B: Novices and part-timers (15%). These are the non-engineers who write scripts occasionally, software engineers in their first few months, interns and students. They program sometimes, but they generally aren’t defined as programmers. This cluster I’ve given a mean competence of 0.80 and a standard deviation of 0.25. I assign them to a separate cluster because (a) they generally aren’t evaluated or managed as programmers, and (b) they rarely spend enough time with software to become highly competent. They’re good at other things.

Cluster B isn’t especially relevant to my analysis, and it’s also the least well-defined. Like the Earth’s atmosphere, its outer perimeter has no well-defined boundary. Uncertainty about its size is also the main reason why it’s hard to answer questions about the “number of programmers” in the world. The percentage would increase (and so would the number of programmers) if the definition of “programmer” were loosened.

Cluster C: Self-managing engineers (10%). These are the engineers who either work in conditions of unusual autonomy (being successful freelancers, company owners, or employees of open-allocation companies) or who exert unusual efforts to control their careers, education and progress. This cluster has a larger mean competence and variance than the others. I’ve assigned it a mean of 1.5 and a variance of 0.35. Consequently, almost all of the engineers who get above 2.0 are in this cluster, and this is far from surprising: 2.0-level (multiplier) work is very rare, and impossible to get under typical management.

If we mix these three distributions together, we get the following profile for the software engineering world:

Skill Percentile
1.0     38.37
1.2     65.38
1.4     85.24
1.6     94.47
1.8     97.87
2.0     99.23
2.2     99.78
2.4     99.95
2.6     99.99

How accurate is this distribution? Looking at it, I think it probably underestimates the percentage of engineers in the tail. I’m around 1.7-1.8 and I don’t think of myself as a 97th-percentile programmer (probably 94-95th). It also says that only 0.77 percent of engineers are 2.0 or higher (I’d estimate it at 1 to 2 percent). I would be inclined to give Cluster C an exponential tail on the right, rather than the quadratic-exponential decay of a Gaussian, but for the purpose of this analysis, I think the Gaussian model is good enough.

Question 3: How much value does a software engineer produce? Marginal contribution isn’t a good measure of “business value”, because it’s too widely variable based on specifics (such company size) but I wouldn’t say that employment market value is a good estimate either, because there’s a surplus of good people who (a) don’t know what they’re actually worth, and (b) are therefore willing to work for low compensation, especially because a steady salary removes variance from compensation. Companies know that they make more (a lot more) on the most talented engineers than on average ones– if they have high-level work for them. The better engineer might be worth ten times as much, but only cost twice as much, as the average. So I think of it in terms of abstract business value: given appropriate work, how much expected value (ignoring variance) can that person deliver?

This quantity I’ve assumed to be exponential with regard to engineer skill, as described above: an increment of 1.0 is a 10-fold increase in abstract business value (ABV). Is this the right multiplier? It indicates that an increment of 0.3 is a doubling of ABV, or that a 0.1-increment is a 26 percent increase in ABV. In my experience, this is about right. For some skill-intensive projects, such as technology-intensive startups, the gradient might be steeper (a 20-fold difference between 2.0 and 1.0, rather than 10) but I am ignoring that, for simplicity’s sake.

The ABV of a 1.0 software engineer I’ve estimated at $125,000 per year in a typical business environment, meaning that it would be break-even (considering payroll costs, office space, and communication overhead) to hire one at a salary around $75,000. A 95th-percentile engineer (1.63) produces an ABV of $533,000 and a 99th-percentile engineer (1.96) produces an ABV of $1.14 million. These numbers are not unreasonable at all. (In fact, if they’re wrong, I’d bet on them being low.)

This doesn’t, I should say, mean that a 99th-percentile engineer will reliably produce $1.14 million per year, every year. She has to be assigned at-level, appropriate, work. Additionally, that’s an expected return (and the variance is quite high at the upper end). She might produce $4 million in one year under one set of conditions, and zero in another arena. Since I’m interested in the macroeconomic effect of increasing engineer autonomy, I can ignore variance and focus on mean expectancy alone. This sort of variance is meaningful to the individual (it’s better to have a good year than a bad one) and small companies, but the noise cancels itself out on the macroeconomic scale.

Putting it all together: with these estimates of the distribution of engineer competence, and the ABV estimates above, it’s possible to compute an expected value for a randomly-chosen engineer in each cluster:

Cluster A: $185,469 per year.

Cluster B: $89,645 per year.

Cluster C: $546,879 per year.

All software engineers: $207,236 per year.

Software engineers can evolve in all sorts of ways, and improved education or more longevity might change the distributions of the clusters themselves. That I’m not going to model, because there are so many possibilities. Nor am I going to speculate on macroeconomic business changes that would change the ABV figures. I’m going to focus on only one aspect of economic change, although there are others with additional positive effects, and that change is the evolution of an engineer from the micromanaged world of Cluster A to the autonomous, highly-productive one of Cluster C. (Upward movement within clusters is also relevant, and that strengthens my case, but I’m excluding it for now.) I’m going to assume that it takes 5 years of education and training for that evolution to occur, and assume an engineer’s career (with some underestimation here, justified by time-value of money) lasts 20 years, meaning there are 15 years to reap the benefits in full, plus two years to account for partial improvement during that five-year evolution.

What this means is that, for each engineer who evolves in this way, it generates $361,410 per year in value for 17 years, or $6.14 million per engineer. That is the objective benefit that accrues to society in engineer skill growth alone when a software engineer moves out of a typical subordinate context and into one like Valve’s open allocation regime. Generating $1 trillion in this way requires liberating 163,000 engineers, or a mere 2.2% of the total pool. That happens even if (a) the number of software engineers remains the same (instead of increasing due to improvements to the industry) and (b) other forms of technological growth, that would increase the ABV of a good engineer, stop, although it’s extremely unlikely that they will. Also, there are the ripple effects (in terms of advancing the state of the art in software engineering) of a world with more autonomous and, thus, more skilled engineers. All of these are substantial, and they improve my case even further, but I’m removing those concerns in the interest of pursuing a lower bound for value creation. What I can say, with ironclad confidence, is that the movement of 163,000 engineers into an open-allocation regime will, by improving their skills and, over the years, their output, generate $1 trillion at minimum. That it might produce $5 or $20 trillion (or much more, in terms of long-term effects on economic growth) in eventual value, through multiplicative “ripple” effects, is more speculatory. My job here was just to get to $1 trillion.

In simpler terms, the technological economy into which our society needs to graduate is one that requires software (not to mention mathematical and scientific) talent at a level that would currently be considered elite, and the only conditions that allow such levels of talent to exist are those of extremely high autonomy, as observed at Valve and pre-apocalyptic Google. Letting programmers direct their own work, and therefore take responsibility for their own skill growth is, multiplied over the vast number of people in the world who have to work with software, easily a trillion-dollar problem. It’s worth addressing now.

Programmers don’t need a union. We need a profession.

Every so often, I read a blog post or essay about the undesirable state of the software industry, and occasionally someone will suggest that we ought to unionize, in order to put an end to long hours and low pay compared to the value we produce. The argument is that, because software engineers are potentially worth millions per year to their employers, collective bargaining is the best way to capture more of this value. I disagree that a labor union is the way to go, because of the highly variable output of a software engineer, and the need for continuing education in this line of work. What we actually need is a profession.

Unions work best for commodity labor, and I use that term non-pejoratively. Commodity work is easily measurable and people can often be individually evaluated for performance. For example, a fishing boat operator is measured according to the quantity of fish she procures. A lot of very important work is commodity labor, so I don’t intend to disparage anyone by using that term. Commodity work can be unionized because there aren’t large and often intangible discrepancies in quality of output, and collective bargaining is often the best way to ensure that the workers are fairly compensated for the value they produce. Software is not commodity work, however. It’s difficult to measure quality, and the field is so specialized that engineers are not remotely interchangeable. When the work is difficult to measure and large disparities of quality exist, you have a situation in which a certain less-egalitarian (in the sense of allowing top performers to receive high compensation, because it’s essential to encourage people to improve themselves) and more self-regulatory structure is required: a profession.

The term professional is one of the most overused words in the English language, conflated often with white-collar work in general. If you work in an air-conditioned environment, someone will call you a professional. Most white-collar workers are not professionals, even in industries like investment banking, consulting, and management. Professionalism has nothing to do with social status, job category, or behavior (in the sense of an “unprofessional comment”). Rather, it’s all about a certain style of ethical structure, and the structure of a profession is nearly nonexistent in most white-collar industries (and under attack within the true professions).

What is a profession?

A profession is an attempt to impose global structure over a category of specialized, cognitively intensive work where the quality of output has substantial ramifications, but is difficult (if not, in the short term, impossible) to measure, giving ethics and competence primary importance. A profession is needed when it’s clear that not everyone can perform the work well, especially without specialized training. Here are some traits that signify the existence of a profession.

  1. Ethical obligations that supersede managerial authority. Professions have global ethical standards that the professional may not break, even under pressure from an employer. “Following orders” is not a defense. Therefore, the professional is both allowed and required to be autonomous in ethical judgment. It is rare for people to be fired, in true professions, because of disagreements with managers or political misfortune, but unethical behaviors are punished severely (in the worst cases, leading not only to termination but expulsion from the profession). This said, ethical demands on a profession may not match the common sense we have of ethics; they must be industry-specific. For example, attorneys’ stringent requirement to keep client information secret (attorney-client privilege) supersedes concerns regarding whether the client’s behavior itself is ethical; it’s not for the attorney to make that call.
  2. Weak power relationships. This is directly related to the above. In order to prevent the ethical lapses that are disturbingly common in non-professional work, and the abuses of power that are often behind those, professions deliberately weaken supervisory power relationships, with the intention of making it difficult for a manager to unilaterally fire an employee or damage his reputation. The result of this is that professionals answer to their companies or the profession directly. Professions attempt deliberately to be “bossless” so people will do what’s right rather than what’s politically expedient.
  3. Continuing improvement and self-direction as requirements. The professions assume change both in terms of what kinds of work will be valuable, and of what tools will be available to do it, and expect people to dedicate time to adapt to it without managerial permission or direction.
  4. Allowance for self-direction. Professionals are expected to place their career objectives at a high priority, since they serve the profession and their employers by becoming better at their work. Traditionally, metered work (that is, work directly relevant to the line of business) expectations in professions have been about 800 to 1200 hours per year, with the expectation that an equal amount will be spent on “off-meter” work such as attending conferences, reading journals, pursuing exploratory projects, mentoring others, and (at senior levels) looking for business or supporting the profession.
  5. Except in egregious cases, an agreement between employee and firm to serve each others’ interests, even after employment ends. Professionals are almost never fired, in the cinematic sense of the word. If a firm wants to get rid of someone, that person is encouraged to look for new employment on working time, and therefore allowed to represent himself as employed and retain an income during the job search. When people actually are fired, they’re typically offered a severance. In the rare case that a professional firm must lay people off or close an office for economic reasons, it gives as much notice as it can of the change, and announces the layoff in the public to protect departing employees’ images. That employee will almost always be given a positive reference (not the neutral “name and dates” reference that is often taken as negative) in exchange for his time. Efforts are made to protect the reputations of departing employees, who are expected to return the favor by speaking well of previous companies and colleagues. Additionally, employees who leave the firm are treated as alumni rather than deserters.

If I were to sum it up, I’d say that professionalism is about liberal service, and my use of the word “liberal” has nothing to do with political ideology. Rather, it’s the same etymology as “liberal arts”, where “liberal” means “pertaining to free people”. Professional service is that which is judged to be intellectually and ethically demanding enough that it should only be done by free people, not serfs or “wage slaves” who lack the autonomy for trust to be placed in them. The efforts that professions exert to curtail managerial authority are in place to prevent situations of ethical compromise, as well as political influences that might result in inferior work.

This is not to say that the professions are workplace utopias. I doubt that any workplace is perfectly clean, politically speaking, and the imperatives of a profession may improve behavior, but can’t change human nature wholesale. What I would argue, however, is that the global structure and protection of a profession protects the individual worker, to some degree, from local adversity. A doctor can be fired by his supervisor, but he’s still a doctor. Regular workers lose their credibility when their employment ends. Professionals don’t.

It’s all about the “Bozo Bit”.

The difference between a profession and regular work is the default assumption about an unknown person’s character, intellect, and competence. In typical industrial work, the “Bozo Bit” starts out in the “On” position, meaning that a typical worker is assumed to be stupid, treacherous, and useless. Since barriers to entry are low, the only thing that defines a worker is wanting money, and being willing to do unpleasant tasks in order to get it. There’s no respect for the average, individual member in such an industry; the default assumption is incompetence, ethical depravity, and childlike stupidity. One has to prove non-bozoism, which is usually established with a managerial role or other formal sanction: a “manager-equivalent” job title. (Non-professional companies typically make the work environment bad enough for non-managers that the default assumption is that anyone competent will become a manager.) Proving non-bozoism is often very difficult to do, because workers who attempt to do so are often fired for attempting to make their case instead of doing their assigned work. If you’re assumed to be a bozo, you probably won’t get the autonomy that would enable you to prove otherwise.

In this style of working environment, people without official managerial sanction (such as coveted job titles) have no credibility and therefore can be discarded on a whim. People with such decorations have only minimal power, because they can be deprived of these assets immediately, and often unilaterally. The result is an environment where all the power is held by the people who disburse funds, job titles, performance reviews, and authority, and in which the workers have none. Except at the top, there is no autonomy, because a manager can reduce the worker’s credibility to zero for any reason or no reason at all.

Professions clear the Bozo Bit (i.e. default it to “Off”) by making it difficult for incompetent people to enter, and punishing objectively unethical people severely. This gives people the assurance that a professional is highly likely to be both competent and ethical. In doing so, they create a high-trust zone. A person who completes medical school is a doctor and, unless the license is revoked because of a serious ethical lapse, has valuable expertise. No manager or employer can take that credibility away for political reasons.

Is software engineering a profession?

No. It’s not.

Professionals have the right to disobey managerial authority if they believe they are ethically right in doing so. On the other hand, a software engineer who refuses to cut corners or hide defects under managerial instruction has no recourse and will probably be fired.

Professionals take great pains to avoid disparaging others’ work in the public, preferring to keep feedback private. Many software engineers work in companies where performance reviews are part of an employee’s packet any time she wishes to transfer, and wherein a negative review can leave a person internally black-listed.

Professionals are allowed and expected to dedicate about half their working time to career development and continuing education. Many software engineers will get fired if they get caught in that.

Professionals have an external credibility that is independent of reporting to a specific manager or being employed at a specific firm, which enables them to serve the profession and the abstract ethical principles it values. The rest of workers don’t. They have managers who can unilaterally fire them from their current jobs (or, at least, damage their reputations through the aggressive performance review systems that are now the norm). Therefore, they have no option other than to serve their direct manager’s career goals, even if those diverge from the company’s needs or the employee’s long-term career objectives.

Most software engineers fall into the “rest of workers” category. They don’t have the right to buck managerial authority, and companies rarely allot time for their continuing education and career needs. Some have to fucking use vacation days to attend conferences!

Are the professions still relevant?

In Corporate America, the professions have been losing ground for at least half a century. Attorneys’ metered work demands have grown substantially, so that associates can’t participate in the off-meter work on which their careers depend, resulting in slowed development. It’s now typical that a “biglaw” associate is expected to log 2000 billable hours (metered work) and it’s therefore nearly impossible for her to build the business contacts necessary for transition into partnership. Academia and basic research have lost most of their funding and prominence. Professionalism remains in medicine for doctors, but the aggressively anti-professional insurance companies are working just as hard to deny care as doctors do to deliver it. Journalism is becoming more about entertainment, as objectivity is thrown to the wind by many participants. In sum, it seems that all of the professions have seen an erosion of status and, for lack of a better word, professionalism over the past few decades.

Why is the corporate world killing the professions? There’s an irreducible disparity between the professions and corporations. Professions are republics that exist to serve global, ethical objectives, and professionals happen to make money because the service they provide is valuable to others. Corporations are autocratic, amoral machines that exist to make money regardless of whether they provide any useful service, and the scope of service expected is local: take direct orders, don’t think for yourself. Workers serve their managers, who serve executives, and executives serve the short-term vacillations of the company’s stock price. A worker doesn’t have the right to prioritize work that she considers more beneficial (for the company, or the industry) than what her manager has told her to do. In the non-professional world, she doesn’t get to make that call.

Contemporary corporate leaders consider the professions to be archaic (I consider them to be archaic, but I won’t go there just yet) relics of a pre-capitalist era, unable to compete in a more ruthless business environment. This explains their scorn for government officials, journalists, and most especially professors who are “insulated” from the market. From a business person’s perspective, what the professions value is, if not supported by the market, then not worth preserving. (One exception: attorney-client privilege is a very strong ethical demand for lawyers, and business leaders are very glad for that one!) For this reason, they’re hostile toward the core ideas coming out of professions: the right to resist authority, the expectation of doing what is right rather than what is expedient, the diversion of half one’s working time into continuing growth and development, and the encouragement to seek external visibility that gives a person credibility independent of employment status. Finally, business leaders view professions, which intercede against managerial authority and thereby confer benefits on the professionals, as extortive institutions like guilds, or perhaps even command economies.

Why are the business leaders wrong? They are right about the superiority of market economies over command economies, but I don’t think that point is in contention anymore. They are wrong because they believe the professions can’t thrive in the market. That they can has been proven by a Seattle video game company called Valve, one of the only companies to truly professionalize software development, and one of the most successful software companies on Earth.

I’ve written a lot about Valve and its open allocation policy, under which engineers are trusted to move their desks to join another team as they wish. There’s no transfer process, and teams don’t have “headcount” constraints. Projects that no one wants to do don’t get done, unless they are genuinely important enough for the executives to create an incentive for it. Project supervision is driven by leadership and consensus rather than executive authority. What is open allocation about, philosophically? (For more about Valve’s open allocation, read this excellent blog post by Yanis Varoufakis.) It’s about professionalizing game and software development. Engineers are actually trusted to work for the company, rather than a manager who might use authority to divert their efforts toward the manager’s own goals (including preservation of the power relationship). Valve’s employees have an ethical commitment– make Valve better– they are expected to hold, but they have autonomy in how they do so. They live in the sort of high-trust zone that professions exist to create.

Not only do I believe that the professions can thrive in the market, but I think that professionalized software development will be superior to the industrial-age managed framework. Most of the progress in software comes from creative successes, which can be thought of as black swans. It is impossible to manage them into existence, and subordination will kill them before they are born. The best you can do, if you have managerial authority, is to use it as rarely as possible. At scale, you can be confident that people will come up with great ideas, and that enough of them will be profitable to justify the time and resources committed.

Ethics of a Technologist

Should software become a profession? I would say that the answer is a resounding “yes”. This emerging profession should not require expensive schooling. One of the things that’s great about programming is that no educational credential is required to enter, and I think we should keep it that way. However, I think that the idea of a global ethical structure is a sound one, and here are some thoughts.

I am using the term “technologist” for two reasons. The first is that I don’t want to limit my scope to software engineers, but to include a larger set of people who work on technology, such as designers and software testers and startup founders. The second is that the most important thing for a technologist to do, at a given time, may not be to write code. Sometimes, writing more code is not the answer: cleaning the existing stuff is better. As professionals, we should expect ourselves to do what is right for the problem we are trying to solve, not what allows us to write the most code.

Here are some basic, bedrock ethical principles that I think should be part of the technologist profession.

  1. Technologists do not create an inferior product for personal or single-party gain. We do not create “logic bombs” to extort clients or “back doors” that allow us to exploit systems we create. We do not create bugs for the purpose of “finding” them later. While we cannot ensure perfect software quality (it’s mathematically impossible to do so) we will not compromise on quality unless we believe it to be in the interest of all parties involved. We deliver the best product or service we can with the resources given to us, and if the resources given are not enough, we voice that concern.
  2. Technologists collaborate. We do not compete by harming another’s performance. We believe the world to be positive-sum and our industry is structurally cooperative. Therefore, to the extent that we compete, it’s in the direction of making ourselves better at our jobs, not making others worse. We also do not make technical decisions for the purpose of reducing another technologist’s ability to perform.
  3. Technologists do not disparage another’s performance to a non-technologist. Ever. All people with managerial authority are considered “non-technologists”, in the context of this item. Put simply: we don’t sell each other to outsiders. People who break this policy are fired for life from the profession. We handle our own affairs, period. If we need to remove a technologist from our team for reasons of incompetence or non-performance, we have the right to tell a non-technologist that this person cannot continue as a member of the team, and we are neither obligated nor allowed to give further reasons. We handle matters among ourselves, and do not attempt to use managerial warlords for personal gain. It’s up to us to form and disband teams, and to expel problematic members.
  4. Technologists choose their own leaders. It is not for non-technologist meddlers to decide who are the leaders of our groups. We shall not answer to stooges selected by executives. We choose our leaders, typically through democratic processes, and leaders who fail to serve the groups they are supposed to lead shall lose that distinction.
  5. Technologists work for the greater good of the company that employs them, the Technologist profession, and the greater good of the world. We serve the world first; our job is to make it better by improving technical processes, solving difficult but important problems, and advancing the state of science and rationality. Secondarily, we serve the Technologist profession and its values. As a tertiary concern, we act in the interests of our employers. We solve their technical problems and work to improve their infrastructure. Technologists are trusted to serve their employers directly, and any management that exists must be purely for providing guidance, not an irresistible authority. As a quaternary concern, technologists are expected and allowed to prioritize their own career growth, making themselves more valuable in the future.
  6. Technologists are trusted to work on any project that will have them. It is the right of the project owner to allow (or expel) members. Projects shall not be constrained by “head count” limitations set by non-technologists, who are to be deemed unqualified to make such distinctions.
  7. Technologists deliver on commitments they make. If a technologist cannot meet a commitment, he or she explains the cause as soon as possible, and attempts in earnest to address the shortfall. Additionally, it is not for managers to pressure technologists into making commitments that they would otherwise consider unreasonable. This “always deliver” policy only applies to freely made commitments, not any made under managerial duress.
  8. Technologists have the right to refuse work, unless it is of existential importance to the firm, or work that they have freely committed to doing. The only time a technologist can be required to complete a project he or she did not choose is when the company is at credible risk of failure or catastrophic loss if the work is not completed. Even then, it is best for the company to try to provide incentives (such as bonuses or promotions) before resorting to authority.
  9. Technologists have the right to inquire about other specialties without facing professional repercussions. Technologists are not fired or dismissed as “not a team player” when they voice concerns about a discrepancy between the work expected of them and the direction they want to take in their career, or the specialties into which they want to grow. Rather, technologists are encouraged to be direct and forthright about the specialties that interest them, and companies shall make reasonable efforts to allow them to find appropriate work.
  10. Making mistakes is tolerable; honesty about mistakes is required. The employment arena for a technologist must be one where people are not punished for making mistakes, or for discovering and revealing mistakes made by others. However, knowingly hiding a mistake, when it risks harm to others, is a grave ethical fault.
  11. Terminated employees recieve proper career support. Companies will sometimes need to fire a technologist, if he is unable to lead (as determined by other technologists) or follow within the context of the firm. When companies terminate, they always grant the right to represent oneself as employed during the job search, and provide a positive reference unless the employee was fired for a grave ethical breach. Companies that would be considered (reasonable person standard) able to afford severance pay it, in a large enough amount to cover the expected time for a job search. This does not apply to pre-funded startups (the risk is well-known) or companies in financial distress (that can’t afford it).
  12. Companies and investors do not create “no poach” or “do not hire” lists. Anyone who breaks these rules is censured severely and turned over to law enforcement. Investors are also disallowed from communicating any information about a technologist that might prevent him from getting further investment, unless a formal breach of ethics was involved.
  13. 1200 hours of metered work, per year, as a maximal obligation under normal circumstances. The general expectation is that a full-time technologist will deliver 400 to 1200 hours per year of metered work (work that is relevant to the direct line of business, and non-discretionary) per year. Companies have the right to increase metered work expectations in unusual circumstances. It is up to the company to decide what those circumstances are, but if they exist, they should be disclosed before an employment agreement begins (e.g. these are situations we consider abnormal, and these are the expectations we’ll have if they occur). Otherwise, the default assumption is that the company should expect no more than this level of direct dedication. Expecting anything more is to expect the employee to take on unreasonable risk of career stagnation. This 1200-hour standard shall be pro-rated for people who have part-time employment agreements.
  14. 800 hours of off-meter work as a minimal expectation of the technologist. “Full time” (2000 hours per year) technologists are usually expected to dedicate the remaining (800 or more) hours to off-meter work: continuing education, attending conferences, exploratory work and research, career-directed open source work, and pursuit of other specialties. This is not an obligation per se, but technologists should expect to spend this time if they wish to be maximally relevant, and companies shall be expected to allow time for it. This eliminates the excuse that many programmers have currently for stagnation and mediocrity, which is that their bosses won’t allocate time for growth. If we become a technologist profession, we will make allowance for such time an inflexible pillar.

This is just a starting set of principles. I’m sure I will remember more, but my intent here is to indicate what a technologist profession will look like.

One final thought: selectivity

My word count’s getting high, indicating that I should wrap this up, but I’d like to address one other concern: Who do we let in? What’s the barrier to entry? Here are my initial thoughts.

No formal education is required to become a technologist. Technologists are expected to have the breadth of knowledge of an average college graduate from a top-50 university, but how they get it is up to them. Age, socioeconomic status, educational matriculations, and national origin are irrelevant. Anyone with the competence can be a technologist, but we shall set the bar for competence very high. I think the best model for this is actuarial science in the United States, where progress is exam-based. Classes and study guides are available, but not required. Additionally, the profession requires and expects that associates will dedicate a significant fraction of their working time to studying for the exams.

There are a few tweaks I’d make to this system. The first is that, instead of a linear series of eight or ten exams, there shall be a larger number, with some being elective. The courses on technological ethics, basic mathematics, code quality and scientific thought would generally be required, for example. Machine learning, startup economics, and compilers courses would be optional. The second is that some courses would require code, and most courses would have a non-exam option whereby, for example, a high-quality open-source contribution could be substituted for a typical exam. (This is because, while most people who are “not a good test taker” are just lazy, the condition does exist and alternative evaluation is appropriate for making the profession maximally inclusive of talent.)

The purpose of these exams would be to provide an alternative path to success and true independent credibility for technologists, and to deprive parochial managers of the ability to reduce a technologist’s credibility to zero. They would not necessarily be required for a technologist to have employment, but they would be designed to be difficult and relevant enough to increase a professional’s employability dramatically– so dramatically as to give the technologist true independence of managerial authority.

That is all I have to say on this matter tonight, so I yield the floor.

The world sucks at finding the right work for engineers.

This is directly in response to Matt Aimonetti’s “Engineers Suck at Finding the Right Jobs“, because I disagree that the problem resides solely with engineers. Rather, I think the problem is bilateral. An equally strong argument could be made that there’s an immense wealth of engineering talent (or, at least, potential) out there, but that our contemporary business leadership lacks the vision, creativity, and intelligence to do anything with it.

Don’t get me wrong: I basically agree with what he is saying. Most software engineers do a poor job at career management. A major part of the problem is that the old-style implicit contract between employers and employees has fallen to pieces, and people who try to follow the old rules will shortchange themselves and fail in their careers. In the old world, the best thing for a young person to do was to take a job– any job– at a reputable company and just be seen around the place, and eventually graduate into higher quality of work. Three to five years of dues-paying grunt work (that had little intrinsic career-building value, but fulfilled a certain social expectation) was the norm, but this cost only had to be paid once. The modern world is utterly different. Doing grunt work does nothing for your career. There are people who get great projects and advance quickly, and others who get bad projects and never get out of the mud. Our parents lived in a world where “90 percent is showing up”, whereas we live in one where frequent job changes are not only essential to a good career, but often involuntary.

Software engineering is full of “unknown unknowns”, which means that most of us have a very meager understanding of what we don’t know, and what our shortcomings are. We often don’t know what we’re missing. It’s also an arena in which the discrepancy between the excellent and the mediocre isn’t a 20 to 40 percent increase, but a factor of 2 to 100. Yet to become excellent, an engineer needs excellent work, and there isn’t much of that to go around, because most managers and executives have no vision. In fact, there’s so little excellent work in software that what little there is tends to be allocated as a political favor, not given to those with the most talent. The most important skill for a software engineer to learn, in the real world, is how to get allocated to the best projects. In fact, what I would say distinguishes the most successful engineers is that they develop, at an early age, the social skills to say “no” to career-negative grunt work without firing oneself in the process. That’s how they take the slack out of their careers and advance quickly.

That’s not the same as picking “the right jobs”, because engineers don’t actually apply to specific work sets when they seek employment, but to companies and managers. Bait-and-switch hiring practices are fiendishly common, and many companies are all-too-willing to allocate undesirable and even pointless work to people in the “captivity interval”, which tends to span from the 3rd to the 18th month of employment, at which point leaving will involve a damaging short job tenure on the resume. (At less than 3 months, the person has the option of omitting that job, medium-sized gaps being less damaging than short-term jobs, which suggest poor performance.) I actually don’t think there’s any evidence to indicate that software engineers do poorly at selecting companies. Where I think they are abysmal is at the skill of placing themselves on high-quality work once they get into these companies.

All of this said, this matter raises an interesting question: why is there so much low-quality work in software? I know this business well enough to know that there aren’t strong business reasons for it to be that way. High-quality work is, although more variable, much more profitable in general. Companies are shortchanging themselves as much as their engineers by having a low-quality workload. So why is there so little good work to go around?

I’ve come to the conclusion that most company’s workloads can be divided into four quadrants based on two variables. The first is whether the work is interesting or unpleasant. Obviously, “interestingness” is subjective, so I tend to assume that work should be called interesting if there is someone out there who would happily do it for no more (and possibly less) than a typical market salary. Some people don’t have the patience for robotics, but others love that kind of work, so I classify it in the “interesting” category, because I’m fairly confident that I could find someone who would love to do that kind of work. For many people, it’s “want-to” work. On the other hand, the general consensus is that there’s a lot of work that very few people would do, unless paid handsomely for it. That’s the unpleasant, “have-to” work.

The second variable is whether the work is essential or discretionary. Essential work involves a critical (and often existential) issue for the company. If it’s not done, and not done well, the company stands to lose a lot of money: millions to billions of dollars. Discretionary work, on the other hand, isn’t in the company’s critical path. It tends to be exploratory work, or support work that the firm could do without. For example, unproven research projects are discretionary, although they might become essential later on.

From these two variables, work can be divided into four quadrants:

Interesting and Essential (1st Quadrant): an example would be Search at Google. This work is highly coveted. It’s both relevant and rewarding, so it benefits an employee’s internal and external career goals. Sadly, there’s not a lot of this in most companies, and closed-allocation companies make it ridiculously hard to get it.

Unpleasant and Essential (2nd Quadrant): grungy tasks like maintenance of important legacy code. This is true “have-to” work: it’s not pleasant, but the company relies on it getting done. Boring or painful work generally doesn’t benefit an employee’s external career, so well-run companies compensate by putting bonuses and internal career benefits (visibility, credibility, promotions) on it: a market solution. These are “hero projects”.

Interesting and Discretionary (3rd Quadrant): often, this takes the form of self-directed research projects and is the domain of “20% time” and “hack days”. This tends to be useful for companies in the long term, but it’s not of immediate existential importance. Unless the project were so successful as to become essential, few people would get promoted based on their contributions in this quadrant. That said, a lot of this work has external career benefits, because it looks good to have done interesting stuff in the past, and engineers learn a lot by doing it.

Unpleasant and Discretionary (4th Quadrant): this work doesn’t look good in a promotion packet, and it’s unpleasant to perform. This is the slop work that most software engineers get because, in traditional managed companies, they don’t have the right to say “no” to their managers. The business value of this work is minimal and the total value (factoring in morale costs and legacy) is negative. 4th-Quadrant work is toxic sludge that should be avoided.

One of the reasons that I think open allocation is the only real option is that it eliminates the 4th-Quadrant work that otherwise dominates a corporate atmosphere. Under open allocation, engineers vote with their feet and tend to avoid the 4th-Quadrant death marches.

The downside of open allocation, from a managerial perspective, is that the non-coercive nature of such a regime means they have to incent people to work on 2nd-Quadrant work, often with promotions and large (six- or seven-figure) bonuses. It seems expensive. Closed allocation enables managers to get the “have-to” work done cheaply, but there’s a problem with that. Under closed allocation, people who are put on these unpleasant projects often get no real career compensation, because management doesn’t have to give them any. So the workers put on such projects feel put-upon and do a bad job of it. If the work is truly 2nd-Quadrant (i.e. essential) the company cannot afford to have it done poorly. It’s better to pay for it and get high quality than to coerce people into it and get garbage.

The other problem with closed allocation is that it eliminates the market mechanic (workers voting with their feet) that allows this quadrant structure to become visible at all, which means that management in closed-allocation companies won’t even know when it has a 4th-Quadrant project. The major reason why closed-allocation companies load up on the toxic 4th-Quadrant work is because they have no idea that it’s even there, nor how to get rid of it.

There’s no corporate benefit to 4th-Quadrant work. So what incentive is there to generate it? Middle management is to blame. Managers don’t care whether the work is essential or discretionary, because they just want the experience of “leading teams”. They’re willing to work on something less essential, where there’s less competition to lead the project (and also a higher chance of keeping one’s managerial role) because their careers benefit either way. They can still say they “led a team of 20 people”, regardless of what kind of work they actually oversaw. Middle managers tend to what little interesting stuff these discretionary projects have for themselves, placing themselves in the 3rd-Quadrant, while leaving the 4th-Quadrant work to their reports.

This is the essence of what’s wrong with corporate America. Closed allocation generates pointless work that (a) no one wants to do, and (b) provides no real business value to the company. It’s a bilateral lose-lose for the company and workers, existing only because it suits the needs of middlemen.

It’s common wisdom in software that 90 to 95 percent of software engineers are depressingly mediocre. I don’t know what the percentage is, but I find that to be fairly accurate, at least in concept. The bulk of software engineers are bad at their jobs. I disagree that this mediocrity is intrinsic. I think it’s a product of bad work environments, and the compounding effect of bad projects and discouragement over time. The reason there are so many incompetent software engineers out there is that the work they get is horrible. It’s not only that they lack the career-management skills to get better work; it’s also that good work isn’t available to them when they start out, and it becomes even less available over time as their skills decline and their motivation and energy levels head toward the crapper.

I don’t see any intrinsic reason why the world can’t have ten, or even a hundred, times as many competent software engineers as it has now, but the dumbed-down corporate environment that most engineers face will block that from coming to fruition.

There’s an incredible amount of potential engineering talent out there, and for the first time in human history, we have the technology to turn it into gold. Given this, why is so much of it being squandered?