Programmer autonomy is a $1 trillion issue.

Sometimes, I think it’s useful to focus on trillion-dollar problems. While it’s extremely difficult to capture $1 trillion worth of value, as made evident by the nonexistence of trillion-dollar companies, innovations that create that much are not uncommon, and these trillion-dollar problems often have not one good idea, but a multitude of them to explore. So a thought experiment that I sometimes run is, “What is the easiest way to generate $1 trillion in value?” Where is the lowest-hanging trillion dollars of fruit? What’s the simplest, most obvious thing that can be done to add $1 trillion of value to the economy?

I’ve written at length about the obsolescence of traditional management, due to the convex input-output relationship in modern, creative work, making variance-reductive management counter-productive. I’ve also shared a number of thoughts about Valve’s policy of open allocation, and the need for software engineers to demand the respect accorded to traditional professions: sovereignty, autonomy, and the right to work directly for the profession without risk of the indirection imposed by traditional hierarchical management. So, for this exercise, I’m going to focus on the autonomy of software engineers. How much improvement will it take to add $1 trillion in value to the economy?

This is an open-ended question, so I make it more specific: how many software engineers would we have to give a high degree (open allocation) of autonomy in order to add $1 trillion? Obviously, the answer is going to depend on the assumptions that are made, so I’ll have to answer some basic questions. Because there’s uncertainty in all of these numbers, the conclusion should be treated as an estimate. However, when possible, I’ve attempted to make my assumptions as conservative as possible. Therefore, it’s quite plausible that the required number of engineers is actually less than half the number that I’ll compute here.

Question 1: How many software engineers are there? I’m going to restrict this analysis to the developed world, because it’s hard to enforce cultural changes globally. Data varies, but most surveys indicate that there are about 1.5 million software engineers in the U.S. If we further assume the U.S. to be one-fifth of “the developed world”, we get 7.5 million full-time software engineers in the world. This number could probably be increased by a factor of two by loosening the definition of “software engineer” or “developed world”, but I think it’s a good working number. The total number of software engineers is probably two to three times that.

Question 2: What is the distribution of talent among software engineers? First, here’s the scale that I developed for assessing the skill and technical maturity of a software engineer. Defining specific talent percentiles requires specifying a population, but I think the software engineer community can be decomposed into the following clusters, differing according to the managerial and cultural influences on their ability and incentive to gain competence.

Cluster A: Managed full-time (75%). These are full-time software engineers who, at least nominally, spend their working time either coding or on software-related problems (such as site reliability). If they were asked to define their jobs, they’d call themselves programmers or system administrators: not meteorologists or actuaries who happen to program software. Engineers in this cluster typically work on large corporate codebases and are placed in a typical hierarchical corporate structure. They develop an expertise within their defined job scope, but often learn very little outside of it. Excellence and creativity generally aren’t rewarded in their world, and they tend to evolve into the stereotypical “5:01 developers”. They tend to plateau around 1.2-1.3, because the more talented ones are usually tapped for management before they would reach 1.5. Multiplier-level contributions for engineers tend to be impossible to achieve in their world, due to bureaucracy, limited environments, project requirements developed by non-engineers, and an assumption that anyone decent will become a manager. I’ve assigned Cluster A a mean competence of 1.10 and a standard deviation of 0.25, meaning that 95 percent of them are between 0.6 and 1.6.

Cluster B: Novices and part-timers (15%). These are the non-engineers who write scripts occasionally, software engineers in their first few months, interns and students. They program sometimes, but they generally aren’t defined as programmers. This cluster I’ve given a mean competence of 0.80 and a standard deviation of 0.25. I assign them to a separate cluster because (a) they generally aren’t evaluated or managed as programmers, and (b) they rarely spend enough time with software to become highly competent. They’re good at other things.

Cluster B isn’t especially relevant to my analysis, and it’s also the least well-defined. Like the Earth’s atmosphere, its outer perimeter has no well-defined boundary. Uncertainty about its size is also the main reason why it’s hard to answer questions about the “number of programmers” in the world. The percentage would increase (and so would the number of programmers) if the definition of “programmer” were loosened.

Cluster C: Self-managing engineers (10%). These are the engineers who either work in conditions of unusual autonomy (being successful freelancers, company owners, or employees of open-allocation companies) or who exert unusual efforts to control their careers, education and progress. This cluster has a larger mean competence and variance than the others. I’ve assigned it a mean of 1.5 and a variance of 0.35. Consequently, almost all of the engineers who get above 2.0 are in this cluster, and this is far from surprising: 2.0-level (multiplier) work is very rare, and impossible to get under typical management.

If we mix these three distributions together, we get the following profile for the software engineering world:

Skill Percentile
1.0     38.37
1.2     65.38
1.4     85.24
1.6     94.47
1.8     97.87
2.0     99.23
2.2     99.78
2.4     99.95
2.6     99.99

How accurate is this distribution? Looking at it, I think it probably underestimates the percentage of engineers in the tail. I’m around 1.7-1.8 and I don’t think of myself as a 97th-percentile programmer (probably 94-95th). It also says that only 0.77 percent of engineers are 2.0 or higher (I’d estimate it at 1 to 2 percent). I would be inclined to give Cluster C an exponential tail on the right, rather than the quadratic-exponential decay of a Gaussian, but for the purpose of this analysis, I think the Gaussian model is good enough.

Question 3: How much value does a software engineer produce? Marginal contribution isn’t a good measure of “business value”, because it’s too widely variable based on specifics (such company size) but I wouldn’t say that employment market value is a good estimate either, because there’s a surplus of good people who (a) don’t know what they’re actually worth, and (b) are therefore willing to work for low compensation, especially because a steady salary removes variance from compensation. Companies know that they make more (a lot more) on the most talented engineers than on average ones– if they have high-level work for them. The better engineer might be worth ten times as much, but only cost twice as much, as the average. So I think of it in terms of abstract business value: given appropriate work, how much expected value (ignoring variance) can that person deliver?

This quantity I’ve assumed to be exponential with regard to engineer skill, as described above: an increment of 1.0 is a 10-fold increase in abstract business value (ABV). Is this the right multiplier? It indicates that an increment of 0.3 is a doubling of ABV, or that a 0.1-increment is a 26 percent increase in ABV. In my experience, this is about right. For some skill-intensive projects, such as technology-intensive startups, the gradient might be steeper (a 20-fold difference between 2.0 and 1.0, rather than 10) but I am ignoring that, for simplicity’s sake.

The ABV of a 1.0 software engineer I’ve estimated at $125,000 per year in a typical business environment, meaning that it would be break-even (considering payroll costs, office space, and communication overhead) to hire one at a salary around $75,000. A 95th-percentile engineer (1.63) produces an ABV of $533,000 and a 99th-percentile engineer (1.96) produces an ABV of $1.14 million. These numbers are not unreasonable at all. (In fact, if they’re wrong, I’d bet on them being low.)

This doesn’t, I should say, mean that a 99th-percentile engineer will reliably produce $1.14 million per year, every year. She has to be assigned at-level, appropriate, work. Additionally, that’s an expected return (and the variance is quite high at the upper end). She might produce $4 million in one year under one set of conditions, and zero in another arena. Since I’m interested in the macroeconomic effect of increasing engineer autonomy, I can ignore variance and focus on mean expectancy alone. This sort of variance is meaningful to the individual (it’s better to have a good year than a bad one) and small companies, but the noise cancels itself out on the macroeconomic scale.

Putting it all together: with these estimates of the distribution of engineer competence, and the ABV estimates above, it’s possible to compute an expected value for a randomly-chosen engineer in each cluster:

Cluster A: $185,469 per year.

Cluster B: $89,645 per year.

Cluster C: $546,879 per year.

All software engineers: $207,236 per year.

Software engineers can evolve in all sorts of ways, and improved education or more longevity might change the distributions of the clusters themselves. That I’m not going to model, because there are so many possibilities. Nor am I going to speculate on macroeconomic business changes that would change the ABV figures. I’m going to focus on only one aspect of economic change, although there are others with additional positive effects, and that change is the evolution of an engineer from the micromanaged world of Cluster A to the autonomous, highly-productive one of Cluster C. (Upward movement within clusters is also relevant, and that strengthens my case, but I’m excluding it for now.) I’m going to assume that it takes 5 years of education and training for that evolution to occur, and assume an engineer’s career (with some underestimation here, justified by time-value of money) lasts 20 years, meaning there are 15 years to reap the benefits in full, plus two years to account for partial improvement during that five-year evolution.

What this means is that, for each engineer who evolves in this way, it generates $361,410 per year in value for 17 years, or $6.14 million per engineer. That is the objective benefit that accrues to society in engineer skill growth alone when a software engineer moves out of a typical subordinate context and into one like Valve’s open allocation regime. Generating $1 trillion in this way requires liberating 163,000 engineers, or a mere 2.2% of the total pool. That happens even if (a) the number of software engineers remains the same (instead of increasing due to improvements to the industry) and (b) other forms of technological growth, that would increase the ABV of a good engineer, stop, although it’s extremely unlikely that they will. Also, there are the ripple effects (in terms of advancing the state of the art in software engineering) of a world with more autonomous and, thus, more skilled engineers. All of these are substantial, and they improve my case even further, but I’m removing those concerns in the interest of pursuing a lower bound for value creation. What I can say, with ironclad confidence, is that the movement of 163,000 engineers into an open-allocation regime will, by improving their skills and, over the years, their output, generate $1 trillion at minimum. That it might produce $5 or $20 trillion (or much more, in terms of long-term effects on economic growth) in eventual value, through multiplicative “ripple” effects, is more speculatory. My job here was just to get to $1 trillion.

In simpler terms, the technological economy into which our society needs to graduate is one that requires software (not to mention mathematical and scientific) talent at a level that would currently be considered elite, and the only conditions that allow such levels of talent to exist are those of extremely high autonomy, as observed at Valve and pre-apocalyptic Google. Letting programmers direct their own work, and therefore take responsibility for their own skill growth is, multiplied over the vast number of people in the world who have to work with software, easily a trillion-dollar problem. It’s worth addressing now.

14 thoughts on “Programmer autonomy is a $1 trillion issue.

  1. A very interesting read. I especially liked the quantitative approach. I need to go through your other articles, and read up more thoroughly on your ideas, but in the meantime, here is a quick-fire response (probably wrong in many many ways):

    Firstly, the world is complex, and there are additional factors which can be considered other than open vs closed allocation.

    Secondly, open vs closed allocation affects more than just productivity. It also (I believe) affects reuse. Not coincidentally, I believe that reuse (both in terms of software components and also in terms of knowledge and expertise) is one of the primary sources of productivity nonlinearity in our discipline.

    Furthermore, our discipline is still very young. We do not understand it very well. A sign of increasing maturity in the discipline will be the emergence of a vocabulary that facilitates it’s fragmentation into sub-discipline. What is good for a front-end engineer working in a consumer startup is not necessarily good for an embedded software engineer working on safety-critical medical equipment.

    Finally, we will not be able to advance the state of our discipline unless we frame our enquiry in the right terms. I am far from certain what those terms are, but it is highly likely that it will involve a combination of human psychology and engineering pragmatics.

    I am looking forward to reading more of your thoughts on these and other issues.

  2. Excellent analysis- and very entertaining. It is hard to choose the right assumptions for this sort of stuff- but I feel like you hit it on the head with a lot of them.

  3. Maybe, the real problem is human nature itself. Have you considered the possibility that most people who gain power and make decisions in corporations are driven by the small-minded need to abuse and hurt others? I seriously doubt that most decision-makers are driven by anything rational like progress or making things better.

    • True executives (people who are capable of multiplier effects across a company, and have the title or recognition that grants them appropriate respect or authority) tend to care most about (a) their own image, and (b) making money. As a group, they don’t have a strong desire to hurt other people, although most of them don’t care if they do. They’re selfish, not sadistic.

      Middle managers (and HR) are cops hired by executives on the belief that workers will steal from the company (this includes slacking or diverting effort to their own careers) if not watched.

      Some cops are good, some are corrupt, and some are sadists. No one would argue that there aren’t dirty cops, but in a well-run society, policemen don’t unilaterally make the laws.

  4. I love how thoroughly you addressed this issue.

    I believe even in a well intentioned world, developers of software are systematically undervalued because so few people understand great software and its potential.

    I think it is pointless to talk about mal-intent scenarios that have been talked about in the comments.

  5. Somewhat off topic, but since this post is about creating value, I will ask my question:

    It has been the assumption that a person who talks about a change the world vision or who actually has a change the world vision is good, while someone who says “I just want to get rich” is assumed to be a pure slimeball. Yet I have noticed the following odd anomalies:

    There have been some tech entrepreneurs who speak about changing the world or who even truly want to change the world that engage in evil behavior ( see some of your anti VC istan essays).

    On the other hand, there are some proprietary trading types and pro poker types who are very compassionate and ethical individuals in everyday life. Conventional wisdom will view the person who wants to create value in this world or who speaks of doing so as good and the person who says “I just want to be rich” or who is just in it for the money as evil, yet there are people whose genuine macroscopic goal or macroscopic activity does not correlate at all with how they treat people.

    I truly have seen and heard of some “visionary change the world” tech entrepreneurs who not only speak of changing the world, but truly want to do so, who are in some ways evil slimeballs, (abusive with employees, stealing stock employee equity, ethical lapses).

    I have also known of some proprietary trader types who are only in it for the money and talk like Gordon Gecko yet in person are actually very great people. Some of these millionaire traders who just want to get rich engage in saintly behavior such as mentoring random people who happen to email them, giving huge bonuses to employees and sharing their wealth with friends and family. One trader, Richard Dennis was famous for hiring and mentoring random blue collar people.

    My questions are:

    1.Why is it that some visionary entrepreneurs who truly do want to change the world treat people horribly in real life, while some “Just want to get rich” trader types are in some ways saintly individuals when dealing with people in real life?

    2. Why is it that paradoxically the zero sum parasitic proprietary trading world is less cutthroat in some ways then the “value creating” traditional technology entrepreneur world?

    • 1. Why are there so many slimy “change the world” types?

      A lot of these supposed idealists think their vision (read: ego) entitles them to treat other people badly. Little people are just details to them. The real idealists have completely inadvertently given these talentless Steve Jobs wannabes a new language to frame their egotistical aspirations.

      “Vision” doesn’t make one a good person. You know how had quite an excess of vision? Hitler. (No, I am not invoking him to be inflammatory. His vision was disgusting but no one would argue that he wasn’t “visionary” in a value-neutral sense of the word.)

      2. Why are many traders decent people, despite stereotype?

      I think that goes back a couple generations. When my parents were growing up, you *weren’t allowed to admit* that you worked for money. People did, but admitting to the fact was something only working-class people did. Middle-class people worked for higher principles; they were *professionals* in the true sense of the word. Admitting that money was a motivation for work, or that getting rich was one’s goal, was just not socially acceptable before the Reagan Era. This was also a time when inequality wasn’t the beast that it was now– college professors actually could own houses in major cities– so people did actually choose their work according to what interested them rather than what paid, because the differences were more like 20 percent than 200.

      Traders admit that they work to make money, which is still not socially acceptable in all circles. In fact, making money is the exact game they play, and it’s not fun if you’re not able to make money _personally_ by doing it.

      Now, as for why traders tend to be a better set of people (although I prefer quants) than VC-istan entrepreneurs, the answer is that the latter are under a False Poverty Effect. When people feel poor and victimized (and having very rich people as your bosses can have this effect) they start to feel like it’s OK to cheat and rob people. VC-istan founders who haven’t made it yet live in false poverty: they had to fight their investors tooth-and-nail to get a $150,000 salary (after 12 months with no salary and maxed-out credit cards) when other people their age are making $500k at hedge funds and buying houses, and it pisses them off. Those who hit and join Those Who Have Completed An Exit can no longer convince themselves of poverty, but by this time, they’ve been steeped in an unethical culture.

      Traders are paid very well in large part *because* their employers know for experience that the False Poverty Effect must be avoided, and they tend to be more ethical because this effect tends not to exist in them.

  6. Pingback: Fundamental Subordinate Dishonesty « Michael O.Church

  7. Pingback: Gervais / MacLeod 21: Why Does Work Suck? | Michael O. Church

  8. Pingback: Don Draper’s firing and Silicon Valley | Michael O. Church

  9. Pingback: What’s a mid-career software engineer actually worth? Try $779,000 per year as a lower bound. | Michael O. Church

  10. Pingback: Can tech fix its broken culture? | Michael O. Church

  11. Pingback: Software’s management issues | Michael O. Church

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s