Why programmers can’t make any money: dimensionality and the Eternal Haskell Tax

To start this discussion, I’m going to pull down a rather dismal tweet from Chris Allen (@bitemyapp):

For those who don’t know, Haskell is a highly productive, powerful language that enables programmers (at least, the talented ones) to write correct code quickly: at 2 to 5 times the development speed of Java, with similar performance characteristics, and fewer bugs. Chris is also right on the observation that, in general, Haskell jobs don’t pay as well. If you insist on doing functional programming, you’ll make less money than people who sling C++ at banks with 30-year-old codebases. This is perverse. Why would programmers be economically penalized for using more powerful tools? Programmers are unusual, compared to rent-seeking executives, in actually wanting to do their best work. Why is this impulse penalized?

One might call this penalty “the Haskell Tax” and, for now, that’s what I’ll call it. I don’t think it exists because companies that use Haskell are necessarily cheaper or greedier than others. That’s not the case. I think the issue is endemic in the industry. Junior programmer salaries are quite high in times like 2014, but the increases for mid-level and senior programmers fall short of matching their increased value to the business, or even the costs (e.g. housing, education) associated with getting older. The only way a programmer can make money is to develop enough of a national reputation that he or she can create a bidding war. That’s harder to do for one who is strongly invested in a particular language. It’s not Haskell’s fault. There’s almost certainly a Clojure Tax and an Erlang Tax and a Scala Tax.

Beyond languages, this applies to any career-positive factor of a job. Most software jobs are career-killing, talent-wasting graveyards and employers know this, so when there’s a position that involves something interesting like machine learning, green-field projects, and the latest tools, they pay less. This might elicit a “well, duh” response, insofar as it shouldn’t be surprising that unpleasant jobs pay well. The reason this is such a disaster is because of its long-term effect, both on programmers’ careers and on the industry. Market signals are supposed to steer people toward profitable investment, but in software, it seems to fall the other way. Work that helps a programmer’s career is usually underpaid and, under the typical awfulness of closed allocation, jealously guarded, politically allocated, and usually won through unreasonable sacrifice.

Why is the Haskell Tax so damning?

As I said, the Haskell Tax doesn’t apply only to Haskell. It applies to almost all software work that isn’t fish-frying. It demolishes upper-tier salaries. One doesn’t, after all, get to be an expert in one’s field by drifting. It takes focus, determination, and hard work. It requires specialization, almost invariably. With five years of solid experience, a person can add 3 to 50 times more value than the entry-level grunt. Is she paid for that? Almost never. Her need to defend her specialty (and refuse work that is too far away from it) weakens her position. If she wants to continue in her field, there are a very small number of available jobs, so she won’t have leverage, and she won’t make any money. On the other hand, if she changes specialty, she’ll lose a great deal of her seniority and leverage, she’ll be competing with junior grunts, and so she won’t make any money either. It’s a Catch-22.

This puts an economic weight behind the brutality and incivility of closed allocation. It deprives businesses of a great deal of value that their employees would otherwise freely add. However, it also makes people less mobile, because they can’t move on to another job unless a pre-defined role exists matching their specialties. In the long run, the effect of this is to provide an incentive against expertise, to cause the skills of talented programmers to rot, and to bring the industry as a whole into mediocrity.

Code for the classes and live with the masses. Code for the masses and live… with the masses.

Artists and writers have a saying: sell to the masses, and live with the classes; sell to the classes, and live with the masses. That’s not really a statement about social class as about the low economic returns of high-end work. Poets don’t make as much money as people writing trashy romance novels. We might see the Haskell Tax as an extension of this principle. Programmers who insist on doing only the high-end work (“coding for the classes”) are likely to find themselves either often out of work, or selling themselves at a discount.

Does this mean that every programmer should just learn what is learned in 2 years at a typical Java job, and be done with it? Is that the economically optimal path? The “sell to the masses” strategy is to do boring, line-of-business, grunt work. Programmers who take that tack still live with the masses. That kind of programming (parochial business logic) doesn’t scale. There’s as much work, for the author, in writing a novel for 10 people as 10 million; but programmers don’t have that kind of scalability, and the projects where there are opportunities for scaling, growth, and multiplier-type contributions are the “for the classes” projects that every programmer wants to do (we already discussed why those don’t pay). So, programming for the masses is just as much of a dead end, unless they can scale up politically– that is, become a manager. At that point, they can sell code, but they don’t get to create it. They become ex-technical, and ex-technical management (with strongly held opinions, once right but now out of date) can be just as suboptimal as non-technical management.

In other words, the “masses” versus “classes” problem looks like this, for the programmer: one can do high-end work and be at the mercy of employers because there’s so little of it to go around, or to low-end commodity work that doesn’t really scale. Neither path is going to enable her to buy a house in San Francisco.

Dimensionality

One of the exciting things about being a programmer is that the job always changes. New technologies emerge, and programmers are expected to keep abreast of them even when their employers (operating under risk aversion and anti-intellectualism) won’t budget the time. What does it mean to be a good programmer? Thirty years ago, it was enough to know C and how to structure a program logically. Five years ago, a software engineer was expected to know a bit about a Linux, MySQL, a few languages (Python, Java, C++, Shell) and the tradeoffs among them. In 2014, the definition of “full stack” has grown to the point that almost no one can know all of it. Andy Shora (author of the afore-linked essay) puts it beautifully, on the obnoxiousness of the macho know-it-all programmer:

I feel the problem for companies desperate to hire these guys and girls, is that the real multi-skilled developers are often lost in a sea of douchebags, claiming they know it all.

Thirty years ago, there was a reasonable approximation of a linear ordering on programmer skill. If you could write a C compiler, understood numerical stability, and could figure out how to program in a new language or for a new platform by reading the manuals, you were a great programmer. If you needed some assistance and often wrote inefficient algorithms, you were either a junior or mediocre. In 2014, it’s not like that at all; there’s just too much to learn and know! I don’t know the first thing, for example, about how to build a visually appealing casual game. I don’t expect that I’d struggle as much with graphics as many do, because I’m comfortable with linear algebra, and I would probably kill it when it comes to AI and game logic, but the final polish– the difference between Candy Crush and an equivalent but less “tasty” game– would require someone with years of UI/UX experience.

The question of, “What is a good programmer?”, has lost any sense of linear ordering. The field is just too vast. It’s now an N-dimensional space. This is one of the things that makes programming especially hostile to newcomers, to women, and non-bullshitters of all stripes. The question of which of those dimensions matter and which don’t is political, subjective, and under constant change. One year, you’re a loser if you don’t know a scripting language. The next, you’re a total fuckup if you can’t explain what’s going on inside the JVM. The standards change at every company and frequently, leaving most people not only at a loss regarding whether they are good programmers, but completely without guidance about how to get there. This also explains the horrific politics for which software engineering is (or, at least, ought to be) notorious. Most of the “work” in a software company is effort spent trying to change the in-house definition of a good programmer (and, to that end, fighting incessantly over tool choices).

I don’t think that dimensionality is a bad thing. On the contrary, it’s a testament to the maturity and diversity of the field. The problem is that we’ve let anti-intellectual, non-technical businessmen walk in and take ownership of our industry. They demand a linear ordering of competence (mostly, for their own exploitative purposes). It’s the interaction between crass commercialism and dimensionality that causes so much pain.

Related to this is the Fundamental Hypocrisy of Employers, a factor that makes it damn hard for a programmer to navigate this career landscape. Technology employers demand specialization in hiring. If you don’t have a well-defined specialty and unbroken career progress toward expertise in that field, they don’t want to talk to you. At the same time, they refuse to respect specialties once they’ve hired people, and people who insist on protecting their specialties (which they had to do to get where they are) are downgraded as “not a team player”. Ten years of machine learning experience? Doesn’t matter, we need you to fix this legacy Rails codebase. It’s ridiculous, but most companies demand an astronomically higher quality of work experience than they give out. The result of this is that the game is won by political favorites and self-selling douchebags, and most people in either of those categories can’t really code.

The Eternal Haskell Tax

The Haskell Tax really isn’t about Haskell. Any programmer who wishes to defend a specialty has a smaller pool of possible jobs and will generally squeeze less money out of the industry. As programming becomes more specialized and dimensional, the Haskell Tax problem affects more people. The Business is now defining silos like “DevOps” and “data science” which, although those movements began with good intentions, effectively represent the intentions of our anti-intellectual colonizers to divide us against each other into separate camps. The idea (which is fully correct, by the way) that a good Haskell programmer can also be a good data scientist or operations engineer is threatening to them. They don’t want a fluid labor market. Our enemies in The Business dislike specialization when we protect our specialties (they want to make us interchangeable, “full stack” generalists) but, nonetheless, want to keep intact the confusion and siloization that dimensionality creates. If the assholes in charge can artificially disqualify 90 percent of senior programmers from 90 percent of senior programming jobs based on superficial differences in technologies, it means they can control us– especially if they control the assignment of projects, under the pogrom that is closed allocation– and (more importantly) pay us less.

The result of this is that we live under an Eternal Haskell Tax. When the market favors it, junior engineers can be well-paid. But the artificial scarcities of closed allocation and employer hypocrisy force us into unreasonable specialization and division, making it difficult for senior engineers to advance. Engineers who add 10 times as much business value as their juniors are lucky to earn 25 percent more; they, as The Business argues, should consider themselves fortunate in that they “were given” real projects!

If we want to fix this, we need to step up and manage our own affairs. We need to call “bullshit” on the hypocrisy of The Business, which demands specialization in hiring but refuses to respect it internally. We need to inflict hard-core Nordic Indignation on closed allocation and, in general, artificial scarcity. Dimensionality and specialization are not bad things at all (on the contrary, they’re great) but we need to make sure that they’re properly managed. We can’t trust this to the anti-intellectual colonial authorities who currently run the software industry, who’ve played against us at every opportunity. We have to do it ourselves.

About these ads

25 thoughts on “Why programmers can’t make any money: dimensionality and the Eternal Haskell Tax

  1. It occurs to me that the reason you make more maintaining a payroll system in Java for Clorox than you do implementing human performance testing in Haskell for Aperture Science is exactly because Clorox is a big, dumb company that doesn’t understand technology or how it’s developed. And they don’t need to because everyone wants bleach and Clorox can pay you to keep the payroll system going for a few more years. Aperture, on the other hand, has no business model. You’re lucky they have R&D funding but until there’s a proven market for turrets, it’s questionable where your salary is coming from.

    I don’t know. The way the world works is troubling but coming up with a solid plan to fix it is not so easy.

  2. I think this article partially explains why society in general punishes those who do jobs that benefit society ( http://www.salon.com/2014/06/01/help_us_thomas_piketty_the_1s_sick_and_twisted_new_scheme/ ).

    Those anti-intellectual authorities who control us are the ones who have bullshit jobs and who take incomes from the real workers deceitfully, which is why people who benefit societies are paid less and those who have bullshit jobs are paid a lot.

    Bullshitters may have been stealing incomes from the real workers.

    • Bullshit jobs could have been in increase, and I can list a few possible reasons.
      1) Governments prefer shifting increasing welfare demands(due to automation) to private sectors because voters don’t like to see tax increases. In this case, bullshit jobs are another form of tax tha should have been welfare tax instead.
      2) As people were having more and more free time during the 20th century, they were increasingly involved more in protests and youth movements, and our masters wanted to remove free time, so our masters created bullshit jobs and imposed artificial scarcity to reduce free time.

      I could be wrong, and please enlighten me if you have more accurate information.

  3. Also, sometimes, high-end workers have hardly marketable skills like poetry.
    Poetry might be inherently hard to market, however it could be that anti-intellectual bastards actively drove out poetry in favor of clumsy romance novels which are easily exploited by them.

    • I did. I wish it hadn’t come out at 5:00 on a Friday afternoon. Just the fact that Google has stack-ranking ought to scandalize the tech world.

      What’s amazing is that even many Googlers are in denial about what “calibration scores” are (stack ranking). Google, internally, makes a lot of wind about how its performance review process is evolved and different and the useful idiots buy it.

      • Stack ranking is notorious for it’s negative effects, yet execs *still* seem to be madly in love with it. Why is that?

          • Presumably! Which interest though: the ‘nobody ever got fired for doing X’ one? the ‘I need to look like I’m doing something & sacking objective (hah!) underperformers is doing something’ one? Maybe the ‘I need to prove my superiority to everyone else & by definition upper management isn’t subject to this dehumanising process so that works out great for me’ one?

        • When you have no vision, cost-cutting is an easy way for an executive to earn his keep. Also, the entropy-increasing aggregation you get from a committee structure (or, even moreso, a politically dysfunctional company) tends to leave cost-cutting as the only thing they can agree on.

          Layoffs are, however, big decisions that alter the course of the company. You actually have to take a stand to push through a layoff. But everyone can agree, at least on paper, on firing “low performers” (ignoring the intensely political process of detecting and firing them at the ground level). Stack ranking is a self-consistent but intensely dishonest rolling layoff.

          In the long term, it shifts power toward management and HR departments, who therefore support it despite its deleterious effects on the company.

        • One manager was frank enough to tell me that he tried to get me in the above average category one year, at the negotiation where all the managers get together and parcel out the rankings.

          He was a good guy but I was not happy to hear how this was done.

  4. Very interesting thoughts.
    I tend to think that “if only more programmers were entrepreneurs” the value of having great IT departments would be appreciated. Instead of trying to change the way management/employers think, outcompete them by doing it right from the beginning.

  5. Brilliant article! Thanks for writing down so many ideas that many of us may have, without being able to put them in words in such a clear way. This cultural fight you refer to makes our life harder, but on the other end it justifies the term *delevoper* to me. We want to do better, we will do better, we will build a social system where continuously learning at work is considered natural.

  6. Another point to consider is that right there are just not enough high quality devs in Haskell, Erlang and Scala. Let’s I’m running an engineering org and I think it would take 5 Haskell devs or 50 Java devs. I can’t find the 5 Haskell devs, but I have a stack of 2,000 java resumes on my desk. Inevitably, my project turns to chaos, at which point I’m willing to shell out some real money for someone to fix it. Now the 2 great Haskell devs I know have a choice. Wrangle 50 mediocre Java engineers and try to fix my problems for 200k/yr or try to keep doing Haskell for 120k. I bet the next manager is going to have an even harder time find great Haskell devs, because 2 of them work for me…. and they cycle continues.

    When you reach a tipping point where there are enough experienced, good Haskell devs to build engineering teams I think you’ll see salaries rise fast. We’re starting to see it now with Scala.

    • I noticed that the companies that choose technologies like Java and .NET have one major perk – they also hire at entry-level, which is the only thing you can get as a not exceptional, but competent fresh graduate. I have never seen an entry level Haskell, Scala or Erlang job, every company requires years of experience in these languages. There are few good developers in these languages for these reasons:
      - you can only get experience in these languages outside your job, so companies are selecting for the really passionate people who go out of their way to become good at something, which is not bad by itself, but in combination with the next reason…
      - because you can only work in them in your free time, it takes much longer to become really good. And most companies’ definition of “high quality X developer” usually is translated as “has had professional X experience before 30″. IMO this is the most significant factor.
      - consider the perspective of the Haskell developer – if you’re working in Haskell, it means you’re in one of the few companies that actually care about technology, innovation and quality and you’re already in a better position than 99.9% of the other developers so why look for another job? How much better can you do? Is the payoff worth the job search? My guess is that the Haskell developers are not on the market for this reason.
      - exaggerations like “5 Haskell devs = 50 Java devs” tend to drive out the more humble and honest programmers. That is not possible in the same organization, with similar management styles and similar projects. Tools do make a difference, but not of this magnitude.

      • I think you missed my point and are making a similar point your self. I was not suggesting that Haskell itself makes someone 10x more productive, but that the average Haskell dev is 10x more productive that the average java dev. Like you said, the Haskell devs tend to be more experienced, passionate and I’d add they tend to just be better. I think it goes well beyond just the number of years of experience. From my experience the 10x is pretty realistic given those parameters. I’ve also found it’s easier to filter out bad Scala and Haskell devs than java devs.

        It’s easier to find 50 average(for java devs) java devs than 5 average haskell devs(for haskell devs). I come more from the Scala world than Haskell and I’m recovering from a long career in Java. I don’t regret using Scala in our company, but we are much more vulnerable to one of our engineers leaving than java shops are. Our scala devs here are extremely well compensated because of this. However, if the team I need to build had to be 2x as big, I don’t think I could have done it. I probably would have gone with Java or Python.

        I agree with your points on why there are few Haskell devs. This is a direct cause of few companies choosing Haskell(certainly not the only one). It’s this vicious circle that is painful for whichever side is on the bleeding edge. If a big company decided to go all Haskell tomorrow, they would pay astronomical rates for engineers because they’d have such a small pool. My point is that you have to look at both sides of the coin to get a complete sense of the problem. It’s not just assholes in big companies with no clue about technology trying to control us(not say they aren’t plentiful :) ).

    • I’ve just been hired for a Scala job with zero Scala experience, but with thirty years of experience overall, a fair amount of Java experience, and a deep knowledge of functional programming languages (not Haskell). And for decent money, too. How? Someone actually believed me for once when I said I could learn anything and my record proves it.

      I’m feeling very optimistic. I’ll see how it goes.

  7. Michael, besides producing output for customers and income for employees, jobs are where a large fraction of a salaried person’s life takes place. If the job is enjoyable that’s a very good thing.

    Is it really reasonable to peg “but I’m worth this much” (to some other firm under some other circumstances) and use that opportunity cost to justify getting more than plenty of money somewhere that’s pleasurable to work?

    Talking about salary here: guaranteed income not performance-based pay or risk-reward.

  8. With five years of solid experience, a person can add 3 to 50 times more value than the entry-level grunt. Is she paid for that? Almost never.

    I’m going to assume this figure comes from the Mythical Man-Month. Can you really justify it with actual measurements? Or is it just an accepted conclusion?

  9. Note that there’s a casual relationship between “average quality of programmer available using that language” and “pay scale for programmers using that language.”

    The assumption is that people who learn programming languages to become better programmers (whether they state it as “because it looks cool” or “for a great good”, or “because it’s what I used in school”) will be better programmers than those who learn a programming language because it improves their employment prospects (whether they say “look at all the jobs avaliable” or “because it’s what I used in school”).

    If the supply of programmers in a language exceed the demand for them, then there will be almost no programmers who learned the language in order to improve their employment prospects. So the average quality of the programmer will be higher. But – because the supply exceeds the demand – the average pay for them will also be lower.

    On the other hand, once the demand exceeds the supply, you’ll start seeing people enter the market who learned the language to improve their employment prospects – so the quality of the average programmer will go down. On the other hand, because demand exceeds supply, their pay will go up.

    And yes, that sucks.

  10. Hi Michael.
    Thank you for this great post!
    Do you think we could translate this article to Russian and publish on our corporate blog (http://habrahabr.ru/company/abbyy/blog/) — of course, with the name of the author clearly indicated and a link to the original text?
    We are a language software development company and our developers will certainly appreciate this brilliant article.
    Thanks,
    Denis

  11. I empathize with the thoughts in this post, but I am trying to understand your conclusion or call-to-action.

    Do you believe that programmers should establish artificial scarcity in the same way that, say, doctors have — with political organizations like the American Medical Association and credentialing via something equivalent to medical school and board certification?

    Or, do you think that programmers should form unions and bargain for the value of their labor against employers?

    I think one thing your post fails to recognize is that, in our economy, people are *rarely* paid for the value they actually add to society (or even to the business they work for).

    This isn’t a great injustice only to programmers, but instead to all “non-bullshit” contributors to society. But this is a problem with our brand of capitalism, not with software engineering careers.

    Ask a teacher, police officer, or firefighter if they are paid proportional to the value they add. Ask a great professor who has educated Nobel laureates whether they are paid even 20% more than their mediocre colleague who can’t even give a lecture to students.

    Instead, most people are paid for political reasons or due to closeness to market transactions. For example, stock brokers don’t add much value to society overall, but are very close to market transactions for pricey financial instruments, and so they can skim off them; CEOs are paid well for the political reason that they sit on the board of a company with the people who set their salary.

    Many pro-market intellectuals try to make arguments that these people are also adding value to society (or make the circular argument that whatever value the market sets for their labor, in itself, represents a proper accounting of “value” to society), but this is generally a lot of hogwash.

    It actually strikes me, as a programmer myself, that coding is one of the rare careers where you are able to a) get paid well (by most standards) and b) work on meaningful/challenging problems (at least, if you escape corporate IT — which you should, immediately).

    Further, in modern times, programming & computer science skills have been irrevocably linked with the ability to create software businesses, which can allow you to bypass all the normal rules of how corporations are run by starting your own. Or you can join some other programmer’s idea for a profitable enterprise, and help her create a business that can run on its own. This will be deeply satisfying — at least, until your company becomes “the new boss, same as the old boss”.

    So, I don’t know.

    I see what you’re saying, but I guess I’m a little more positive/optimistic than you are about our position in all of this, as programmers. I think rather than programmers being in decline, we are increasingly at a premium. I think the rest of the world is realizing it is “program or be programmed”. I am much more worried about a non-programmer’s future on a 30-year horizon than a competent programmer’s position in all of this.

    The future might sound like a market flooded with amateurs, but that is also why for the problems that actually require talent, the demand will always be there for the experienced and competent. Bad programmers can’t make good programs. And novice programmers only get lucky sometimes.

    I also think that over the course of a programmer’s career, it’s true that the hurdle for what is considered technical competency continues to rise, which requires a rather ruthless form of auto-didacticism. But I don’t think that’s any different than many other technically sophisticated career, e.g. aerospace, electric engineering, surgery.

    In any case, it has been a long time since The Business told me what to do. But then again, maybe in your mind, I am “The Business” — after all, I run a tech startup with 25 employees.

    On the minor point, I agree. If the world were just, great programmers would be compensated greatly and their compensation would grow steadily with their years of experience — exponentially or even linearly. But do we live in a just world, or a real one?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s