The software industry handles failure poorly.

The startup world venerates failure. Well, sort of. It venerates business failure, so long as it happens in a way that the investors approved of. The VCs will manage the careers of the founders, and no one else involved with the failing company (i.e. the people who lose jobs) is considered to matter. “Failure is awesome!” “Go out and fail!” There’s plenty of talk among technology people about “failing fast”, and yet what I’ve observed is that our industry handles failure quite poorly.

Of course, the software industry is full of failure. In the corporate context, the term failure is somewhat subjective, but I’ve heard that 60 to 80 percent of software projects fail. I don’t doubt that number at all. I’m going to put it higher: I’d guess that 97 percent of software efforts fail. Admittedly, my standard of success is high. How do I get to 97%? Well, about 40 percent of projects are cancelled before they deliver anything. That’s a clear case of failure (“total hard failure”). Another 30 percent can be salvaged and reformed into something, but underperform the expectations set based on their headcount and resources, and are written off as embarrassments (“partial hard failure”).

In other words, 70 out of 100 software projects are hard failures. That’s not controversial. Of the remaining 30, half of those (or 15% of the whole) are soft failures: well-made, good products that go unused, or are regarded as failures, for purely political reasons. In other words, there was no technical failure, but the product did not succeed politically. The software itself was just fine– possibly, quite excellent– it’s not going to get an engineer promoted. This leaves 15% that could be seen as successes. Of them, four-fifths (12% of the whole) devolve quickly into disliked legacy monsters that, even though they get launched and become critical to the business, are viewed with enough disdain that, while managers of the projects might get promoted, the engineers are under constant criticism for the shortcomings of the system (which may have more to do with time constraints than engineering shortfalls). This leaves only 3% of software projects that are able to (a) solve the problem, (b) do so using a resource budget deemed acceptable, (c) succeed politically, and (d) continue to be well-regarded enough to make the case for an engineer’s promotion. The other 97% of projects? They add some value to the business but, from a careerist perspective, they’re a total waste of time, because they don’t confer the political capital that would give their engineers the credibility to work on better, harder, more critical problems.

Given the low success rate of software projects, even though many of those causes of failure aren’t the engineers’ fault, it’s probably not surprising that this industry has such a high burnout rate. Our success rate is comparable to what one might expect out of an R&D lab. No one holds a researcher in low regard if many of her ideas fail; if that doesn’t happen, that often means that her aims aren’t ambitious enough. Sub-50% success rates are expected in R&D, with the understanding that the successes will pay for the exploration costs (a polite term for the costs of the failures). We, however, are held responsible and sometimes fired for software project failures, which are presented as never acceptable.

The 70% that I call hard failures can lead to project cancellations and, in bad economic times, lost jobs. So can the 15% that are soft failures (i.e. working software rejected for political reasons) although it is less common. Successfully completing something that goes unused will not get a person fired, but it does not protect a person from layoffs. The 12% that evolve into hated legacy assets rarely end jobs. In fact, they create jobs in the company, but they’re usually undesirable maintenance jobs. Those projects turn out well for managers but, for engineers, it’s rare that they make a case for promotion. Managers can say that they “successfully delivered” these projects and it’s not really a lie, but the programmers are expected to stick around and maintain them. Taking this all in, one sees that the projects that can really make a case for someone as a programmer require a large number of things to go right, and almost nothing to go wrong.

With the battle scars that come with age, programmers tend to develop prejudices that make little sense to anyone else. I’ve heard people say that it’s “impossible” to write production code in a dynamically typed language. It’s not. I personally prefer static typing, but high-quality production code is written in dynamic languages every day. I’ve heard others bash practices and tools based on previous experiences with software failure. It’s a multiple-cause attribution behavior that doesn’t make much sense.

When negative outcomes are rare, it’s sensible to attribute them to all causes. Let’s consider traffic accidents. On a drive, the positive outcome (no collision) is common and the negative one is very rare. Most traffic accidents have multiple causes: the driver was exhausted, it was snowing, and the intersection was poorly designed. It’s rare that such accidents happen because one thing goes wrong; it usually takes multiple things going wrong at the same time. The conclusions that are drawn, however, are valid: driving while exhausted is dangerous, snow makes driving more dangerous, and badly-designed intersections are dangerous. Blaming all causes makes sense.

However, when negative outcomes are common, it’s often because a single cause can ruin the whole thing. This is the nature of software. True successes are damn rare, most projects fail, and small things can cause great work to be all for nought. This is worsened by the fact that, in the corporate theater, people in power can easily obfuscate the causes of things going wrong (or, to put it more bluntly, shift blame). In the example of the traffic accident, there were multiple causes that were genuinely involved. In a corporate failure, only one cause is necessary to make it happen, but the executives can generate multiple causes that might have been involved, and they’ll frequently do so, for obvious self-serving reasons.

Let’s say, for a concrete example, that a software project is started in Python, because it’s what the engineering team knows and likes using. (It’s not my favorite language, but it works.) Four months in, there’s an executive shake-up and a new CTO comes in with a strong pro-Java bias, because it’s what he knows. In addition to this technical bias, he needs to Make Decisions on his first day, “to prove he’s serious”, so he waltzes over to one of the teams under him (this team) and, without taking time to determine why the team chose the tools that it did, he says, “There’s no way that Python can scale; use Java instead”.

What happens? Over time, the strong Python engineers leave, while the weak ones stay and begrudgingly switch over to Java. Additionally, the language overhaul turns out to be more complicated than expected and, due to time constraints, it can’t be completed. So the project ends up being a Python/Java hybrid app with poor communication between the two halves. The app is delivered, but it’s slow and buggy because the good Python engineers have left and the weak ones weren’t able to write performant code in any language. The real cause of the failure? A CTO who interfered with a functioning team. The official version of events? The CTO was right; Python simply “cannot scale”. It can’t be the idiot CTO that killed that project; Guido von Rossum himself did it.

What’s worst isn’t just that this pattern of behavior exists, but how quick software engineers are to buy into the “official” story, and attribute negative outcomes to the technologies that their projects relied upon. I’ve met so many software engineers who absolutely hate technologies that weren’t to blame, at all, for the project failures attributed to them. Did the project really fail because of Python? Or because test-driven development just takes too damn much time? Or because XML is a hideous mess of a standard (which it is, but that’s almost always irrelevant)? I strongly doubt it. In almost all cases, I’d guess that the project failed because of managerial or political factors (read: executive incompetence) and that technologies were simply blamed because the “official” narratives get written by (no surprise) executives. Programmers and technologies have been a dumping ground for executives’ failures for decades and, as bad as that is, it’s even worse that so many programmers are stupid enough to believe these official narratives– and go on to future jobs and spout idiocies like “Python cannot scale.”

Software project failures are painful, especially for programmers. Although failure is common in our industry, we don’t really get permission to fail. Companies can’t fire everyone when a software project fails, because even the most evil companies don’t want that much turnover. Still, if a project lands in hard failure, it’s pretty much a guarantee that at least one person has to lose a job, and the manager’s going to make sure that it isn’t him. Soft failures don’t necessarily end jobs, but they stall careers, make people nervous, and push people to quit.

After a few years in this industry, everyone has enough experience with failed projects (including, again, successful projects that failed politically or that devolved into mediocrity for political reasons, especially including feature creep) to have battle scars, and that gives each of us a long list of technologies and patterns that we find ourselves hating. Don’t get me wrong, either: many of our bugbears are legitimately hideous. I can’t stand the singleton directories (e.g. “com”) of Java projects, the broken handling of loop variables in Python closures, or the monkey-patch facilities of Ruby. These are all ugly things, but I think it’s extremely rare that any one of them ever single-handedly caused a project to fail.

To me, it’s weird and inconsistent. The startup world venerates failure, but only when founders do it. A VC-backed founder who steers a business into a reef is lauded for taking the risk and either “acqui-hired” or given another company. A programmer who fails (or, more often, who has the misfortune of working on a project that fails, which is often not her fault) is just regarded as a loser. Programmers are expected to have career narratives of one success after another. This is inconsistent with the fact that about 40 percent of software projects fail totally, and another 57 percent fall into murky non-success or uninspiring mediocrity. Worse yet, programmers often don’t know (due to managerial secrecy and the pervasive cover-up culture) the real reasons why their projects failed and, even if they do know, can’t speak the truth in public or in a professional context such as a job interview (where “bad-mouthing” an ex-manager is taken to be a crime as serious as murder or rape). Thus, they come up with a myriad of faulty explanations for why things went badly on previous projects. “What we learned… is that Python cannot scale.” That’s not true at all, but it’s more socially acceptable to say that on a job interview than to say the truth for most values of “the truth” (which involve frank incompetence at executive levels).

Writ large, these non-truths get repeated often enough that they generate a cloud of tech hate. Every language, every framework, every tool, just sucks. Python “can’t scale” and Java programmers “are all morons” and C “is insecure and dangerous” and Haskell “is impossible to hire for”. That’s the narrative. Some tools do suck, and far more often a tool is inappropriate to the purpose (e.g. Java when one wants rapid development, or Python for low-latency systems) to which it’s put. However, nuance is lost in the cloud of embittered tech-bashing, and the image that it projects of our work as a whole (and of us) is that most of what we build is pure garbage. Is it any wonder, with such a culture, that programming remains a low-status profession?

14 thoughts on “The software industry handles failure poorly.

  1. Interesting food for thought as usual. However regarding ” Is it any wonder, with such a culture, that programming remains a low-status profession?”, I believe that the bashing is a largely result of the low status rather than the low status a result of the bashing. I suppose it’s vicious cycle. As a community we definitely play into our own subordination.

    In any case, I appreciate your thoughts in general. They’ve helped to clarify certain aspects of a developer career which have troubled me but which I couldn’t quite articulate.

    • It is, of course, a vicious cycle.

      The bad behavior comes from the crab mentality, and a large dollop of individual ego. You have a lot of smart people who come into industrial programming and, because they misread social cues, they fail to recognize that they’re being steered into subordinate roles. Six months later, they realize what’s happened and say, “I’m better than this job”, which expresses itself as “I’m better than *these people*” which results in them all beating the others down.

  2. That’s a hell of a straw man article, yikes. No, 70% of software projects don’t “fail”. The study that you’re probably regurgitating defined “success” as completing without being over budget, and for 2014 that was 16% of software projects. The canceled or inoperable projects were 30%. Which makes sense, since software developers aren’t cheap (despite their “low status”).

    Even at 30% failures it sounds pretty bad. Maybe we need more MBA’s keeping an eye on things.

    • Oversight with an MBA mentality is often the cause of the failures. When a software development is treated as a manufacturing (by those who only understand manufacturing), a great team can often still be successful, but it’s very difficult for a good or average team to overcome that pathology.

      Such an approach is not limited to MBAs (nor is exhibited by all MBAs of course). But I would argue that business education is probably a main contributor to its prevalence.

    • Failure includes being late, which may or may not be a subset of over budget.

      I think that’s a sign of low status, to call that a failure. If you remodel your kitchen, estimate is $25,000, and eight weeks. If it takes $28,000 and ten weeks, has your remodel failed? (These are numbers I made up, not sure what a remodel should cost or how long it should take.)

      • This is my point. Programmers get blamed every time expectations are missed (e.g. a 20% overrun) and management takes the credit when things go unexpectedly well. It works that way, because we have such low status as a professional group.

    • I estimated that 40% are total hard failures (i.e. cancelled or inoperable). So there’s not much disagreement. Then I estimated that 45% were partial failures or soft failures (i.e. projects that were delivered and worked, but went unused or failed for political reasons).

  3. Pingback: The Number-One Rule of Business Is to Stay Business-Focused – upsurgemarketinggroup

  4. Turning programming into a high-status profession seems like an up-hill battle. If it ever becomes a high-status profession, when and how do you think it would?

    • We have the core ingredient, which is demand for our services. That’s huge. If no one wanted what we were selling (e.g. most of academia) then I’d say that it’s impossible and we should all do something else. However, there’s a huge demand for our services. We just fuck it up when it comes to leveraging that.

      For example, if you have a 125+ IQ and your work is *really* that important that your boss demands daily status updates, you should be getting $375k per year, at a minimum, to offset the pain and stress.

      Note: I’m not saying that developers deserve to make $375k/year. I’m saying that if there’s an actual business need for daily status updates and the one-sided aggressive transparency of “Agile”, then we ought to leverage said business need as far as we can. “Sure, I’ll work that way; here’s my price.”

      What we lack is organizational integrity. The demand for our work exists, but we’re letting it be used as an excuse to maltreat us (“oh, we need daily updates because your work is so critical”) rather than to pay us well and give us respect and authority.

      First, while we have an admirable openness to new and non-traditional entrants, we also let in a lot of unqualified open-plan brogrammer types who are attracted by the salaries (we might consider $120k “mediocre”, in part because out jobs force us to live in expensive places, but they wouldn’t get half of that outside of a tech bubble) and by the so-called “work hard, play hard” culture where they can drink beer at their desks starting at 3:30 as long as they’re in the office till 9:00.

      We need to professionalize around our raw intelligence (as unfashionable as IQ may be as a concept, we ought to leverage the fact that only 5% of the population can ever do what we do at a high level). At the same time, we need to let high-IQ non-traditional people (i.e., the 38-year-old woman who picked up a Python book after having three children and leaving a different career) in.

      Right now, our profession is exclusive but it’s exclusive on stupid stuff that has nothing to do with intellectual or creative ability. I was able to get in, and reach a managerial level, because I’m a 6-foot-tall white male of above-average attractiveness… not because I’m one of the smartest cunts out there (a fact that has only hurt me).

      We need to find a way to exclude the unqualified, anti-intellectual, open-plan brogrammers while, at the same time, making the profession more open to women, minorities, and older people. That will improve our position and organizational capacity substantially. The truth is that the VCs already *know* that the best programmers hit their stride in their late 30s and 40s. So why do they create an ageism culture? They want naive people that they can take advantage of. The fear is that anyone over 35 who’s not a manager is a potential unionist.

      Second, we need not only to professionalize (which means defining what we are, what we do, and how we provide services for the client as equals rather than subordinates) but we need to start looking out for each other. The cowboy culture hasn’t worked, the corporate IT drone culture hasn’t worked. We need to restore the pay-it-forward culture that Silicon Valley had before it got ruined by the narcissistic private-sector career politicians (i.e. the executive types who don’t give a shit about technology except as a way to make money) and we need to do so at a national, if not global, scale.

      It can certainly be done, and the reason I say that is because there’s one ingredient that really matters: demand for the services. If that doesn’t exist, the only way to create a high-status profession is through dishonesty… and that’s not a game I’d ever want to play. In our case, though, it very much does exist. We just haven’t found a way to organize around it.

    • > We need to find a way to exclude the unqualified, anti-intellectual, open-plan brogrammers while, at the same time, making the profession more open to women, minorities, and older people.

      I surmise inclusion of various kinds of people could be done relatively safely as long as it is costly, difficult, or culturally unacceptable to display undesirable behaviors( of the unqualified, anti-intellectual, open-plan brogrammers). In other words, it’s about nudging people into certain behaviors by environmental design.

      > We need to restore the pay-it-forward culture.

      Institutions designed for pay-it-forward culture may help keep people from derailing from such culture. I’d start with institution of union, profession, etc. What more can we do to establish pay-it-forward culture?

      > We need to professionalize around our raw intelligence

      If there was a better way to test intelligence than merely referring to an IQ score, such professionalization might work. What would be a good test for raw intelligence? Any example?

      • Measuring intelligence is a rat’s nest, and raw intelligence really doesn’t matter in general. There are people with high IQs who are complete degenerates, and vice versa. The truth is that the SAT is basically an IQ test (with some prep factor and gameability, but IQ tests would have that as well, if there were any stakes attached to them) and I know plenty of really smart people who test poorly (and vice versa).

        Besides, “IQ” is actually a shorthand for “hard-to-fake creative and intellectual ability”. Whether it’s genetic hard-wired intelligence or acquired through years of hard work… doesn’t matter. Besides, most people, even most people who claim to have high IQs, haven’t taken a real IQ test. This is just as well because the tests are pretty meaningless beyond about 125, which is only about the 94th percentile.

        What we actually care about is true *merit*, which is hard to define and even harder to measure. I think, though, we can look at different models and compare. The actuarial exams provide a present signal of merit; the startup world with its emphasis on pedigree (Harvard or Stanford => founder; anyone else => employee) uses a dated signal. No organization gets leader selection “right”, in the sense of doing it perfectly, but some do a better job than others.

  5. “We need to find a way to exclude the unqualified, anti-intellectual, open-plan brogrammers while, at the same time, making the profession more open to women, minorities, and older people.”

    This seems wildly, outrageously presumptuous.

    If you judge a worker as unqualified, it’s reasonable to not hire them, or give them a bad performance evaluation, or even layoff or fire that worker. But to suggest that they be excluded beyond that. To say, that even if another manager finds their skills a great fit for a job and their work performance excellent, that you have some grand authority to dictate that someone be unqualified and excluded is presumptuous and absurd.

    And for you to cast judgement that certain genders or racial groups are over/under valued and should be demoted/promoted is equally pretentious.

  6. Massimo, Michael is simply attacking the fact that programmers are evaluated more on their ability to play the game and meet normative cultural expectations (white heterosexual male) than any real ability to deliver (good) systems. Most managers have no clue what makes a good programmer or any way to actually interview for it. Sure, you can interview for hard technical skills but those are easy. I like to see how people think through problems. It is a lot more fun and gives me something actually useful. Managers also often have no clue how to really run software development.

    Statistically, genders and racial groups are undervalued in a society where faux social fit is valued above all. We all prefer to hire ourselves and statisics suggest that good candidates are close to equally prevalent in all populations. IQ and motivations vary little by traits like sex but the number of people working in your company probably does.

    Besides which, Michael is free to have an opinion and having an opinion is not pretentious rather calling someone pretentious and encouraging them not to have an opinion leads to stifled creativity and a dictatorial society. You are also free to not like his opinion but shutting him up, no.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s