The software industry handles failure poorly.

The startup world venerates failure. Well, sort of. It venerates business failure, so long as it happens in a way that the investors approved of. The VCs will manage the careers of the founders, and no one else involved with the failing company (i.e. the people who lose jobs) is considered to matter. “Failure is awesome!” “Go out and fail!” There’s plenty of talk among technology people about “failing fast”, and yet what I’ve observed is that our industry handles failure quite poorly.

Of course, the software industry is full of failure. In the corporate context, the term failure is somewhat subjective, but I’ve heard that 60 to 80 percent of software projects fail. I don’t doubt that number at all. I’m going to put it higher: I’d guess that 97 percent of software efforts fail. Admittedly, my standard of success is high. How do I get to 97%? Well, about 40 percent of projects are cancelled before they deliver anything. That’s a clear case of failure (“total hard failure”). Another 30 percent can be salvaged and reformed into something, but underperform the expectations set based on their headcount and resources, and are written off as embarrassments (“partial hard failure”).

In other words, 70 out of 100 software projects are hard failures. That’s not controversial. Of the remaining 30, half of those (or 15% of the whole) are soft failures: well-made, good products that go unused, or are regarded as failures, for purely political reasons. In other words, there was no technical failure, but the product did not succeed politically. The software itself was just fine– possibly, quite excellent– it’s not going to get an engineer promoted. This leaves 15% that could be seen as successes. Of them, four-fifths (12% of the whole) devolve quickly into disliked legacy monsters that, even though they get launched and become critical to the business, are viewed with enough disdain that, while managers of the projects might get promoted, the engineers are under constant criticism for the shortcomings of the system (which may have more to do with time constraints than engineering shortfalls). This leaves only 3% of software projects that are able to (a) solve the problem, (b) do so using a resource budget deemed acceptable, (c) succeed politically, and (d) continue to be well-regarded enough to make the case for an engineer’s promotion. The other 97% of projects? They add some value to the business but, from a careerist perspective, they’re a total waste of time, because they don’t confer the political capital that would give their engineers the credibility to work on better, harder, more critical problems.

Given the low success rate of software projects, even though many of those causes of failure aren’t the engineers’ fault, it’s probably not surprising that this industry has such a high burnout rate. Our success rate is comparable to what one might expect out of an R&D lab. No one holds a researcher in low regard if many of her ideas fail; if that doesn’t happen, that often means that her aims aren’t ambitious enough. Sub-50% success rates are expected in R&D, with the understanding that the successes will pay for the exploration costs (a polite term for the costs of the failures). We, however, are held responsible and sometimes fired for software project failures, which are presented as never acceptable.

The 70% that I call hard failures can lead to project cancellations and, in bad economic times, lost jobs. So can the 15% that are soft failures (i.e. working software rejected for political reasons) although it is less common. Successfully completing something that goes unused will not get a person fired, but it does not protect a person from layoffs. The 12% that evolve into hated legacy assets rarely end jobs. In fact, they create jobs in the company, but they’re usually undesirable maintenance jobs. Those projects turn out well for managers but, for engineers, it’s rare that they make a case for promotion. Managers can say that they “successfully delivered” these projects and it’s not really a lie, but the programmers are expected to stick around and maintain them. Taking this all in, one sees that the projects that can really make a case for someone as a programmer require a large number of things to go right, and almost nothing to go wrong.

With the battle scars that come with age, programmers tend to develop prejudices that make little sense to anyone else. I’ve heard people say that it’s “impossible” to write production code in a dynamically typed language. It’s not. I personally prefer static typing, but high-quality production code is written in dynamic languages every day. I’ve heard others bash practices and tools based on previous experiences with software failure. It’s a multiple-cause attribution behavior that doesn’t make much sense.

When negative outcomes are rare, it’s sensible to attribute them to all causes. Let’s consider traffic accidents. On a drive, the positive outcome (no collision) is common and the negative one is very rare. Most traffic accidents have multiple causes: the driver was exhausted, it was snowing, and the intersection was poorly designed. It’s rare that such accidents happen because one thing goes wrong; it usually takes multiple things going wrong at the same time. The conclusions that are drawn, however, are valid: driving while exhausted is dangerous, snow makes driving more dangerous, and badly-designed intersections are dangerous. Blaming all causes makes sense.

However, when negative outcomes are common, it’s often because a single cause can ruin the whole thing. This is the nature of software. True successes are damn rare, most projects fail, and small things can cause great work to be all for nought. This is worsened by the fact that, in the corporate theater, people in power can easily obfuscate the causes of things going wrong (or, to put it more bluntly, shift blame). In the example of the traffic accident, there were multiple causes that were genuinely involved. In a corporate failure, only one cause is necessary to make it happen, but the executives can generate multiple causes that might have been involved, and they’ll frequently do so, for obvious self-serving reasons.

Let’s say, for a concrete example, that a software project is started in Python, because it’s what the engineering team knows and likes using. (It’s not my favorite language, but it works.) Four months in, there’s an executive shake-up and a new CTO comes in with a strong pro-Java bias, because it’s what he knows. In addition to this technical bias, he needs to Make Decisions on his first day, “to prove he’s serious”, so he waltzes over to one of the teams under him (this team) and, without taking time to determine why the team chose the tools that it did, he says, “There’s no way that Python can scale; use Java instead”.

What happens? Over time, the strong Python engineers leave, while the weak ones stay and begrudgingly switch over to Java. Additionally, the language overhaul turns out to be more complicated than expected and, due to time constraints, it can’t be completed. So the project ends up being a Python/Java hybrid app with poor communication between the two halves. The app is delivered, but it’s slow and buggy because the good Python engineers have left and the weak ones weren’t able to write performant code in any language. The real cause of the failure? A CTO who interfered with a functioning team. The official version of events? The CTO was right; Python simply “cannot scale”. It can’t be the idiot CTO that killed that project; Guido von Rossum himself did it.

What’s worst isn’t just that this pattern of behavior exists, but how quick software engineers are to buy into the “official” story, and attribute negative outcomes to the technologies that their projects relied upon. I’ve met so many software engineers who absolutely hate technologies that weren’t to blame, at all, for the project failures attributed to them. Did the project really fail because of Python? Or because test-driven development just takes too damn much time? Or because XML is a hideous mess of a standard (which it is, but that’s almost always irrelevant)? I strongly doubt it. In almost all cases, I’d guess that the project failed because of managerial or political factors (read: executive incompetence) and that technologies were simply blamed because the “official” narratives get written by (no surprise) executives. Programmers and technologies have been a dumping ground for executives’ failures for decades and, as bad as that is, it’s even worse that so many programmers are stupid enough to believe these official narratives– and go on to future jobs and spout idiocies like “Python cannot scale.”

Software project failures are painful, especially for programmers. Although failure is common in our industry, we don’t really get permission to fail. Companies can’t fire everyone when a software project fails, because even the most evil companies don’t want that much turnover. Still, if a project lands in hard failure, it’s pretty much a guarantee that at least one person has to lose a job, and the manager’s going to make sure that it isn’t him. Soft failures don’t necessarily end jobs, but they stall careers, make people nervous, and push people to quit.

After a few years in this industry, everyone has enough experience with failed projects (including, again, successful projects that failed politically or that devolved into mediocrity for political reasons, especially including feature creep) to have battle scars, and that gives each of us a long list of technologies and patterns that we find ourselves hating. Don’t get me wrong, either: many of our bugbears are legitimately hideous. I can’t stand the singleton directories (e.g. “com”) of Java projects, the broken handling of loop variables in Python closures, or the monkey-patch facilities of Ruby. These are all ugly things, but I think it’s extremely rare that any one of them ever single-handedly caused a project to fail.

To me, it’s weird and inconsistent. The startup world venerates failure, but only when founders do it. A VC-backed founder who steers a business into a reef is lauded for taking the risk and either “acqui-hired” or given another company. A programmer who fails (or, more often, who has the misfortune of working on a project that fails, which is often not her fault) is just regarded as a loser. Programmers are expected to have career narratives of one success after another. This is inconsistent with the fact that about 40 percent of software projects fail totally, and another 57 percent fall into murky non-success or uninspiring mediocrity. Worse yet, programmers often don’t know (due to managerial secrecy and the pervasive cover-up culture) the real reasons why their projects failed and, even if they do know, can’t speak the truth in public or in a professional context such as a job interview (where “bad-mouthing” an ex-manager is taken to be a crime as serious as murder or rape). Thus, they come up with a myriad of faulty explanations for why things went badly on previous projects. “What we learned… is that Python cannot scale.” That’s not true at all, but it’s more socially acceptable to say that on a job interview than to say the truth for most values of “the truth” (which involve frank incompetence at executive levels).

Writ large, these non-truths get repeated often enough that they generate a cloud of tech hate. Every language, every framework, every tool, just sucks. Python “can’t scale” and Java programmers “are all morons” and C “is insecure and dangerous” and Haskell “is impossible to hire for”. That’s the narrative. Some tools do suck, and far more often a tool is inappropriate to the purpose (e.g. Java when one wants rapid development, or Python for low-latency systems) to which it’s put. However, nuance is lost in the cloud of embittered tech-bashing, and the image that it projects of our work as a whole (and of us) is that most of what we build is pure garbage. Is it any wonder, with such a culture, that programming remains a low-status profession?

Pyramid vs. obelisk

I may not be able to save the world, but I think I’ve come up with the fundamental idea that will be necessary in order to restore (and, afterward, preserve) the integrity of, at the very least, the software industry.

Software may or may not be “eating the world”, but we’re learning that the class of people who can rigorously solve problems using precise thought is one that ought to be held in high demand. Such people need not necessarily be writing software programs per se, but we’re in a world that cannot grow based on ego and bluster (the traditional traits used to select business leaders) alone. That approach has been played out, and we need competence to get any further. There are a variety of literacies that a leader needs in order to do her job well, and that set of needed capacities is too large for me to tackle right now, so let’s zero in on one important job, which is software engineering. As a society, we need it. We need it, done well. Unfortunately, as a society, we seem to be getting worse at it.

The problem is that most of the people who employ programmers have seen fit, over the past two decades, to diminish this kind of work as much as possible. There have been so many efforts to replace professional programmers with interchangeable, unqualified peons that it is, frankly, sickening. This is, of course, the real purpose of the “Agile Scrum” fad that has infested our industry, as well as the fetishism for open-plan offices. Dan Lyons’s new book, Disrupted, describes the ageist culture that emerges when we allow the industry to drive software, rather than the other way around. We’ve seen competent engineers driven out in favor of cheap, young, and (most importantly) obedient commodity-grade programmers hired to inflate headcount numbers and appease investors. It needs to stop, and now, before this industry gets so thoroughly infested with incompetence and anti-intellectualism that a generation’s worth of work and experience will need to be discarded. So how do we stop it?

Although it shouldn’t be this way, software engineering is pyramidal in shape. For every engineer who gets to work on a real project, there seem to be fifteen grunts only trusted to churn through Jira tickets. We need to reform our professional structure to be more like an obelisk, instead.

Architecturally, a pyramid is the most basic, inefficient way of achieving prominence (or, less pedantically, height). It’s imposing, it’s gigantic, and it historically required massive amounts of slave labor as well as natural resources. Ancient Egyptians built pyramids because they didn’t have the architectural know-how (namely, arches) to build much else at scale. In economics, one invokes the pyramidal shape when describing arrangements that deliver massive yield to the few (at the top of the pyramid) while exploiting the larger lower classes (at the bottom). Hence, the term “pyramid scheme”. One could argue that pyramids (of both kinds) are entropic. A pile of sand or stones will assume an approximately conical shape, and it’s likely that this inspired the (architecturally unsophisticated) pyramids of Ancient Egypt. Likewise, an ill-organized or disorganized set of people will devolve into an oligarchic arrangement, with many supporting the few.

Professional pyramids mean that positional scarcity will exist at every level in the profession (the term “profession”, here, used loosely) as the number of positions declines as an exponential function of experience. It’ll be as hard to stay in the profession as to get in. Except for sociopaths who enjoy social competition for its own sake, most people find this undesirable. Doctors and attorneys wouldn’t be happy if they had to continue fighting for fewer spots with each year of age, even 30 years after graduating. Not only would it be bad for individuals to have that degree of positional competition, but it would also be a waste for society. To push out 50-year-old doctors simply because they “failed” to arrogate managerial positions would be to discard hard-won expertise, and it would generate a culture of inexperience.

Left to their own devices, management hierarchies become pyramidal. If a company of 10,000 people decides on a 12:1 reporting ratio (as is typical, in software) then it’s going to have 9,167 grunts, 764 first-level managers, 64 directors (managers of managers), and only 5 people at the executive level as traditionally defined. Why can’t there be 100 directors instead of 64? There could be, but few companies would want to pay for them if given the choice. That pyramid seems painful, and it would be, if that’s how companies worked. At the top, they often don’t, and I’ve written before about the superior collective intelligence of MBAs as opposed to software programmers. People with elite MBAs aren’t stupid. They realize that they’re going to get screwed under the traditional pyramidal regime. Not only does the pyramid system make it unlikely for people to advance, but they also leave the people in high positions unprotected, because there are too few of them. CEOs promote loyalists not because the companies need so many executives, but because the CEOs are smart enough to realize that sharing rank and prominence is essential for their own survival.

One could argue that business executives, as a group, “conspire” against their companies by working together in order to convince their organizations that they’re more needed than they really are, and that more of them are needed than really are, and that they themselves as a class (executives) can recognize talent in a way that no one else (not shareholders, not workers) can. By doing this, they’ve managed to create massive demand for themselves and their skill set (which is perceived to be more rare than it actually is). On this topic, I’m hesitant to use the word “conspire” because I don’t consider it morally wrong. Corporate life has become a culture of resource extraction, and the executives have simply been the best players at that game. Ultimately, there are very few people who would hesitate, if offered a magical device that would imbue in their employers an extreme and superstitious overvaluation of their work, to use it for personal benefit. Executives have simply managed to do exactly that as a group, rather than as individuals, the latter being far more difficult.

Corporate executives, based on manufactured demand, as well as doctors and attorneys based on actual demand, have taken their leverage and replaced the pyramidal structure with a professional obelisk shape. An obelisk is upright, rising far higher than a pyramid of comparable mass could, until the pinnacle. With a professional obelisk, there’s plenty of room to grow until one encounters positional scarcity. Doctors don’t need to beat down other doctors and climb management hierarchies just to advance; they just need to gain more skills and deliver more value. Likewise, MBA-toting corporate executives might face stiff competition if they want to be Fortune-500 CEOs (because there are only 500 such spots) but they deal with minimal competition if they just want 7-figure incomes and fancy titles by age 50.

An obelisk structure shows that a value is placed on expertise: as long as people continue to improve themselves, there’ll be room for them at a level higher than where they are. Progress will be recognized. Why does the medical profession intentionally create an obelisk-shaped, guild structure? It doesn’t want “Agile” 17-year-olds performing surgery and driving out competent surgeons who didn’t pursue management jobs. It also doesn’t want people taking obscene risks due to an up-or-out culture that persists into one’s 40s and 50s. Finally, it doesn’t want a culture where the median practitioner has less than a decade of experience.

The pyramid is the entropic formation. Without a deliberate professional structure, it will emerge. Why so? An unprotected “profession” (note the oxymoron) will expand until it absorbs the most unqualified people, who draw barely more compensation than they’d get anywhere else, but who are able enough to convince people (unqualified buyers) to buy their services that they can stay afloat. The unqualified and minimally invested (the “quacks”) will, by sheer numbers, outvote the genuine practitioners (who’ve often dedicated their lives to becoming good at something) and dominate the culture– until their incompetence starts hurting people. In medicine and law, this outcome is considered so unacceptable that it’s not controversial for these industries to have professional structures that function as labor cartels. We need to start thinking the same way about software. We need to start driving out the charlatan brogrammers for whom the infantilizing open-plan startup culture was created.

So how do we get an obelisk-shaped, protected profession for software engineers?

Let’s focus on something that hasn’t worked, and look at why it has failed. Many companies have created “engineering ladders” that purport to measure professional development for individual contributors and allow people to grow without becoming managers. There might be a manager-equivalent title (Staff Engineer) and a Director-equivalent one (Principal Engineer, or Architect) and a VP-equivalent one (Fellow). What goes wrong? To start, the disparity in difficulty makes the whole thing extremely self-congratulatory on the part of the managers.

To become a Director- or VP-equivalent engineer, you have to compete against some of the smartest people in the company. It’s legitimately difficult. There are smart, qualified people who don’t make it. If anything, these dual-track systems serve the interests of management by allowing the executives to overstate their value and capability relative to engineers; if one has to be a genuinely great engineer to become Director-equivalent, the thinking is, then engineers will hold Directors and VPs on the much-easier-to-climb managerial tracks in higher regard.

Ultimately, these systems don’t work. Rather, they serve to document and justify an existing pyramidal scarcity. The organization will only allow a small number of engineers to climb into positions with the credibility and compensation afforded to senior managers, because engineers lack the collective organizational skill to thwart the penny-pinching game being played against them. Thus, a pyramid forms, and little is solved, because positional scarcity persists at each level.

I’ve been around for long enough to know that trusting employers is not a strategy. In the long term, it always fails. Don’t get me wrong: there are good companies out there, and there are managerial strategies (e.g. open allocation) that can improve the odds of creating one. Some individual employers probably can be trusted, but this doesn’t scale, because most organizations become pathological and self-serving at some point. So, if we can’t trust employers, then we can’t put stock in employer-assigned titles as a way to insure the credibility of the individual programmer. That means that we need an employer-independent system for doing so. We need a source of credibility for the individual that no employer can touch.

Doctors and attorneys have employer-independent credibility, being vouched-for by their professional associations (in the U.S., the AMA and the ABA). That’s necessary because the core of a profession is that one’s ethical obligations supersede managerial authority. That is, a doctor can’t use the superior orders defense to justify killing a patient. Of course, that only works if the professional has some sort of employer-independent credibility– in which case, losing a job is a transient financial annoyance rather than a possible career killer. If the individual lives entirely on his reputation as made by prior managers, his lack of leverage becomes a lack of moral agency. That is, of course, the problem that most programmers face. We lack moral or even creative autonomy, because we lack employer-independent credibility; once we leave school, our value is derived from the titles and recognitions given to us by profit-seeking corporations.

Some programmers hoped that the open-source world would fill this gap. It can’t. That’s not its job. Evaluating a programmer’s Github profile is a subjective task, and the only objective way of measuring one’s open-source contributions is based on how many people use the software. Here’s the issue, though. Few engineers have the time and resources available not only to build usable open-source projects, but to market their work to the global programming community enough that it actually used, and then to support it once it is. The positional scarcity and power-law distribution of open-source prominence emerge not because the open-source community is full of mean people (it’s not) but because (a) there’s no barrier to entry when it comes to just putting code out there, while (b) most of this code will never be used, because people have an entirely understandable (and, in fact, prudent) aversion to using unproven software.

How do we create an employer-independent credibility for software engineers? As far as I can tell, there are two solutions. The first is one that I hate: to require formal, institutional education. I hate it because it’s hostile to career-switchers, astronomically expensive, and can generate a culture of anti-intellectual credentialism. This leaves the second, which is to create an exam-based system, similar to what the actuarial sciences use. As programmers pass more exams, their professional credibility can grow in an employer-independent way. This would not only give us job security, but the improvement in our general leverage would enable us to negotiate for better working conditions and launch better projects, to the benefit of society as a whole.

Are there problems with exams? Of course there are. For one thing, I happen to believe that “bad test taker” is a real thing. (I was always a good test taker, but I’ve met intelligent people who aren’t.) We’d have to adapt our system to legitimate learning disabilities and to allow project-based alternatives for great programmers who struggle with traditional exams. We’d need to decide what to charge for the exams (as well as the presumably more expensive project evaluation) and how to offer financial aid. We’d have to decide what would be on the exams, and how many there would be. (I, personally, doubt that I’d be qualified to design such a curriculum.) What is an Associate-level programmer, and what’s a Fellow-level engineer, and what do we expect each to know? What are the continuing education requirements going to look like? These aren’t easy problems to solve. On the contrary, they’re quite difficult and organizationally complex.

Here’s the thing, about that. Designing a curriculum and exam track for software engineers is a massive amount of work, much of it ugly. I doubt I’m up to it, personally, because I’d rather have it built by experts in their respective fields of computer science. However, the core idea is here is one that will actually work. Blindly trusting employers hasn’t worked, and fetishizing startups has been a hilarious failure. Over the past twenty years, the programming profession has lost ground. We’ve been infantilized with open-plan offices and “Agile Scrum” practices that senior engineers don’t need. We’ve been left without any form of employer-independent credibility, which is what we’ll need if we want to become a true profession. We’re in a state of squalid failure. Our professional shape is a pyramid rather than an obelisk, making the job painfully political at every level. I’d rather see an imperfect solution (such as a professional curriculum and an employer-independent, exam-based advancement system) that fixes this than no solution at all. This problem won’t be solved by one person and it certainly won’t be solved within a year, but I’m confident that I know, at least, where to start.

Insights into why a contravariant type can’t be a Haskell Functor

In learning Haskell’s core types, type classes, and concepts, one often finds counterexamples useful in learning what certain abstract concepts “really are”. One of the more commmon type class hierarchies is Functor-Applicative-Monad. We encounter Functor in the context of things that are “mappable”. It’s a producer, and using fmap we can “post-transform” all of the things that it produces in a way that preserves any underlying structure.

We quickly become familiar with a number of useful functors, like Maybe, [], (->) r, IO. Functors seem to be everywhere. So, one asks, what might be a parameterized type that’s not a functor? The go-to example is something that’s contravariant in its type parameter, like a -> Int, as a type-level function of parameter a. There doesn’t seem to be any useful way to make it a Functor in a. That said, seemingly “useless” type-level artifacts are often quite useful, like the Const Functor, defined as Const r a ~= r (the a is a phantom type and functions being fmaped are ignored). So usefulness in this world isn’t always intuitive. Still, it remains true that something like a -> Int is not a functor? Why?

Let’s work with the type a -> Bool, just because it’s the simplest example of what goes wrong. Is there a way to make a Functor instance for that? To me, it’s not intuitive that that thing isn’t a Functor. It’s “almost” a set (stepping around, for a moment, debates about what is a set) and Set, meaning the actual collection type in Data.Set is “almost” a Functor. It’s not one, because a Functor demands that the mapping be structure-preserving, which a set’s notion of mapping is not (for example, if you map the squaring function on the set {2, -2}, you get a smaller set, {4}). There are many ways to get at the point that Set is not a Functor, most relying on the fact that Set a‘s methods almost invariably require Eq a and Ord a. But what about the type-agnostic “(<-) a” (my notation) above? It places no such constraints on a.

To answer this, let’s try to create the method, fmap, on which Functor relies.


-- pretend Haskell supports (<-) b a as a syntax for (->) a b = a -> b

instance Functor (<-) Bool where

-- fmap :: (a -> b) -> f a -> f b
-- in this case, (a -> b) -> (a -> Bool) -> (b -> Bool)
fmap f x = ??

The type signature that we’re demanding of our fmap is an interesting one: (a -> b) -> (a -> Bool) -> (b -> Bool). Notice that such a function doesn’t have any a to work with. One might not exist: the Functor needs to be valid over empty types (e.g. Void; note that [Void] is not empty but has exactly one element, []). In typical Functor cases, the a = Void case is handled soundly. For example, consider the list case: ([] :: [Void] maps to [] :: [b] for any b), and any non-empty list has as to work with. In this case, fmap‘s type signature gives us two things that we can do with a‘s but no a. This means that we have to ignore the first two arguments; we can’t use them.

instance Functor (<-) Bool where

-- fmap :: (a -> b) -> f a -> f b
-- in this case, (a -> b) -> (a -> Bool) -> (b -> Bool)
fmap _ _ = (?? :: b -> Bool)

This rules out any useful Functors. In fact, our need for an expression that inhabits b -> Bool for any b limits us to two non-pathological possibilities: the constant functions! Since we don’t have any b to work with, either, we have to commit to one of those. Without loss of generality, I’m going to use the definition fmap _ _ = const True. With a few tweaks to what I’ve done, Haskell will actually allow such a “Functor” instance to be created. But will it be right? It turns out that it won’t be. To show this, we consult the laws for the Functor type class. In fact, we only need the simplest one (Identity): fmap id === id to refute this one. That law is violated by the constant “Functor” above:

fmap id (const True) = const True -- OK!
fmap id (const False) = const True -- whoops!

So it turns out that (<-) Bool (my notation) has no Functor instances at all (the only properly polymorphic candidate fails the type class’s most basic law) and the more complicated cases fail similarly.

Why “Agile” and especially Scrum are terrible

Agility is a good thing, no doubt, and the Agile Manifesto isn’t unreasonable. Compared to a straw-man practice called “Waterfall”, Agile is notably superior. Yet, so much of Agile as-practiced is deeply harmful, and I don’t really think that the Agile/Waterfall dichotomy is useful in the first place.

There’s a variety of Agile, called Scrum, that I’ve seen actually kill a company. By “kill”, I don’t mean “the culture wasn’t as good afterward”. Rather, I mean that its stock dropped by almost 90 percent in less than two years.

What is Agile?

Agile grew up in web consulting, where it had a certain amount of value: when dealing with finicky clients who don’t know what they want, one typically has to choose between one of two options. The first is to manage the client: get expectations set, charge appropriately for rework, and maintain a relationship of equality rather than submission. The second is to accept client misbehavior (as, say, many graphic designers must) and orient one’s work flow around client-side dysfunction.

Programmers tend not to be great at managing clients. We’re very literal people. We like systems that have precisely defined behaviors. This makes working with non-technical people harder for us than it needs to be, because we tend to take every request literally rather than trying to figure out what they actually want. Corporate Agile, removed from the consulting environment, goes further and assumes that the engineers aren’t smart enough to figure out what their internal “customers” want. This means that the work gets atomized into “user stories” and “iterations” that often strip a sense of accomplishment from the work, as well as any hope of setting a long-term vision for where things are going.

In general, people tend to create two types of jobs, whether inside a company or as clients when off-loading work. At the high end, they hire for expertise that they don’t have. At the low end, they dump all the stuff they don’t want to do. It’s probably obvious that one class of consultant gets respect and the other doesn’t. Mismanaged consulting firms often end up becoming garbage disposals for the low end work. Scrum seems to be tailored toward the body shops, where client relationships are so mismanaged that the engineers have to be watched on a daily basis, because they’ve become a dumping ground for career-incoherent work that no one wants to do (and that probably isn’t very important, hence the low rate and respect).

 

Violent transparency

There are a lot of workplace trends that are making the programming career extremely unattractive, especially to the sorts of creative, intelligent people that it’s going to need in order to fill the next generation.

Open-plan offices are the most egregious example. They aren’t productive. It’s hard to concentrate in them. They’re anti-intellectual, insofar as people become afraid to be caught reading books (or just thinking) on the job. When you force people to play a side game of appearing productive, in addition to their job duties, they become less productive. Open-plan offices aren’t even about productivity in the first place. It’s about corporate image. VC-backed startups that needed to manage up into investors used these plans to make their workspaces look busy. (To put it bluntly, an open-plan programmer is more valued as office furniture than for the code she writes.) For a variety of cultural reasons, that made the open-plan outfit seem “cool” and “youthy” and now it’s infesting the corporate world in general.

It’s well known that creative people lose their creativity if asked to explain themselves while they are working. It’s the same with software. Programmers often have to work in an environment of one-sided transparency. These Agile systems, so often misapplied, demand that they provide humiliating visibility into their time and work, despite a lack of reciprocity. Instead of working on actual, long-term projects that a person could get excited about, they’re relegated to working on atomized, feature-level “user stories” and often disallowed to work on improvements that can’t be related to short-term, immediate business needs (often delivered from on-high). This misguided but common variant of Agile eliminates the concept of ownership and treats programmers as interchangeable, commoditized components.

Scrum is the worst, with its silliness around two-week “iterations”. It induces needless anxiety about microfluctuations in one’s own productivity. There’s absolutely no evidence that any of this snake oil actually makes things get done quicker or better in the long run. It just makes people nervous. There are many in business who think that this is a good thing because they’ll “work faster”. I’ve been in software for ten years, as a manager and a worker bee. It isn’t true.

Specific flaws of “Agile” and Scrum

1. Business-driven engineering.

Agile is often sold in comparison to “Waterfall”. What is Waterfall?

Waterfall is legitimately terrible. It’s a “work rolls downhill” model in which each tier of the organizational hierarchy picks off what it consider to be “the fun stuff”, and passes the rest down the line. Projects are defined by executives, design is done by architects, personal deadlines are set by middle managers, implementation is performed by the top-tier grunts (programmers) and then operations and testing are handed off to the lower tiers of grunts. It’s extremely dysfunctional. When people have a sense that all of the important decisions have been made, they’re not motivated to do their best work.

Waterfall replicates the social model of a dysfunctional organization with a defined hierarchy. Agile, quite often, replicates the social model of a dysfunctional organization without a well-defined hierarchy.

I have mixed opinions about job titles like “Senior” and “Principal” and the like. They matter, and I probably wouldn’t take a job below the Principal/Director-equivalent level, but I hate that they matter. If you look at Scrum, it’s designed to disentitle the senior, most capable engineers who have to adhere to processes (work only on backlog items, spend 5-10 hours per week in status meetings) designed for people who just started writing code last month.

Like a failed communist state that equalizes by spreading poverty, Scrum in its purest form puts all of engineering at the same low level: not a clearly spelled-out one, but clearly below all the business people who are given full authority to decide what gets worked on.

Agile, as often implemented, increases the feedback frequency while giving engineers no real power. That’s a losing bargain, because it means that they’re more likely to be jerked around or punished when things take longer than they “seem” they should take. It creates a lot of anxiety and haste, but there’s little of what actually matters: building excellent products.

Silicon Valley has gotten a lot wrong, especially in the past five years, but one of the things that it got right is the concept of the engineer-driven company. It’s not always the best for engineers to drive the entire company, but when engineers run engineering and set priorities, everyone wins: engineers are happier with the work they’re assigned (or, better yet, self-assigning) and the business is getting a much higher quality of engineering.

If your firm is destined to be business-driven, that’s fine. Don’t hire full-time engineers, though, if you want talent. You can get the best people as consultants (starting in the $200-per-hour range, and going up steeply from there) but not as full-timers, in an business-driven company. Good engineers want to work in engineer-driven firms where they will be calling shots regarding what gets worked on, without having to justify themselves to “scrum masters” and “product owners” and layers of non-technical management.

Ultimately, Agile (as practiced) and Waterfall both are forms of business-driven engineering, and that’s why neither is any good at producing quality software or happy employees.

2. Terminal juniority

Software engineering, unfortunately, has developed a culture of terminal juniority, lending support to the (extremely misguided) conception of programming as a “young man’s game”, even though almost none of the best engineers are young, and quite a few of the best are not men.

The problem with Agile’s two-week iterations (or “sprints”) and user stories is that there is no exit strategy. There’s no “We won’t have to do this once we achieve [X]” clause. It’s designed to be there forever: the business-driven engineering and status meetings will never go away. Architecture and R&D and product development aren’t part of the programmer’s job, because those things don’t fit into atomized “user stories” or two-week sprints.

As a result, the sorts of projects that programmers want to take on, once they master the basics of the craft, are often ignored, because it’s annoying to justify them in terms of short-term business value. Technical excellence matters, but it’s painful to try to sell people on this fact if they want, badly, not to be convinced of it.

I once worked at a company where a product manager said that the difference between a senior engineer and a junior engineer was the ability to provide accurate estimates. Um, no. That’s insulting, actually. I hate estimates, because they generate politics and don’t actually make the work get done faster or better (in fact, it’s usually the opposite).

The worst thing about estimates is that they push a company in the direction of doing work that’s estimable. This causes programmers to favor the low-yield, easy stuff that the business doesn’t actually need (even if bumbling middle managers think otherwise) but that is “safe”. Anything that’s actually worth doing has a non-zero chance of failure and too many unknown unknowns for estimates to be useful. Agile’s focus on short-term business value ends up pushing people away from the kinds of work that the company actually needs, in favor of each programmer’s real-time reputation management.

What this culture says to me is that programming is a childish thing to be put away by age 35. I don’t agree with that mentality. In fact, I think it’s harmful; I’m in my early 30s and I feel like I’m just starting to be good at programming. Chasing out our elders, just because they’re seasoned enough to know that this “Agile”/Scrum process has nothing to do with computer science and that it adds no value, is a horrible idea.

3. It’s stupidly, dangerously short-term. 

Agile is designed for and by consulting firms that are marginal. That is, it’s for firms that don’t have the credibility that would enable them to negotiate with clients as equals, and that are facing tight deadlines while each client project is an existential risk. It’s for “scrappy” underdogs. Now, here’s the problem: Scrum is often deployed in large companies and funded startups, but people join those companies (leaving financial upside on the table, for the employer to collect) because they don’t want to be underdogs. They want safety. That’s why they’re in those jobs, rather than starting their own companies. No one wants to play from behind unless there’s considerable personal upside in doing so. “Agile” in a corporate job means pain and risk without reward.

When each client project represents existential or severe reputational risk, Agile might be the way to go, because a focus on short-term iterations is useful when the company is under threat and there might not be a long term. Aggressive project management makes sense in an emergency. It doesn’t make sense as a permanent arrangement; at least, not for high-talent programmers who have less stressful and more enjoyable options.

Under Agile, technical debt piles up and is not addressed because the business people calling the shots will not see a problem until it’s far too late or, at least, too expensive to fix it. Moreover, individual engineers are rewarded or punished solely based on the optics of the current two-week “sprint”, meaning that no one looks out five “sprints” ahead. Agile is just one mindless, near-sighted “sprint” after another: no progress, no improvement, just ticket after ticket after ticket.

People who are content to shuffle mindless through their careers can work that way, but all of the really good engineers hate it when they go to work on a Monday morning and don’t know what they’ll be working on that day.

4. It has no regard for career coherency.

Atomized user stories aren’t good for engineers’ careers. By age 30, you’re expected to be able to show that you can work at the whole-project level, and that you’re at least ready to go beyond such a level into infrastructure, architecture, research, or leadership.

In an emergency, whether it’s a consultancy striving to appease an important client or a corporate “war room”, career coherency can wait. Few people will refuse to do a couple weeks of unpleasant or otherwise career-incoherent work if it’s genuinely important to the company. The work’s importance creates the career benefit. If you can say “I was in the War Room and had 20 minutes per day with the CEO”, that excuses having done grunt work.

When there’s no emergency, however, programmers expect their career growth to be taken seriously and will leave if it’s not. Saying, “I was on a Scrum team” says, “Kick me”. It’s one thing to do grunt work because the CEO recognized that an unpleasant task would add millions of dollars of value (and that he’d reward it accordingly) but doing grunt work just because you were assigned “user stories” and Jira tickets shows that you were seen as a loser.

5. Its purpose is to identify low performers, but it has an unacceptably high false positive rate. 

Scrum is sold as a process for “removing impediments”, which is a nice way of saying “spotting slackers”. The problem with it is that it creates more underperformers than it roots out. It’s a surveillance state that requires individual engineers to provide fine-grained visibility into their work and rate of productivity.

There’s a fallacy in civil liberties debates: the “nothing to hide” argument. Some people like to argue that invasions of privacy– by, say, law enforcement– are non-issues because they’re not themselves criminals. It misses a few key things. For one thing, laws change. Ask anyone who was Jewish in Central Europe in the 1930s. Even for respectable people, a surveillance state is an anxiety state.

The fact of being observed changes the way people work. In creative fields, it’s for the worse. It’s almost impossible to work creatively if one has to justify the work on a day-by-day basis.

Another topic coming to mind here is status sensitivity. Programmers love to make-believe that they’ve transcended a few million years of primate evolution related to social status, but the fact is: social status matters, and you’re not “political” if you acknowledge the fact. Older people, women, racial minorities, and people with disabilities tend to be status sensitive in the workplace, because it’s a matter of survival for them. Constant surveillance into one’s work indicates a lack of trust and low social status, and the most status-sensitive people (even if they’re the best workers) are the first ones to decline when surveillance ramps up. If they feel like they aren’t trusted (and what else is communicated by a culture that expects every item of work to be justified?) then they lose motivation quickly.

Agile and especially Scrum exploit the nothing-to-hide fallacy. Unless you’re a “low performer” (witch hunt, anyone?) you shouldn’t mind a daily status meeting, right? The only people who would object to justifying their work in terms of short-term business value are the “slackers” who want to steal from the company, correct? Well, no. Obviously not.

The violent transparency culture is designed for the most status-insensitive people: young, usually white or Asian, privileged, never-disabled males who haven’t been tested, challenged, or burned yet at work. It’s for people who think that HR and management are a waste of time and that people should just “suck it up” when demeaned or insulted.

Often, it’s the best employees who fall the hardest when Agile/Scrum is introduced, because R&D is effectively eliminated, and the obsession with short-term “iterations” or sprints means that there’s no room to try something that might actually fail.

The truth about underperformers is that you don’t need “Agile” to find out who they are. People know who they are. The reason some teams get loaded down with disengaged, incompetent, or toxic people is that no one does anything about them. That’s a people-level management problem, not a workflow-level process problem.

6. The Whisky Goggles Effect.

There seems to be some evidence that aggressive management can nudge the marginally incompetent into being marginally employable. I call this the Whisky Goggles Effect: it turns the 3s and 4s into 5s, but it makes you so sloppy that the 7s and 9s want nothing to do with you. Unable to get their creative juices flowing under a system where everything has to be justified in terms of short-term business value, the best programmers leave.

From the point of view of a manager unaware of how software works, this might seem like an acceptable trade: a few “prima donna” 7+ leave under the Brave New Scrum, while the 3s and 4s become just-acceptable 5s. The problem is that the difference between a “7” programmer and a “5” programmer is substantially larger than that between a “5” and a “3”. If you lose your best people and your leaders (who may not be in leadership roles on the org-chart) then the slight upgrade of the incompetents, for whom Scrum is designed, does no good.

Scrum and Agile play into what I call the Status Profit Bias. Essentially, many people in business judge their success or failure not in objective terms, but based on the status differential achieved. Let’s say that the market value of a “3” level programmer is $50,000 per year and, for a “5” programmer, it’s $80,000. (In reality, programmer salaries are all over the map: I know 3’s making over $200,000 and I know 7’s under $70,000, but let’s ignore that.) Convincing a “5” programmer to take a “3”-level salary (in exchange for startup equity!) is marked, psychologically, not as $30,000 in profit (which is tiny) but as a 2-point profit.

Agile/Scrum and the age discrimination culture in general are about getting the most impressive status profits, rather than actual economic profits that would be attained, usually, by hiring the people who’ll deliver the most value (even if you go 50% above market rate, because top programmers are worth 10-30 times their market rate).

The people who are least informed about what social status they “should” have are the young. You’ll find a 22-year-old 6 who thinks that he’s a 3 and who will submit to Scrum, but the 50-year-old 9 is likely to know that she’s a 9 and might begrudgingly take 8.5-level conditions but she’s not about to drop to a 3, or even a 6. Seeking status profits is, of course, useless. These points don’t mean anything. You win in business by bringing in more money than you pay out in costs, not by exerting margins in meaningless status points. There may be a whole industry in bringing in 5-level engineers and treating (and paying) them like 4’s, but under current market conditions, it’s far more profitable to hire an 8 and treat him like an 8.

7. It’s dishonestly sold.

To cover this point, I’m going to focus on Scrum. Some people argue that Scrum is “extremist Agile” and others say that it’s the least Agile variant of Agile. (This, perhaps, underlies one of the problems here; we can hardly agree on what is and is not “Agile”.)

Something like Scrum has its place: a very limited, temporary one. The dishonest salesmanship is in the indication that it works everywhere, and as a permanent arrangement.

What Scrum is good for

Before the Agile fad made it a permanent process, “Scrum” was a term sometimes used for what corporations might also call a “Code Red” or a “War Room emergency”. If there was an unexpected, rapidly-evolving problem, you’d call together your best people and form a cross-cutting team.

This team would have no clear manager, so you’d elect a leader (like a “Scrum Master”) and give him authority. You’d want him not to be an official “people manager” (since he needs to be as impartial as possible). Since the crisis is short-term, individuals’ career goals can be put on hold. It’s considered a “sprint” because people are expected to work as fast as they can to solve the problem, and because they’ll be allowed to go back to their regular work once it’s over. Also, in this kind of emergency, you need to make it clear that no one is being evaluated individually. The focus is on the mission and the mission only.

When you impose process and demand one-sided transparency of all the workers, people feel watched. They get the sense that they’re being stalked and will be labelled “low performers” as soon as their week-by-week, individual fluctuations go the wrong way. So they become resentful. That’s dysfunctional. On the other hand, when you call together a group of known high-performers to handle a specific and time-limited crisis, they don’t mind giving frequent status updates so long as they know that it’s the exigency of the situation, rather than their own low social status, that is the reason behind it.

There are two scenarios that should come to mind. The first is a corporate “war room”, and if specific individuals (excluding high executives) are spending more than about 4 weeks per year in war-room mode, than something is wrong with the company, because emergencies ought to be rare. The second is that of a consultancy struggling to establish itself, or one that is bad at managing clients and establishing an independent reputation, and must therefore operate in a state of permanent emergency.

Two issues

Scrum, in this way, acknowledges the idea that emergency powers must sometimes be given to “take-charge” leaders who’ll do whatever they consider necessary to get a job done, leaving debate to happen later. That’s fine. It’s how things should be, sometimes.

A time-honored problem with emergency powers is that they often don’t go away. In many circumstances, those empowered by an emergency see fit to prolong it. This is, most certainly, a problem with management. Dysfunction and emergency require more managerial effort than a well-run company in peace time. The result is that there are many managers, especially in technology, who seek out opportunities for emergencies and emergency powers.

It is also more impressive for an aspiring demagogue (a “scrum master”?) to be a visible “dragonslayer” than to avoid attracting dragons to the village in the first place. The problem with Scrum’s aggressive insistence on business-driven engineering is that it makes a virtue (“customer collaboration”) out of attracting, and slaying, dragons (known as “requirements”) when it might be more prudent not to lure them out of their caves in the first place.

“Agile” and Scrum glorify emergency. That’s the first problem with them. They’re a reinvention of what the video game industry calls “crunch time”. It’s not sustainable to work that way. An aggressive focus on individual performance, a lack of concern for people’s career objectives in work allocation, and a mandate that people work only on the stated top priority, all of these provide a lot of value in a short-term emergency, but become toxic and demoralizing in the long run. People will tolerate short-term losses of autonomy, but only if there’s a clear point ahead when they’ll get their autonomy back.

The second issue is that these practices are sold dishonestly. There’s a whole cottage industry that has grown up around teaching companies to be “Agile” in their software development. The problem is that most of the core ideas aren’t new. The terminology is fresh, but the concepts are mostly outdated, failed “scientific management” (which was far from scientific, and didn’t work). Again, these processes sometimes work toward hitting well-defined targets in emergency circumstances, but permanent Scrum is a disaster.

If the downsides and upsides of “Agile” and Scrum were addressed, then I wouldn’t have such a strong problem with the concepts. If a company has a team of only junior developers and needs to deliver features fast, it should consider using these techniques for a short time, with the plan to remove them as its people grow and timeframes become more forgiving. However, if Scrum were sold for what it is– a set of emergency measures that can’t be used to permanently improve productivity– then there would be far fewer buyers for it, and the “Agile” consulting industry wouldn’t exist. Only through a dishonest representation of these techniques (glorified “crunch time”, packaged as permanent fixes) are they made salable to the corporate world (“the enterprise”) at large.

Looking forward

It’s time for this culture of terminal juniority, low autonomy, and aggressive management to die. These aren’t just bad ideas. They’re more dangerous than that, because there’s a generation of software engineers who are absorbing them without knowing any better. There are far too many young programmers being doomed to mediocrity by the idea that business-driven engineering and “user stories” are how things have always been done. This ought to be prevented; the future integrity of our industry may rely on it. “Agile”, at least as bastardized in every implementation that I’ve seen, is a bucket of nonsense that has nothing to do with programming and certainly nothing to do with computer science. Business-driven engineering, in general, is a dead end. It ought to be tossed back into the muck from which it came.

My name is cleared.

This came from a very high-ranking person at Google, earlier today. It is posted here with the author’s permission.

Dear Michael,

Your continuing outspokenness, with regard to your time at Google, has come to my attention. After reading over your record in detail, I and many others on the senior leadership team agree that you were poorly managed and inaccurately reviewed. Your performance was strong given the short time you were here, and you showed very high potential for growth.

We find it regrettable that you left Google. Having reviewed your history, we want to impress upon you that we see your experiences as unjust and certainly not reflective on you in any negative way but, additionally, extremely unusual for Google. We hope that you will also see it this way, and not continue to regard it as reflecting any underlying pattern at Google. Had you been under different supervision, we believe that you would have been very successful here. You are clearly a driven, passionate, and talented technologist.

We wish for every person we hire to have a productive and successful career here, and of all the companies where I have worked, Google does the best job of achieving this. However, perfection in management is impossible for any company, especially given our size and the fast-paced nature of technology in general.

We wish you the best of success in your career going forward.

Sincerely,

[Googler]

My name is cleared.

I’m done criticizing Google. On the whole, Google is a great technology company in many ways, and has plenty of exceptional people with whom it was a privilege to work.

It’s time to move on. Back to building.