I’ll cold-open with a somewhat pointed, but valid question: why are there so few great software engineers? The answer: the software industry makes it extremely difficult to become good at engineering, unless a person is economically autonomous (read: wealthy, or well-connected enough to secure venture funding). Here are some of the obstacles that engineers face if they want to become great.
- Low expectations of individual contributors. Non-managerial programmers are rarely expected, in most work environments, to have multiplier effects that make the team more productive, and therefore are rarely able to have the kinds of infrastructural contributions that would bring them feedback on what works, architecturally speaking, and what doesn’t. Instead, they’re often put to work on parochial business problems of low general interest or application, and often their contributions are to maintain some small piece of a massive system, so they rarely get a sense of large-scale engineering.
- Rarity of high-quality work. The kind of work that engineers need in order to become great is uncommon and requires substantial concentration. It’s almost impossible to get it without managerial buy-in (because one needs freedom from interruption and low-quality distractions) and, for the tiny amount of such work that companies allow to exist, it is allocated politically, rather than based on any sensible principles.
- Obscene program size. When the program-to-programmer relationship is many-to-one, one gets sensible and small programs and highly productive engineers, but individual efforts are hard to track and measure. Consequently, many managers prefer a world where the program-to-programmer relationship is one-to-many and “headcount” can be tracked at a “per-program” level. These massive single-program teams tend to end up in a state where one person (the “blazer”) is highly productive and the others (the “trailers”) spend the bulk of their time keeping up with the changes, so individual productivity is low. Why is this low productivity tolerable? Because software engineers are massively underpaid compared to the business value they actually provide, so it’s acceptable (from a corporate perspective) to have them running at 25% efficiency.
- Political distractions. When it takes as much time to prove that one is qualified to do interesting work as can be spent actually doing it, it becomes hard to maintain a high level of competence, much less improve upon it. It takes three to four years of campaigning in most companies to be considered credible enough to get the best projects, and by this point, a person’s technical skills have declined.
All of these issues exist, but there’s an obvious question. For high-potential software engineers, why don’t the political problems “solve themselves”? If they’re so good, why aren’t their managers falling over themselves to clear obstacles and let them do what they do best? Businesses would benefit immensely if they could discover and tap the top talent, so why don’t they? The answers are inherent in the concept of “The Design Paradox”, which I’ve split into two related “paradoxes”, one psychological and the other political.
First Design Paradox: people who lack design sense (“taste”) cannot assess it in others.
People with taste but not vision can recognize tasteful visionaries apart from charlatans, but people without taste are hopeless. This is a serious problem in software, because what differentiates a good programmer from a bad one is not how fast code can be written or how many lines are produced per day, but having the taste necessary to design usable, maintainable systems. Are those technical assets actually usable, or will they be tomorrow’s despised legacy horrors? This specifically technological variety of taste needs to be refined with experience, and that experience is hard to come by, so the taste itself is rare. People who don’t interact with software at a code level on a regular basis will have no intuition for what beautiful software is. You’re not likely to find this taste in a person off the street, and if there isn’t a strong reason to believe you have it, you almost certainly don’t.
This requires that technological taste be present at the very top of the company. The CEO will generally not be reading source code, but if the CEO lacks taste, then getting a tasteful CTO or VP/Eng is a crapshoot. If the CTO lacks taste, he won’t be able to tell which of his reports are capable and which are charlatans. Charlatans tend to be more skilled at office politics– they have more experience, due to a lifelong need to compensate for their own mediocrity– so there’s adverse selection involved.
What should a CEO do if he’s self-aware to understand his lack of taste in software architecture? Something very few managers like to do: give back some control. I think one of the reasons why Valve’s open allocation works so well is that it’s an admission of a reality that exists in every complex company: that executives don’t have all the answers. Workers are trusted to vote with their feet and throw their efforts behind the most tasteful projects– and they don’t have to wrangle with a rusty, corrupt permission system in order to do so.
Most discussions of “The Design Paradox” pertain to the first one, but I’m more interested in the second.
Second Design Paradox: people with taste will perform worse in environments with a history of poor taste, giving tasteless people a competitive advantage.
By “perform worse”, I mean that they will be worse at their jobs, and politically less successful. This is actually somewhat counter-intuitive. It’s akin to saying that the best writers are worse than average people at reading poorly-constructed prose– probably not true, for that particular example. In programming, however, it is true. Under a certain definition of “worst” (which I’ll explain, below) the best writers of code are the worst readers of it. That’s right. A critical skill for a corporate software engineer (reading other people’s code) is, under a certain set of circumstances, strongly negatively correlated to the ability to design good systems. What is that set of circumstances? It’s when the systems that must be understood are themselves incoherent or badly designed. Everyone struggles with it, but the best programmers have even more of a problem with bad code. Good designers will learn good technologies quickly and struggle immensely with bad ones.
I know this from painful experience. When I read code, I’m trying to connect what I am reading with how I would solve the problem. Usually, there are discrepancies, and if it’s good code, these are informative. Whenever I encounter a difference between my mental model of the problem and how it is actually solved, it helps me understand it in a deeper and more detailed way. So I enjoy reading good code, just as I enjoy reading a well-written mathematical paper, because there’s a convergence that occurs as I really learn something. The problem with bad code is that there’s such a stark and irreducible difference between how I would solve it and the way it is done, because it is done so tastelessly, and also because the only thing learned from it is parochial garbage pertaining to how ineffective or poorly-trained programmers think. You don’t learn the fundamentals of computer science from an AbstractProxyVisitorSingletonFactory. I can’t find convergence with a pile of FactoryFactory garbage because I would never write FactoryFactory garbage. There is simply no problem or set of circumstances (except a desire to severely punish an employer) that could bring me to write that kind of code, ever.
The Second Design Paradox is an aesthetic cousin of the Peter Principle, which is that “employees tend to rise to their level of incompetence”. There are two ways one can approach this. The first, which is wrong, is the argument that higher-level positions in a company are more difficult to perform, and that people therefore ascend as long as they are competent but reach a point where the work is too difficult for them. The reason this is wrong is that, although it is abstractly more difficult to do an executive job well, the managerial ranks of almost every company are so self-protecting and political that merit and promotion divorce early, and it becomes easier to succeed politically as one ascends. The standards of personal integrity and intelligence are lower once one has “made it”. In most companies, it’s hard to get into the managerial or executive “club”, but relatively easy, once in, to keep rising. In fact, the tribal self-protection of managers is at such an extreme that the obviously incompetent executives get what are known as “exit promotions”, which make their titles more attractive so they can “fail up” into another company and become someone else’s problem.
The second interpretation of the Peter Principle, which is the right one, is that companies tend to tap people for leadership roles based on their success as subordinates, while good leaders make shitty subordinates and good subordinates make ineffective leaders, because the ones who subordinate well are the people who just don’t care about doing things right.
Now I have to reconcile two statements that seem to contradict each other. One is the polite maxim that “Good leaders know when to follow, and do it well”. That’s true. The other is “Good leaders make shitty subordinates”, which is equally true. How can this be? There is a fundamental difference between following and subordinating. One who follows is choosing to do so, because she perceives it to be better for the group and, thus, also herself, to let a more informed or impartial person make calls. That’s the hardest thing about being a leader: knowing when to trust your own judgment, when to defer to someone else’s, and when to step down outright. What I like about Valve’s model, with its absence of entitled, permanent leaders called “managers”, is the recognition of this fact. On the other hand, a person who subordinates is accepting authority outright. Subordination is unconditional following and obedience that comes out of a preternatural acceptance of rank. When you follow, you have a choice. When you subordinate, there is no choice. You are surrendering to rank.
Most corporate managers aren’t leaders, but an internal police force trained to enforce personnel policies that come from a “Theory X” belief that most workers are ethically somewhere between depravity and juvenility and will steal from the company (slacking and prioritization of one’s own career goals both included as “stealing”) if not closely supervised. The problem is that indoctrination into the managerial religion makes it harder to lead, because it embraces such a negative view of the people being managed. True leaders never show contempt for those who are following them. They recognize that they have a transient functional superiority, not an innate moral one, and that if it is better for the group for them to take an inferior (following) role, they should do so.
Design has a natural need for leadership, because of the need for simplicity in the finished product. Many should be encouraged to suggest ideas and avenues for exploration (divergent creativity) but one person must sit in the center with the task of pruning, simplifying and prioritizing (convergent creativity). It’s rare that you get a good design without exactly one person who takes on the “killing” role. If no one takes on that nasty job, conceptual integrity vanishes as requirements and boondoggles are added. If two or more people take it on, all with some share of veto power, then the political machinations that ensue often create a product whose shape reflects the parochial political environment in which it was formed (Conway’s Law) rather than any universal insight about a problem. Decentralized power is good for government because the objectives are global concerns like justice and sustainability that are too large for one party to manage, but for the finished aesthetics of a local entity like a new product, you actually need a single leader who will accept input from many (thereby following, at the right scale, when appropriate) but prune when it is required to preserve the integrity of the whole product.
The question, then, is whether human organizations are effective at bringing design leaders into leadership roles, and I would say, based on experience, that the answer is “no”. Why? Because these organizations tend to promote people based on their ability to subordinate, resulting in a low quality of leadership. They don’t have the design sense, at the top, to recognize emerging design leaders, but it’s obvious to them who are the best subordinates and who are the grumblers.
In software, anyone who takes a maintenance role is expected to follow, by definition. When you maintain code, you can’t just rip it to shreds because you think it’s badly designed, because you’ll break others’ work. You have to modify the software in accord with the contract (often unspecified) that it has set with its users. Even if the CTO of a company takes on a maintenance project (and sometimes this is exactly the right thing for a leader to do) she is taking on a role where she must, to some extent, follow. That’s not a bad thing.
However, in most large companies, maintenance isn’t done by people who choose it for the good of the company, but by people who get stuck on it because they’re junior and don’t have the clout to do anything better. They don’t have the power to do maintenance right. They chug along at 20 lines of code per month, fixing minor bugs and adding low-priority features, working below their frontier of ability, and waiting to be tapped for a role with more substance. They’re not following, because they don’t have a choice. If they voice their desire to work on something better, or assert their creativity in any substantial way, they’ll probably be fired within a year. They’re subordinate. People with design sense tend to be turned off by this sort of thing, so they rarely pay their dues and climb the ranks. The ones who do are those who never bothered to learn design sense, and who think “functional programming” is some academic mumbo-jumbo with no place in the real world.
In the long run, the result of this is that most corporate software is of very low design quality, which aggravates the problem. Left unchecked, it creates an arena where the lack of taste becomes a political advantage, because the people who lack taste are better equipped to adapt to the bad architectural decisions the group becomes stuck with. They rise into positions of authority and then make more bad design decisions, utterly inadvertently because bad designs are the only thing they’ve been exposed to. At this point, the organization isn’t just producing an occasional piece of bad software. It becomes a BadSoftwareFactory.