Gervais / MacLeod 18: more on trust, Square Root Syndrome, Brownian and Progressive Time

In Part 17, I discussed the financial considerations of starting a technology company financed by passive equity-holders. In that model, these investors are enabled to enjoy the high rate of return associated with human creative risk, but do not take an active management role. I used the term “lifestyle business” but I’ve since realized that I’m talking about something more specific: mid-growth businesses. “Lifestyle”, as I’m using it, isn’t about size, but about intended growth rate. It refers to businesses that prioritize long-term cultural health over rapid expansion, but that have a clear interest in growth itself. It’s not headcount or revenue growth that the mid-growth business optimizes for, but healthy growth: growth that doesn’t compromise the culture.

A low-growth business might be a restaurant with inherent scale limitations, or a “4-hour work week” business intended to run itself later in time. The ceiling is fairly low, and consequently there’s not a lot of interest in passive equity financing. Banks will make loans (debt financing) and usually require personal liability. Failure rates are considerable (business is always risky) but not so high as to make this completely untenable. On the contrast, a high-growth business is insistent on rapid growth– in headcount, revenues, footprint, and market dominance. 100 percent per year is barely socially acceptable; 150-200% is expected. Investors take an active managerial role and lose interest if growth falls short of 10% per month. The major downside of the high-growth business is that the vast majority run out of money and fail.

I’ve been trying to figure out a way to address the needs of the 1%- and 2%-per-month growth businesses. That doesn’t deserve to be sneezed at! If one could invest $10,000 into a business whose value grew reliably at 1.5% per month, that would turn into $356,000 after 20 years. That’s not the kind of thing that screams “fuck-you money” to thrill-seeking prospectors, but it certainly would make most passive investors happy. The conclusion I’ve come to is that we need passive equity financing of a very large number (a “fleet”) of highly capable but slow-growing (by VC-istan standards) businesses. Right now, regulations exist to keep “dumb money” away from such “lifestyle” businesses, judging small investors incapable of getting a decent deal, considering the “principal-agent problem” involved. My methodology (in the previous essay) fixes that by making compensation and profit sharing extremely simple and transparent while handling “HR expediencies” in a such a way that they don’t compound over time. Good. Solved that problem, at least on a technical front. That’s one of the major obstacles right now against my vision of connecting passive capital with human creativity in a way that doesn’t involve the career volatility and ethical compromises of VC-istan. It’s not the only obstacle, but it’s the biggest one.

Compensation is the harder of the two trust problems in organizations: do the owners (principals) trust the workers (agents) not to steal from them by overcharging or through various devious manipulations (e.g. “holding the company hostage” with key information)? Extreme transparency on compensation and culture helps a lot there. The more openness there is about the way decisions are being made, the more it is made clear that devious play-the-company-against-itself tricks won’t result in outsized personal yield, the less likely defection is. There’s a second question of trust, which is traditionally left to managers (owners don’t get involved): can we trust people to get the work done? That’s a fun one, too. Why do so many projects fail to ship? Why are so many people seemingly incompetent at self-executivity (including many actual executives)? That requires introducing the phenomenon of wasted time in an interesting context: Brownian time.

Brownian Time

I once found myself in a discussion about the value of an hour of time. A friend of mine were trying to determine whether it was possible to value work on hour-by-hour basis? We realized that, for most work days, only 3 hours (the square root of 9) actually mattered. The other 5-6 were spent in meetings, goofing off, or general “zoned out” low productivity, for most people. This seems to be the norm, and it’s not an intentional or morally bad thing. People just can’t hold intense concentration over an 8-hour contiguous block of time that someone else picked, five days in a row at the exact same time. It’s not possible.

We realized that this idea (“Square Root Syndrome”) applied to more than just hours in the day; it was visible at larger scales. Three hours of the workday really matter; the rest is wasted. Days in a week? It seems typical to have real victories on 2 out of the 5; the other 3 are unstellar and see minimal useful work. Stuff gets done, but rarely important stuff. Apply this to the 49 weeks in a typical work-year: 7 weeks pertain to real highlights– macroscopic achievements, lines on the resume– and the other 42 are forgettable. Then look at a 36-year software engineering career. Six years of the typical engineer’s career are spent on jobs that really deliver– lead to promotions, “fuck you money”, interesting contributions to humanity. The other 30? Wasted on executive pet projects, startups that go nowhere, ingrate bosses, and bad ideas. That’s not how it is for everyone, but those numbers seem pretty typical.

The depressing conclusion of this is that out of a whole career, only a little bit counts: 3 hours/day times 2 days/week times 7 weeks/year times 6 years gives us 252 hours that are really worth a damn. Of course, that’s not how actual careers work. It could be zero hours in a person’s career that count, meaning that there’s no progress and it isn’t really a career. It could be several thousand that matter. I’ll get to that later on.

This recursive “square root” relationship is what I call Brownian Time. It shows us the downside of unstructured, chaotic behaviors. If there’s no feedback or conscious work at using time properly, you get square-root scaling.  So you get twice as much out of a 4-hour meeting as you get out of a 1-hour meeting. (That’s generous; for most 4-hour meetings, one gets less.) I don’t know that the actual human pattern follows an exact power of 0.5, but it’s not far off. Why is it Brownian? It pertains to the a fractal pattern called Brownian Motion, which is a model for a variety of random processes including stock prices. If a stock has volatility (variance) of 1% per day, its volatility over a 256-day year is 16%. Most of the ups and downs cancel each other out. If one could call the good days in advance (and, of course, one can’t) then one could hold the stock for only a few days that year and realize all of its gains (or losses) in that small slice of the year. Brownian Motion is random “drift” that scales with the square root of time. If your goal is twice as far away, it’ll take 4 times as much drifting to get there.

In practice, I don’t care much for “Agile” software development practices, which often become the opposite. If you have mutual trust between managers, software engineers, and customers, then it can work well, but it’s not needed. If there isn’t that trust, Agile breaks down horribly and becomes not only a justification for intense micromanagement that borders on emotional bullying (several status checks per day) but with rigidity in such micromanagement that generates undesirable process complexity. Good teams are already doing things that look like Agile without necessarily calling that: keeping each other informed, taking full ownership of quality issues, and prioritizing important issues over silly and egotistical ones, without going nuts if (oh, my God!) something goes into version control without an associated “story”.

Yet I decided to read into “the dark side” and found a good talk about Agile by Mike Cohn, and I got a sense of what Agile really is and why it exists. Cohn isn’t trying to give evil managers a cudgel or mire teams in 45-minute “standups” where managers get to sit down. He’s trying to fix the Brownian Time problem. Between the burndown charts and time-management protocols, he’s trying to create a framework in which a team’s use of time show linear productivity rather than a square-root relationship. That, I’d say, is admirable.

Related to “Brownian Time”, of course, is Brownian Management. Requirements accumulate with no discrimination regarding which are the real requirements and which are “nice-to-haves” and which are fourth quadrant bullshit laid on because it’s free to tell someone what to do. Managers squabble over headcount and people are moved. Authority topologies change and expectations (always poorly laid out) shift. People are pushed left, than right. Ninety percent of work exists to counteract side effects of other work. From a microscopic level, people seem busy, but from a macroscopic perspective, nothing’s getting done. From an outside perspective, it looks infuriating. Ten thousand people are being paid to accomplish what looks like it should be done by 100. In other words, Square-Root Syndrome seems to apply to groups of people just as it does to time.

Progressive Time

If all the value in a creative person’s career could be condensed into 252 hours, that would be quite an unhappy conclusion. An incredible amount of time would be wasted. If we’re going to graph the per-hour economic yield of a typical person’s career (which rarely tells the whole story, because there are interactions between one hour and the next) we’ll probably find something like that. Nassim Taleb made almost all (over 98%, if I recall correctly, of his $40 million lifetime P&L) of his lifetime earnings as a trader in one day: the October 1987 stock market crash (“black swan”). This is why narratives work for us: they capture the small number of high-impact moments; Rocky Music plays in a “practice montage” of three minutes that stands in for 3 years of intense training. Sure, one can put the most critical parts of a drama (unfolding over five years) in a 3-hour movie, and that tells the whole story. Yet we know, from experience, that you can’t get to any level of readiness for the critical moments with a measly 3 hours of preparation. It takes deliberate practice. It requires progress, and that involves making sure future hours are informed by the past and present, so the right decisions are made, and the best ideas are had, in those 252 critical hours.

One neat thing about learning as opposed to economic “doing” is that it scales better. You don’t get Square Root Syndrome with building up a knowledge base. In fact, you probably get synergy: faster-than-linear scaling of economic value. However, economic value itself is not what I intend to measure. Here’s I’m just talking about productivity: the pushing forward of a project (which might be to learn a new concept). With well-structured learning processes, people continue to push forward at an approximately linear rate, rather than experiencing the Square Root Syndrome of Brownian Time.

Most individual productivity strategies (such as the Pomodoro Technique) are designed to bring peoples’ awareness and planning up to a level where linear Progressive Time is common, rather than square-root Brownian Time. The idea behind Agile, executed well, is the same: to put enough microscopic consciousness of time into the process to remove the drift that causes Brownian Time. If a team is getting bogged down with Brownian Management, escalating technical debt, or other scaling problems, it should show up on the burndown chart.

I still don’t like Agile, because it’s built on fundamental closed-allocation assumptions. I dislike the idea of having a totalitarian Product Owner with unilateral priority-setting authorities, on the assumption that engineers will “just go do it”. Totalitarian “get it done” management is appropriate for existential threats, but those are rare and shouldn’t be assumed in normal planning. It would also be better if engineers were empowered to push for Progressive Time on their own terms (self-executivity). I think there are some good ideas in “Agile” that deserve further inspection, but I wouldn’t buy the thing wholesale, and I’ve seen it become a disaster in practice.

Square-Root Syndrome and Hierarchy’s Role

I’ve already stated my hypothesis that something like a Square-Root Syndrome applies to people. If there are 100 people in an organization, then it’s probably doing the work of 10 people. Again, I don’t know that 0.5 is the exact right power to apply, but it’s not a far-off guess for a start. I’ll get back to that.

Why is “bigness” maladaptive? Why aren’t biological cells 20 meters in diameter? The answer is simple. At that size, it will starve. Surface area grows quadratically in the diameter of the cell, while mass and need for nourishment grow as the cube. In other words, a cell’s ability to nourish itself grows as the 0.6667th power of the size. The same seems to hold with organizations, although we’re no longer talking about a 3-dimensional physical space, but an N-dimensional abstract space– ideas, information, social connections, business strategies. I’m keeping this hand-wavy intentionally, but let’s focus on the N (it works as metaphor, at least) and talk about dimensionality.

Let’s say that we’ve hired four people whose fluencies (0 to 10) in various programming languages are as follows:

Person | Java | Python | Haskell |  C  |
-------+------+--------+---------+-----+
Alan   |    6 |      2 |       0 |  7  |
Barb   |    7 |      6 |       0 |  3  |
Carl   |    5 |      5 |       5 |  4  |
Diana  |    0 |      7 |      10 |  5  |
----------------------------------------


Who is the best programmer? Clearly, there’s no good way to answer that. Alan is the best at C, Barb is the best at Java, and Diana is the best at Haskell and Python. What about Carl? He’s not especially strong in any of the languages, but if there’s a project that requires Java and Haskell, he’s the only one who is ready (non-zero fluency) to do it! At 4 dimensions, we already have a world in which there’s no well-defined concept of the “best” or “worst” of these four programmers.

Dimensionality is relevant to organizations because, even though organizational dimensionality isn’t well-defined (there isn’t a clear set of “4 meaningful dimensions” that exist platonically, because what dimensions are relevant is somewhat subjective) it pertains to the optimal size of an organization. The more dimensionality there is in a business problem, the more it favors larger organizations with more specialized players. At least as metaphor, the idea of a cell in N-dimensional space works. Capacity for nourishment grows (in size p) as p^(N-1), and need for it grows as p^N, so overall organizational productivity grows as p^(N-1)/N and per-person productivity evolves in proportion to p^(-1/N)– it decreases.

What is the appropriate N for a typical corporation? Surprisingly, it’s disappointingly low. Businesses need a lot of different skill sets to operate, so one might expect this to make the case for high underlying dimensionality. If there are 10 dimensions on which people are evaluated for fit, then we get N = 10 and we scale as p^0.9, meaning we only get 7% more inefficient for each doubling in size. However, let’s consider two things. First, the proper value for N might not be an integer; it could be something like 2.35. This is a “fuzzy logic” situation where it’s subjective which dimensions matter, and how much. (This is an abstract fractal space, not a clean geometric one.) Does it matter if Carol speaks German (a candidate 5th dimension)? It depends on what the company is doing and what it needs. So the matter of which dimensions are included and excluded (a social phenomenon, not an explicit mathematical one) is unclear and could be akin to dimensions “possibly mattering, but intermittently and not all that much”. The effective N is much lower than the number of actual candidate dimensions (which is, at least, in the hundreds and arguably infinite). Second, organizational decision making is executed by humans, who can’t even visualize more than 2 dimensions easily. Three is possible, but a stretch. Four is just way outside of our experience. People making important “big company” decisions are not going to take stock of all the possible candidate dimensions. Everything gets collapsed into 2 dimensions: vertical (social status, importance, proximity to decision makers) and horizontal or “lateral” (all that other crap). Then, N = 2, and one gets exactly the square-root scaling. Since the “lateral” dimension is treated as inherently inferior (anything important would live in the vertical dimension) it might be more reasonable to treat the effective N as some lower value: 1.9? 1.85? 1.1? I won’t even begin to claim what the right number is, but it’s between 1 and 2 for most companies, and that induces something worse than square-root scaling.

If one finds this fractalized organizational pseudomathematics to be “hand wavy”, I’ll agree that it is, but there’s an important message in it. The more hierarchical and social-status driven an organization is (i.e. the lower the effective dimensionality, or the more social forces there are that collapse the organization into an org-chart or a “ladder”) the worse its capacity for nourishment (in this case, information rather than physical food) will fall behind its need. It will starve.

This is one of the inherent problems with big organizations. Their high underlying (“true”) dimensionality of needs requires size, but humans can only visualize two dimensions well as they work out their social plans, and this low effective dimensionality leads to information starvation, opacity, and inefficiency.

Management as a factor

The metaphor above discusses the biological cell, which does not scale to enormous size because of its surface-area-to-volume ratio would become too low to sustain it. Organizations have this issue as they get big: important players are on the surface, while most sit in the starving interior. This is made worse by the exertion of hierarchy, whose effect is to prioritize one point on the surface– the executive apex. (That’s where the low effective dimensionality, above, comes from.) How does this pathetic scaling relate to Brownian Management? It comes down to the MacLeod Clueless.

Losers sit away from the information surface area. They like the interior– it’s warm and comfortable and someone is closer to the outside than you in any direction– and avoid the edge. Clueless tend to be nearer to that edge, but are starved of important knowledge, or lack the competence to get it. Incidentally, they’re also the culprits in the bumbling, non-strategic, inconsistent direction of others’ time that becomes Brownian Management. They play a major role in the duplication of efforts, the go-nowhere projects, and overall waste of such a large amount of time. They get the information handed to them, which is rarely what they’d need to be properly strategic, and if the Sociopaths at the surface are engaged in zero-sum squabbling, they’ll cancel each other out. Why don’t Losers, who tend to be more strategic, fight back against the waste of Brownian time? The answer is that it won’t get them anything. They’re more likely to get fired than noticed in a good way, so they keep their heads down and implement ideas they know to be bad. Only when the ill-conceived project starts demanding above-board personal sacrifice (i.e. it becomes a “death march”) do they push back, and usually by leaving.

Solving It

Understanding of organizational efficiency usually comes down to discussions of percentages. “I was only at half speed today.” That’s not the right way to understand this particular problem. First, there’s the matter of faster-than-linear returns on performance (convexity). Even without that, though, we see that often organizational inefficiency isn’t some percentage cut. That would be tolerable. Eighty percent efficiency, meaning 20 percent is dropped on the floor? That’s a cost of doing business. Square-root scaling, however, means that efficiency goes to zero with growth. You might start out at an acceptable 80% efficiency, but find yourself at 8% when you scale up by an order of magnitude.

Preventing personnel congestion is a matter of conservative hiring. Only hire multipliers who will make the whole group more productive. It’s not that mere adders should be considered unacceptable. For commodity labor, that’s perfectly fine. However, if the work is a commodity, you can specify it contractually and hire it on the market. Why bring a new person on board (and increase communication complexity) for that? I’m loathe to use the word synergy because it has become such an MBA buzzword, but that’s exactly what I’m talking about.

I believe that a company that grows conservatively can avoid Square-Root Syndrome in its people. Communication topologies and political complexities will get more complicated, but that can be offset by sharing of ideas and collaboration. So long as growth is slow enough to remain strategic and cooperative, it’s a good thing and will probably improve per-person efficiency. The problem that VC-istan companies seem to inflict on themselves is that they grow so fast that internal competition (for larger equity shares, executive roles) emerges and the whole thing implodes.

However, if you hire for synergy and avoid the Square-Root Syndrome of rapid expansion and turnover, you get to a point where the second trust problem (investors’ ability to trust in the organization to do the work) solves itself. Hire great software engineers and give them just enough direction to outline the problem, and just enough incentive (profit sharing, not equity in some far-off liquidation that might involve horrible investor preferences that wiping out common stock) to care about the profit motive, and they’ll get their work done.

Thus, most important on a day-to-day level is avoiding Square-Root Syndrome in time: getting employees to work in Progressive Time. That doesn’t mean that every idea has to come to fruition or that failure won’t be tolerated. Instead, it’s the opposite. It’s okay to fail so long as you can affirmatively answer the question: did you learn something? The difference between Brownian and Progressive Time is that the latter has a memory. The first is bumbling blindly and retracing worn paths, usually under (MacLeod Clueless) managerial dictation. The second is exploration that builds a knowledge base and enables future explorations to be more successful.

VC-istan, by the way, lives in Brownian Time. Now that M&A has replaced R&D, institutional knowledge of failures just dissipates, resulting in massive duplications of effort that swell up every few years. There is progress, but it’s at the Brownian drift rate (with selection imposing macroscopic forward movement; in other words, mindless evolution) rather than anything that could legitimately be considered deliberate forward progress.

What’s the practical way to do all of this? How does one inject these principles into a software company?

1. Self-Executivity in Progressive Time. Personnel reviews aren’t about “How loyal were you to your boss’s career goals this year?” No, they’re about: did you work in Progressive Time? What did you learn? What did you teach? What multiplier effects did you have that made the whole company better? Why is this a better place to work in 2013 than it was in 2012? Why are you better in 2013 than in 2012?
   • By the way, I fucking hate the term performance review. If I were running a company, there’d be no such thing. There’d be regular impact reviews. You’re assumed to be performing. You’re trusted (and those who prove unworthy of trust are packaged out) to be working hard and in good faith. The impact meeting is to discuss unintended effects (that he or she might not see) of a person’s work and behavior on the company. Very low (or negative) impact doesn’t mean you’re a horrible person who deserves to be humiliated; it’s assumed to be no-fault but means that you need to do things differently.
2. “10-Year Fit” / Invest in Employees. I will confess that I’m somewhat of a “job hopper“, and I’m shameless about it. Most companies (and even many managers in the more progressive firms) don’t invest in their peoples’ careers and don’t deserve loyalty. Progressive Time is not compatible with a head-down-and-following-orders attitude toward work. However, I am personally tiring out of the job-hopping lifestyle. So instead of the typical corporation where one has to be a lucky protege to have a real career, I’d build a company around the concept of the 10-year fit, and aim to invest in employees and get progressive returns amid convexity. Fuck aiming for 10 years, let’s make it 50. You can leave and I’ll make sure that you have a great title and reference, but my job is to make things so great that you never want that option.
3. Agile that Doesn’t Suck. The good thing about Agile is that it exists to coerce time into a linear, progressive march rather than the haphazard, Brownian stumbling of managed work when it isn’t monitored. The problem with it is that it involves closed-allocation assumptions and limitations of self-organization. Perhaps Agile could be adapted to an open-allocation world, however. That deserves a lot more investigation.
4. Three-hour Workday. Employees are expected to work full-time in spirit, not in hours. Project plans will be based on 3 dedicated hours: that’s three hours of “metered work”, certainly not inlcuding goofing off and eating and water-pooler chat. Three is intended as the minimum obligation; of course, no one will be tracking and a person who delivers the typical corporate workday (9 hours at 33% efficiency) is in good standing instead of three solid hours. Three hours is also a right: 180 minutes of uninterrupted “building time” per day during daylight hours without meetings or those god-awful impromptu status pings. Employees should ideally be spending the other 4-7 “off-meter” hours each day to learn new skills (Progressive Time) or experiment or use Coursera or share ideas with each other.
   • In practice, few people would be able to get away with a strict 3-hour day. It’s somewhat of a planning fiction that accounts for the difficulty of estimation and the extreme long-term importance of off-meter time. In this model firm that I’m building up, I can’t see anyone working less than 6 hours per day and fulfilling the softer, off-meter obligations such as continuing education, and the average would be the standard 8.
5. Culture of Mentoring. The most junior engineers are expected to use their off-meter hours to learn from more senior people. The most highly-compensated people, if they wish to remain so, are expected to share knowledge and multiply their expertise across the company. This expectation of mentoring (for senior hires) and progress (for junior hires) would be the only interference with self-executive culture. If we are to stay self-executive, we must be competitive in the market place; to be competitive, we must be progressive in the growth of internal skill.
6. Well-defined Audit Cycle. With the overall goal being to have each employee in Progressive Time, there’d need to be some sort of incremental work monitoring. As much as “status” meetings are disliked, there’d need to be an understanding of how often a person or team is expected to ship or demo something. Demos would invite the whole company (not one manager). I think I’d have junior hires on a 3-week audit cycle (in which, “here’s what I’ve learned” is perfectly acceptable) and senior engineers expected to demo once every 8 weeks (as a minimum; I’d encourage the same 3-week cycle). The most senior, fellow-level, engineers wouldn’t have an audit cycle; since they’d be expected to be continuously multiplying their expertise across the company and mentoring new people, such a thing would be irrelevant.

So, that solves the second trust problem: how does one ensure people get their work done? You need the right structure. It’s not about Agile or gamification or anything out of management books, and self-executivity is a necessary but not sufficient condition. You also need to put everyone in linear (Progressive) rather than square-root (Brownian) time. You need to make that a cultural pillar: not working hard, but working mindfully.

Programmer autonomy is a $1 trillion issue.

Sometimes, I think it’s useful to focus on trillion-dollar problems. While it’s extremely difficult to capture $1 trillion worth of value, as made evident by the nonexistence of trillion-dollar companies, innovations that create that much are not uncommon, and these trillion-dollar problems often have not one good idea, but a multitude of them to explore. So a thought experiment that I sometimes run is, “What is the easiest way to generate $1 trillion in value?” Where is the lowest-hanging trillion dollars of fruit? What’s the simplest, most obvious thing that can be done to add $1 trillion of value to the economy?

I’ve written at length about the obsolescence of traditional management, due to the convex input-output relationship in modern, creative work, making variance-reductive management counter-productive. I’ve also shared a number of thoughts about Valve’s policy of open allocation, and the need for software engineers to demand the respect accorded to traditional professions: sovereignty, autonomy, and the right to work directly for the profession without risk of the indirection imposed by traditional hierarchical management. So, for this exercise, I’m going to focus on the autonomy of software engineers. How much improvement will it take to add $1 trillion in value to the economy?

This is an open-ended question, so I make it more specific: how many software engineers would we have to give a high degree (open allocation) of autonomy in order to add $1 trillion? Obviously, the answer is going to depend on the assumptions that are made, so I’ll have to answer some basic questions. Because there’s uncertainty in all of these numbers, the conclusion should be treated as an estimate. However, when possible, I’ve attempted to make my assumptions as conservative as possible. Therefore, it’s quite plausible that the required number of engineers is actually less than half the number that I’ll compute here.

Question 1: How many software engineers are there? I’m going to restrict this analysis to the developed world, because it’s hard to enforce cultural changes globally. Data varies, but most surveys indicate that there are about 1.5 million software engineers in the U.S. If we further assume the U.S. to be one-fifth of “the developed world”, we get 7.5 million full-time software engineers in the world. This number could probably be increased by a factor of two by loosening the definition of “software engineer” or “developed world”, but I think it’s a good working number. The total number of software engineers is probably two to three times that.

Question 2: What is the distribution of talent among software engineers? First, here’s the scale that I developed for assessing the skill and technical maturity of a software engineer. Defining specific talent percentiles requires specifying a population, but I think the software engineer community can be decomposed into the following clusters, differing according to the managerial and cultural influences on their ability and incentive to gain competence.

Cluster A: Managed full-time (75%). These are full-time software engineers who, at least nominally, spend their working time either coding or on software-related problems (such as site reliability). If they were asked to define their jobs, they’d call themselves programmers or system administrators: not meteorologists or actuaries who happen to program software. Engineers in this cluster typically work on large corporate codebases and are placed in a typical hierarchical corporate structure. They develop an expertise within their defined job scope, but often learn very little outside of it. Excellence and creativity generally aren’t rewarded in their world, and they tend to evolve into the stereotypical “5:01 developers”. They tend to plateau around 1.2-1.3, because the more talented ones are usually tapped for management before they would reach 1.5. Multiplier-level contributions for engineers tend to be impossible to achieve in their world, due to bureaucracy, limited environments, project requirements developed by non-engineers, and an assumption that anyone decent will become a manager. I’ve assigned Cluster A a mean competence of 1.10 and a standard deviation of 0.25, meaning that 95 percent of them are between 0.6 and 1.6.

Cluster B: Novices and part-timers (15%). These are the non-engineers who write scripts occasionally, software engineers in their first few months, interns and students. They program sometimes, but they generally aren’t defined as programmers. This cluster I’ve given a mean competence of 0.80 and a standard deviation of 0.25. I assign them to a separate cluster because (a) they generally aren’t evaluated or managed as programmers, and (b) they rarely spend enough time with software to become highly competent. They’re good at other things.

Cluster B isn’t especially relevant to my analysis, and it’s also the least well-defined. Like the Earth’s atmosphere, its outer perimeter has no well-defined boundary. Uncertainty about its size is also the main reason why it’s hard to answer questions about the “number of programmers” in the world. The percentage would increase (and so would the number of programmers) if the definition of “programmer” were loosened.

Cluster C: Self-managing engineers (10%). These are the engineers who either work in conditions of unusual autonomy (being successful freelancers, company owners, or employees of open-allocation companies) or who exert unusual efforts to control their careers, education and progress. This cluster has a larger mean competence and variance than the others. I’ve assigned it a mean of 1.5 and a variance of 0.35. Consequently, almost all of the engineers who get above 2.0 are in this cluster, and this is far from surprising: 2.0-level (multiplier) work is very rare, and impossible to get under typical management.

If we mix these three distributions together, we get the following profile for the software engineering world:

Skill Percentile
1.0     38.37
1.2     65.38
1.4     85.24
1.6     94.47
1.8     97.87
2.0     99.23
2.2     99.78
2.4     99.95
2.6     99.99


How accurate is this distribution? Looking at it, I think it probably underestimates the percentage of engineers in the tail. I’m around 1.7-1.8 and I don’t think of myself as a 97th-percentile programmer (probably 94-95th). It also says that only 0.77 percent of engineers are 2.0 or higher (I’d estimate it at 1 to 2 percent). I would be inclined to give Cluster C an exponential tail on the right, rather than the quadratic-exponential decay of a Gaussian, but for the purpose of this analysis, I think the Gaussian model is good enough.

Question 3: How much value does a software engineer produce? Marginal contribution isn’t a good measure of “business value”, because it’s too widely variable based on specifics (such company size) but I wouldn’t say that employment market value is a good estimate either, because there’s a surplus of good people who (a) don’t know what they’re actually worth, and (b) are therefore willing to work for low compensation, especially because a steady salary removes variance from compensation. Companies know that they make more (a lot more) on the most talented engineers than on average ones– if they have high-level work for them. The better engineer might be worth ten times as much, but only cost twice as much, as the average. So I think of it in terms of abstract business value: given appropriate work, how much expected value (ignoring variance) can that person deliver?

This quantity I’ve assumed to be exponential with regard to engineer skill, as described above: an increment of 1.0 is a 10-fold increase in abstract business value (ABV). Is this the right multiplier? It indicates that an increment of 0.3 is a doubling of ABV, or that a 0.1-increment is a 26 percent increase in ABV. In my experience, this is about right. For some skill-intensive projects, such as technology-intensive startups, the gradient might be steeper (a 20-fold difference between 2.0 and 1.0, rather than 10) but I am ignoring that, for simplicity’s sake.

The ABV of a 1.0 software engineer I’ve estimated at $125,000 per year in a typical business environment, meaning that it would be break-even (considering payroll costs, office space, and communication overhead) to hire one at a salary around $75,000. A 95th-percentile engineer (1.63) produces an ABV of $533,000 and a 99th-percentile engineer (1.96) produces an ABV of $1.14 million. These numbers are not unreasonable at all. (In fact, if they’re wrong, I’d bet on them being low.)

This doesn’t, I should say, mean that a 99th-percentile engineer will reliably produce $1.14 million per year, every year. She has to be assigned at-level, appropriate, work. Additionally, that’s an expected return (and the variance is quite high at the upper end). She might produce $4 million in one year under one set of conditions, and zero in another arena. Since I’m interested in the macroeconomic effect of increasing engineer autonomy, I can ignore variance and focus on mean expectancy alone. This sort of variance is meaningful to the individual (it’s better to have a good year than a bad one) and small companies, but the noise cancels itself out on the macroeconomic scale.

Putting it all together: with these estimates of the distribution of engineer competence, and the ABV estimates above, it’s possible to compute an expected value for a randomly-chosen engineer in each cluster:

Cluster A: $185,469 per year.

Cluster B: $89,645 per year.

Cluster C: $546,879 per year.

All software engineers: $207,236 per year.

Software engineers can evolve in all sorts of ways, and improved education or more longevity might change the distributions of the clusters themselves. That I’m not going to model, because there are so many possibilities. Nor am I going to speculate on macroeconomic business changes that would change the ABV figures. I’m going to focus on only one aspect of economic change, although there are others with additional positive effects, and that change is the evolution of an engineer from the micromanaged world of Cluster A to the autonomous, highly-productive one of Cluster C. (Upward movement within clusters is also relevant, and that strengthens my case, but I’m excluding it for now.) I’m going to assume that it takes 5 years of education and training for that evolution to occur, and assume an engineer’s career (with some underestimation here, justified by time-value of money) lasts 20 years, meaning there are 15 years to reap the benefits in full, plus two years to account for partial improvement during that five-year evolution.

What this means is that, for each engineer who evolves in this way, it generates $361,410 per year in value for 17 years, or $6.14 million per engineer. That is the objective benefit that accrues to society in engineer skill growth alone when a software engineer moves out of a typical subordinate context and into one like Valve’s open allocation regime. Generating $1 trillion in this way requires liberating 163,000 engineers, or a mere 2.2% of the total pool. That happens even if (a) the number of software engineers remains the same (instead of increasing due to improvements to the industry) and (b) other forms of technological growth, that would increase the ABV of a good engineer, stop, although it’s extremely unlikely that they will. Also, there are the ripple effects (in terms of advancing the state of the art in software engineering) of a world with more autonomous and, thus, more skilled engineers. All of these are substantial, and they improve my case even further, but I’m removing those concerns in the interest of pursuing a lower bound for value creation. What I can say, with ironclad confidence, is that the movement of 163,000 engineers into an open-allocation regime will, by improving their skills and, over the years, their output, generate $1 trillion at minimum. That it might produce $5 or $20 trillion (or much more, in terms of long-term effects on economic growth) in eventual value, through multiplicative “ripple” effects, is more speculatory. My job here was just to get to $1 trillion.

In simpler terms, the technological economy into which our society needs to graduate is one that requires software (not to mention mathematical and scientific) talent at a level that would currently be considered elite, and the only conditions that allow such levels of talent to exist are those of extremely high autonomy, as observed at Valve and pre-apocalyptic Google. Letting programmers direct their own work, and therefore take responsibility for their own skill growth is, multiplied over the vast number of people in the world who have to work with software, easily a trillion-dollar problem. It’s worth addressing now.

XWP vs. JAP

The software industry is a fascinating place. As programmers, we have the best and worst job in the world. What we do is so rewarding and challenging that many of us have been doing it for free since we were eight. We’re paid to think, and to put pure logic into systems that effectively do our bidding. And yet, we have (as a group) extremely low job satisfaction. We don’t last long: our half-life is about six years. By 30, most of us have decided that we want to do something else: management, quantitative finance, or startup entrepreneurship. It’s not programming itself that drives us out, but the ways in which the “programmer” job has been restructured out of our favor. It’s been shaved down, mashed, melted and molded into commodity grunt work, except for the top 2 percent or so of our field (for whom as much time is spent establishing that one is, in fact, in the top 2 percent, as is spent working). Most of us have to sit in endless meetings, follow orders that make no sense, and maintain legacy code with profanity such as “VisitorFactoryFactory” littered about, until we move “into management”, often landing in a role that is just as tedious but carries (slightly) more respect.

I’m reaching a conclusion, and it’s not a pleasant one, about our industry and what one has to do to survive it. My definition of “survive” entails progress, because while it’s relatively easy to coast, engineers who plateau are just waiting to get laid off, and will usually find that demand for their (increasingly out of date) skills has declined. Plainly put, there’s a decision that programmers have to make if they want to get better. Why? Because you only get better if you get good projects, and you only get good projects if you know how to play the game. Last winter, I examined the trajectory of software engineers, and why it seems to flat-line so early. The conclusion I’ve come to is that there are several ceilings, three of which seem visible and obvious, and each requires a certain knack to get past it. Around 0.7 to 0.8 there’s the “weed out” effect that’s  rooted in intellectual limitations: inability to grasp pointers, recursion, data in sets, or other intellectual concepts people need to understand if they’re going to be adequate programmers. Most people who hit this ceiling do so in school, and one hopes they don’t become programmers. The next ceiling, which is where the archetypical “5:01″ mediocrities live, is around 1.2. This is where you finish up if you just follow orders, don’t care about “functional programming” because you can’t see how it could possibly apply to your job, and generally avoid programming outside of an office context.

The next ceiling is around 1.8, and it’s what I intend to discuss. The 0.7 ceiling is a problem of inability, and at 1.2 it’s an issue of desire and willingness. There are a lot of programmers who don’t have a strong desire to get any better than average. Average is employable, middle-class, comfortable. That keeps a lot of people complacent around 1.2. The ceiling at 1.8, on the other hand, comes from the fact that it’s genuinely hard to get allocated 1.8+ level work, which usually involves technical and architectural leadership. In most companies, there are political battles that the projects’ originators must fight to get them on the map, and others that engineers must involve themselves in if they want to get on to the best projects. It’s messy and hard and ugly and it’s the kind of process that most engineers hate.

Many engineers at the 1.7 to 1.8 level give up on engineering progress and take this ceiling as a call to move into management. It’s a lot harder to ensure a stream of genuinely interesting work than it is to take a middle management position. The dream is that the managerial position will allow the engineer to allocate the best technical work to himself and delegate the crap. The reality is that he’s lucky if he gets 10 hours per week of coding time in, and that managers who cherry-pick the good work and leave the rest to their subordinates are often despised and therefore ineffective.

This said, there’s an idea here, and it deserves attention. The sudden desire to move into management occurs when engineers realize that they won’t progress by just doing their assigned work, and that they need to hack the project allocation process if they want to keep getting better. Managerial authority seems like the most direct route to this because, after all, it’s managers who assign the projects. The problem with that approach is that managerial work requires an entirely different skill set, and that while this is a valid career, it’s probably not what one should pursue if one wants to get better as a software engineer.

How does one hack project allocation? I’m going to introduce a couple terms. The first is J.A.P.: “Just A Programmer”. There are a lot of people in business who see programming as commodity work: that’s why most of our jobs suck. This is a self-perpetuating cycle: because of such peoples’ attitudes toward programmers, good engineers leave them, leaving them with the bad, and reinforcing their perception that programming is order-following grunt work that needs to be micromanaged or it won’t be done right at all. Their attitude toward the software engineer is that she’s “just a programmer”. Hence the term. There’s a related cliche in the startup world involving MBA-toting “big-picture guys” who “just need a programmer” to do all the technical work in exchange for a tiny sliver of the equity. What they get, in return, is rarely quality.

Worse yet for the business side, commodity programmers aren’t 90 or 70 or 50 percent as valuable as good engineers, but 5 to 10 percent as useful, if that. The major reason for this is that software projects scale horribly in terms of the number of people involved with them. A mediocre engineer might be 20 percent as effective, measured individually, as a good one, but four mediocre engineers will only be about 35 percent (not 100) as effective as a single good engineer.

Good programmers dread the “Just A Programmer” role in which they’re assessed on the quantity of code they crank out rather than the problems they solve and the quality of their solutions, and they avoid such positions especially because commodity-programmer roles tend to attract ineffective programmers, and effective people who have to work with ineffective programmers become, themselves, ineffective.

This said, a 1.8 engineer is not a “commodity programmer”. At this level, we’re talking about people who are probably in the top 2 or 3 percent of the software industry. We’re talking about people who, in a functioning environment, will deliver high-quality and far-reaching software solutions reliably. They can start from scratch and deliver an excellent “full-stack” solution. (In a dysfunctional environment, they’ll probably fail if they don’t quit first.)  The political difficulty, and it can be extreme, lies with the fact that it’s very difficult for a good engineer to reliably establish (especially to non-technical managers, and to those managers’ favorites who may not be technically strong) that she is good. It turns out that, even if it’s true, you can’t say to your boss, “I’m a top-2% engineer and deserve more autonomy and the best projects” and expect good results. You have to show it, but you can’t show it unless you get good projects.

What this means, in fewer words, is that it’s very difficult for a software engineer to prove he’s not a commodity programmer without hacking the politics. Perversely, many software environments can get into a state where engineering skill becomes negatively correlated with political success. For example, if the coding practices are “best practices”, “design pattern”-ridden Java culture, with FactoryVisitorSingletonSelection patterns all over the place, bad engineers have an advantage on account of being more familiar with damaged software environments, and because singleton directories called “com” don’t piss them off as much (since they never venture outside of an IDE anyway).

Software wants to be a meritocracy, but the sad reality is that effectiveness of an individual programmer depends on the environment. Drop a 1.8+ engineer into a Visitor-infested Java codebase and he turns into a bumbling idiot, in the same way that an incompetent player at a poker table can fluster experts (who may not be familiar with that particular flavor of incompetence). The result of this is that detecting who the good programmers are, especially for a non-programmer or an inept one, is extremely difficult, if not impossible. The 1.7 to 1.8 level is where software engineers realize that, in spite of their skill, they won’t be recognized as having it unless they can ensure a favorable environment and project allocation, and that it’s next to impossible to guarantee these benefits in the very long run without some kind of political advantage. Credibility as a software engineer alone won’t cut it, because you can’t establish that creativity unless you get good projects.

Enter the “X.W.P.” distinction, which is the alternative to being a “J.A.P.” It means an “X Who Programs”, where X might be an entrepreneur, a researcher, a data scientist, a security specialist, a quant, or possibly even a manager. If you’re an XWP, you can program, and quite possibly full-time, but you have an additional credibility that is rooted in something other than software engineering. Your work clearly isn’t commodity work; you might have a boss, but he doesn’t believe he could do your job better than you can. XWP is the way out. But you also get to code, so it’s the best of both worlds.

This might seem perverse and unusual. At 1.8, the best way to continue improving as a software engineer is not to study software engineering. You might feel like there’s still a lot to learn in that department, and you’re right, but loading up on unrecognized skill is not going to get you anywhere. It leads to bitterness and slow decline. You need something else.

One might think that an XWP is likely to grow as an X but not as a software engineer, but I don’t think that’s necessarily true. There certainly are quants and data scientists and entrepreneurs and game designers who remain mediocre programmers, but they don’t have to. If they want to become good engineers, they have an advantage over vanilla software engineers on account of the enhanced respect accorded their role. If a Chief Data Scientist decides that building a distributed system is the best way to solve a machine learning problem, and he’s willing to roll his sleeves up and write the code, the respect that this gives him will allow him to take the most interesting engineering work. This is how you get 1.8 and 2.1 and 2.4-level engineering work. You start to bill yourself as something other than a software engineer and get the respect that entitles you to projects that will make you better. You find an X and become an X, but you also know your way around a computer. You’re an X, and you know how to code, and your “secret weapon” (secret because management in most companies won’t recognize it) is that you’re really good at it, too.

This, perhaps, is the biggest surprise I’ve encountered in the bizarre world that is the software engineering career. I am so much without words that I’ll use someone else’s, from A Song of Ice and Fire: To go west, you must first go east.

What is spaghetti code?

One of the easiest ways for an epithet to lose its value is for it to become over-broad, which causes it to mean little more than “I don’t like this”. Case in point is the term, “spaghetti code”, which people often use interchangeably with “bad code”. The problem is that not all bad code is spaghetti code. Spaghetti code is an especially virulent but specific kind of bad code, and its particular badness is instructive in how we develop software. Why? Because individual people rarely write spaghetti code on their own. Rather, certain styles of development process make it increasingly common as time passes. In order to assess this, it’s important first to address the original context in which “spaghetti code” was defined: the dreaded (and mostly archaic) goto statement.

The goto statement is a simple and powerful control flow mechanism: jump to another point in the code. It’s what a compiled assembly program actually does in order to transfer control, even if the source code is written using more modern structures like loops and functions. Using goto, one can implement whatever control flows one needs. We also generally agree, in 2012, that goto is flat-out inappropriate for source code in most modern programs. Exceptions to this policy exist, but they’re extremely rare. Most modern languages don’t even have it.

Goto statements can make it difficult to reason about code, because if control can bounce about a program, one cannot make guarantees about what state a program is in when it executes a specific piece of code. Goto-based programs can’t easily be broken down into component pieces, because any point in the code can be wormholed to any other. Instead, they devolve into an “everything is everywhere” mess where to understand a piece of the program requires understanding all of it, and the latter becomes flat-out impossible for large programs. Hence the comparison to spaghetti, where following one thread (or noodle) often involves navigating through a large tangle of pasta. You can’t look at a bowl of noodles and see which end connects to which. You’d have to laboriously untangle it.

Spaghetti code is code where “everything is everywhere”, and in which answering simple questions such as (a) where a certain piece of functionality is implemented, (b) determining where an object is instantiated and how to create it, and (c) assessing a critical section for correctness, just to name a few examples of questions one might want to ask about code, require understanding the whole program, because of the relentless pinging about the source code that answer simple questions requires. It’s code that is incomprehensible unless one has the discipline to follow each noodle through from one side to the other. That is spaghetti code.

What makes spaghetti code dangerous is that it, unlike other species of bad code, seems to be a common byproduct of software entropy. If code is properly modular but some modules are of low quality, people will fix the bad components if those are important to them. Bad or failed or buggy or slow implementations can be replaced with correct ones while using the same interface. It’s also, frankly, just much easier to define correctness (which one must do in order to have a firm sense of what “a bug” is) over small, independent functions than over a giant codeball designed to do too much stuff. Spaghetti code is evil because (a) it’s a very common subcase of bad code, (b) it’s almost impossible to fix without causing changes in functionality, which will be treated as breakage if people depend on the old behavior (potentially by abusing “sleep” methods, thus letting a performance improvement cause seemingly unrelated bugs!) and (c) it seems, for reasons I’ll get to later, not to be preventable through typical review processes.

The reason I consider it important to differentiate spaghetti code from the superset, “bad code”, is that I think a lot of what makes “bad code” is subjective. A lot of the conflict and flat-out incivility in software collaboration (or the lack thereof) seems to result from the predominantly male tendency to lash out in the face of unskilled creativity (or a perception of such, and in code this is often an extremely biased perception): to beat the pretender to alpha status so badly that he stops pestering us with his incompetent displays. The problem with this behavior pattern is that, well, it’s not useful and it rarely makes people better at what they’re trying to do. It’s just being a prick. There are also a lot of anal-retentive wankbaskets out there who define good and bad programmers based on cosmetic traits so that their definition of “good code” is “code that looks like I wrote it”. I feel like the spaghetti code problem is better-defined in scope than the larger but more subjective problem of “bad code”. We’ll never agree on tabs-versus-spaces, but we all know that spaghetti code is incomprehensible and useless. Moreover, as spaghetti code is an especially common and damaging case of bad code, assessing causes and preventions for this subtype may be generalizable to other categories of bad code.

People usually use “bad code” to mean “ugly code”, but if it’s possible to determine why a piece of code is bad and ugly, and to figure out a plausible fix, it’s already better than most spaghetti code. Spaghetti code is incomprehensible and often unfixable. If you know why you hate a piece of code, it’s already above spaghetti code in quality, since the latter is just featureless gibberish.

What causes spaghetti code? Goto statements were the leading cause of spaghetti code at one time, but goto has fallen so far out of favor that it’s a non-concern. Now the culprit is something else entirely: the modern bastardization of object-oriented programming. Inheritance is an especially bad culprit, and so is premature abstraction: using a parameterized generic with only one use case in mind, or adding unnecessary parameters. I recognize that this claim– that OOP as practiced is spaghetti code– is not a viewpoint without controversy. Nor was it without controversy, at one time, that goto was considered harmful.

One of the biggest problems in comparative software (that is, the art of comparing approaches, techniques, languages, or platforms) is that most comparisons focus on simple examples. At 20 lines of code, almost nothing shows its evilness, unless it’s contrived to be dastardly. A 20-line program written with goto will usually be quite comprehensible, and might even be easier to reason about than the same program written without goto. At 20 lines, a step-by-step instruction list with some explicit control transfer is a very natural way to envision a program. For a static program (i.e. a platonic form that need never be changed and incurs no maintenance) that can be read in one sitting, that might be a fine way to structure it. At 20,000 lines, the goto-driven program becomes incomprehensible. At 20,000 lines, the goto-driven program has been hacked and expanded and tweaked so many times that the original vision holding the thing together has vanished, and the fact that a program can be in a piece of code “from anywhere” means that to safely modify the code requires confidence quantified by “from everywhere”. Everything is everywhere. Not only does this make the code difficult to comprehend, but it means that every modification to the code is likely to make it worse, due to unforeseeable chained consequences. Over time, the software becomes “biological”, by which I mean that it develops behaviors that no one intended but that other software components may depend on in hidden ways.

Goto failed, as a programming language construct, because of these problems imposed by the unrestricted pinging about a program that it created. Less powerful, but therefore more specifically targeted, structures such as procedures, functions, and well-defined data structures came into favor. For the one case where people needed global control flow transfer (error handling) exceptions were developed. This was a progress from the extreme universality and abstraction of a goto-driven program to the concretion and specificity of pieces (such as procedures) solving specific problems. In unstructured programming, you can write a Big Program that does all kinds of stuff, add features on a whim, and alter the flow of the thing as you wish. It doesn’t have to solve “a problem” (so pedestrian…) but it can be a meta-framework with an embedded interpreter! Structured programming encouraged people to factor their programs into specific pieces that solved single problems, and to make those solutions reusable when possible. It was a precursor of the Unix philosophy (do one thing and do it well) and functional programming (make it easy to define precise, mathematical semantics by eschewing global state).

Another thing I’ll say about goto is that it’s rarely needed as a language-level primitive. One could achieve the same effect using a while-loop, a “program counter” variable defined outside that loop that the loop either increments (step) or resets (goto) and a switch-case statement using it. This could, if one wished, be expanded into a giant program that runs as one such loop, but code like this is never written. What the fact that this is almost never done seems to indicate is that goto is rarely needed. Structured programming thereby points out the insanity of what one is doing when attempting severely non-local control flows.

Still, there was a time when abandoning goto was extremely controversial, and this structured programming idea seemed like faddish nonsense. The objection was: why use functions and procedures when goto is strictly more powerful?

Analogously, why use referentially transparent functions and immutable records when objects are strictly more powerful? An object, after all, can have a method called run or call or apply so it can be a function. It can also have static, constant fields only and be a record. But it can also do a lot more: it can have initializers and finalizers and open recursion and fifty methods if one so chooses. So what’s the fuss about this functional programming nonsense that expects people to build their programs out of things that are much less powerful, like records whose fields never change and whose classes contain no initialization magic?

The answer is that power is not always good. Power, in programming, often advantages the “writer” of code and not the reader, but maintenance (i.e. the need to read code) begins subjectively around 2000 lines or 6 weeks, and objectively once there is more than one developer on a project. On real systems, no one gets to be just a “writer” of code. We’re readers, of our own code and of that written by others. Unreadable code is just not acceptable, and only accepted because there is so much of it and because “best practices” object-oriented programming, as deployed at many software companies, seem to produce it. A more “powerful” abstraction is more general, and therefore less specific, and this means that it’s harder to determine exactly what it’s used for when one has to read the code using it. This is bad enough, but single-writer code usually remains fairly disciplined: the powerful abstraction might have 18 plausible uses, but only one of those is actually used. There’s a singular vision (although usually an undocumented one) that prevents the confusion. The danger sets in when others who are not aware of that vision have to modify the code. Often, their modifications are hacks that implicitly assume one of the other 17 use cases. This, naturally, leads to inconsistencies and those usually result in bugs. Unfortunately, people brought in to fix these bugs have even less clarity about the original vision behind the code, and their modifications are often equally hackish. Spot fixes may occur, but the overall quality of the code declines. This is the spaghettification process. No one ever sits down to write himself a bowl of spaghetti code. It happens through a gradual “stretching” process and there are almost always multiple developers responsible. In software, “slippery slopes” are real and the slippage can occur rapidly.

Object-oriented programming, originally designed to prevent spaghetti code, has become (through a “design pattern” ridden misunderstanding of it) one of the worst sources of it. An “object” can mix code and data freely and conform to any number of interfaces, while a class can be subclassed freely about the program. There’s a lot of power in object-oriented programming, and when used with discipline, it can be very effective. But most programmers don’t handle it well, and it seems to turn to spaghetti over time.

One of the problems with spaghetti code is that it forms incrementally, which makes it hard to catch in code review, because each change that leads to “spaghettification” seems, on balance, to be a net positive. The plus is that a change that a manager or customer “needs yesterday” gets in, and the drawback is what looks like a moderate amount of added complexity. Even in the Dark Ages of goto, no one ever sat down and said, “I’m going to write an incomprehensible program with 40 goto statements flowing into the same point.”  The clutter accumulated gradually, while the program’s ownership transferred from one person to another. The same is true of object-oriented spaghetti. There’s no specific point of transition from an original clean design to incomprehensible spaghetti. It happens over time as people abuse the power of object-oriented programming to push through hacks that would make no sense to them if they understood the program they were modifying and if more specific (again, less powerful) abstractions were used. Of course, this also means that fault for spaghettification is everywhere and nowhere at the same time: any individual developer can make a convincing case that his changes weren’t the ones that caused the source code to go to hell. This is part of why large-program software shops (as opposed to small-program Unix philosophy environments) tend to have such vicious politics: no one knows who’s actually at fault for anything.

Incremental code review is great at catching the obvious bad practices, like mixing tabs and spaces, bad variable naming practices, and lines that are too long. That’s why the more cosmetic aspects of “bad code” are less interesting (using a definition of “interesting” synonymous with “worrisome”) than spaghetti code. We already know how to solve them in incremental code review. We can even configure our continuous-integration servers to reject such code. As for spaghetti code, where there is no clear definition, this is difficult if not impossible to do. Whole-program review is necessary to catch that, but I’ve seen very few companies willing to invest the time and political will necessary to have actionable whole-program reviews. Over the long term (10+ years) I think it’s next to impossible, except among teams writing life- or mission-critical software, to ensure this high level of discipline in perpetuity.

The answer, I think, is that Big Code just doesn’t work. Dynamic typing falls down in large programs, but static typing fails in a different way. The same is true of object-oriented programming, imperative programming, and to a lesser but still noticeable degree (manifest in the increasing number of threaded state parameters) in functional programming. The problem with “goto” wasn’t that goto was inherently evil, so much as that it allowed code to become Big Code very quickly (i.e. the threshold of incomprehensible “bigness” grew smaller). On the other hand, the frigid-earth reality of Big Code is that there’s “no silver bullet”. Large programs just become incomprehensible. Complexity and bigness aren’t “sometimes undesirable”. They’re always dangerous. Steve Yegge got this one right.

This is why I believe the Unix philosophy is inherently right: programs shouldn’t be vague, squishy things that grow in scope over time and are never really finished. A program should do one thing and do it well. If it becomes large and unwieldy, it’s refactored into pieces: libraries and scripts and compiled executables and data. Ambitious software projects shouldn’t be structured as all-or-nothing single programs, because every programming paradigm and toolset breaks down horribly on those. Instead, such projects should be structured as systems and given the respect typically given to such. This means that attention is paid to fault-tolerance, interchangeability of parts, and communication protocols. It requires more discipline than the haphazard sprawl of big-program development, but it’s worth it. In addition to the obvious advantages inherent in cleaner, more usable code, another benefit is that people actually read code, rather than hacking it as-needed and without understanding what they’re doing. This means that they get better as developers over time, and code quality gets better in the long run.

Ironically, object-oriented programming was originally intended to encourage something looking like small-program development. The original vision behind object-oriented programming was not that people should go and write enormous, complex objects, but that they should use object-oriented discipline when complexity is inevitable. An example of success in this arena is in databases. People demand so much of relational databases in terms of transactional integrity, durability, availability, concurrency and performance that complexity is outright necessary. Databases are complex beasts, and I’ll comment that it has taken the computing world literally decades to get them decent, even with enormous financial incentives to do so. But while a database can be (by necessity) complex, the interface to one (SQL) is much simpler. You don’t usually tell a database what search strategy to use; you write a declarative SELECT statement (describing what the user wants, not how to get it) and let the query optimizer take care of it. 

Databases, I’ll note, are somewhat of an exception to my dislike of Big Code. Their complexity is well-understood as necessary, and there are people willing to devote their careers entirely to mastering it. But people should not have to devote their careers to understanding a typical business application. And they won’t. They’ll leave, accelerating the slide into spaghettification as the code changes hands.

Why Big Code? Why does it exist, in spite of its pitfalls? And why do programmers so quickly break out the object-oriented toolset without asking first if the power and complexity are needed? I think there are several reasons. One is laziness: people would rather learn one set of general-purpose abstractions than study the specific ones and when they are appropriate. Why should anyone learn about linked lists and arrays and all those weird tree structures when we already have ArrayList? Why learn how to program using referentially transparent functions when objects can do the trick (and so much more)? Why learn how to use the command line when modern IDEs can protect you from ever seeing the damn thing? Why learn more than one language when Java is already Turing-complete? Big Code comes from a similar attitude: why break a program down into small modules when modern compilers can easily handle hundreds of thousands of lines of code? Computers don’t care if they’re forced to contend with Big Code, so why should we?

However, more to the point of this, I think, is hubris with a smattering of greed. Big Code comes from a belief that a programming project will be so important and successful that people will just swallow the complexity– the idea that one’s own DSL is going to be as monumental as C or SQL. It also comes from a lack of willingness to declare a problem solved and a program finished even when the meaningful work is complete. It also comes from a misconception about what programming is. Rather than existing to solve well-defined problems and then get out of the way, as small-program methodology programs do, they become more than that. Big Code projects often have an overarching and usually impractical “vision” that involves generating software for software’s sake. This becomes a mess, because “vision” in a corporate environment is usually bike-shedding that quickly becomes political. Big Code programs always reflect the political environment that generated them (Conway’s Law) and this means that they invariably look more like collections of parochialisms and inside humor than the more universal languages of mathematics and computer science.

There is another problem in play. Managers love Big Code, because when the programmer-to-program relationship is many-to-one instead of one-to-many, efforts can be tracked and “headcount” can be allocated. Small-program methodology is superior, but it requires trusting the programmers to allocate their time appropriately to more than one problem, and most executive tyrannosaurs aren’t comfortable doing that. Big Code doesn’t actually work, but it gives managers a sense of control over the allocation of technical effort. It also plays into the conflation of bigness and success that managers often make (cf. the interview question for executives, “How many direct reports did you have?”) The long-term spaghettification that results from Big Code is rarely an issue for such managers. They can’t see it happen, and they’re typically promoted away from the project before this becomes an issue.

In sum, spaghetti code is bad code, but not all bad code is spaghetti. Spaghetti is a byproduct of industrial programming that is usually, but not always, an entropic result of too many hands passing over code, and an inevitable outcome of large-program methodologies and the bastardization of “object-oriented programming” that has emerged out of these defective, executive-friendly processes. The antidote to spaghetti is an aggressive and proactive refactoring effort focused on keeping programs small, effective, clean in source code, and most of all, coherent.

Functional programming is a ghetto

Functional programming is a ghetto.

Before any flamewars can start, let me explain exactly what I mean. I don’t mean “functional programming sucks”. Far from it. The opposite, actually. Not all ghettos are poor, crime-ridden, and miserable. Jewish ghettos existed for centuries in Europe, from the Renaissance to World War II, and many were intellectual centers of the world. Some were quite prosperous. Harlem was, at one time, an upper-middle-class African-American community at the center of some of America’s most important artistic contributions. The same is true of functional programming. It’s the underappreciated intellectual capital of the programming world, in that its ideas (eventually) trickle down into the rest of the industry, but it’s still a ghetto. A ghetto is the urban analog of a geographic enclave: it’s included in the metropolis, but culturally isolated and usually a lot smaller than the surrounding city. It often harbors those who’ve struggled outside of it. Those who become too used to its comforts view the outside world with suspicion, while those on the outside have a similar attitude of distrust toward those within. Ghettos usually imply that there’s something involuntary about being there, but that’s often not the case. Chinatowns are voluntary ghettos, in the non-pejorative sense, as are some religious communities like monasteries. “Functional programming” is, likewise, a voluntary ghetto. We’ve carved out an elite niche in the software industry, and many of us refuse to work outside of it, but we’re all here by choice.

What is functional programming? Oddly enough, what I’m about to talk about is not functional programming in the purist sense, because most “functional programmers” are not averse to using side effects. The cultural issues surrounding functional programming are not about some abstract dislike of computational effects, but rather an understanding of the necessity of managing the complexity they create, and using tools (especially languages) that make sane development possible. Common Lisp, Scala, and Ocaml are not purely functional languages, but they provide native support for the abstractions that make functional programming possible. What real functional programmers do is “multi-paradigm”– mostly functional, but with imperative techniques used when appropriate. What the debate comes down to is the question of what should be the primary, default “building block” of a program. To a functional programmer, it’s a referentially-transparent (i.e. returning the same output every time per input, like a mathematical function) function. In imperative programming, it’s a stateful action. In object-oriented programming, it’s an object, a more general construct that might be a referentially-transparent function, might represent an action in a hand-rolled domain-specific language (DSL) or might be something else entirely. Of course, most “object-oriented programming” becomes a sloppy mix of multiple styles and ad-hoc DSLs, especially as more than one developer comes to work on an object-oriented project. That’s a rant for later.

In general, functional programming is right. The functional approach is not right for every problem, but there is a right answer regarding the default abstractions for building most high-level programs: immutable data, and referentially transparent functions should be the default, except in special cases where something else is clearly more appropriate. Why? A computational action is, without more knowledge, impossible to test or reason about, because one cannot control the environment in which it exists. One needs to be able to know (and usually, to control) the environment in which the action occurs in order to know if it’s being done right. Usually, the tester wants to be able to cover all special cases of this action, in which case knowing the environment isn’t enough; controlling it is also necessary. But at this point, the state of the environment in which the test happens is an implicit parameter to the action, and making it explicit would make it a referentially-transparent function. In many cases, it’s better to do so– when possible. It might not be. For example, the environment (e.g. the state in a computer cluster database) might be too large, complex, or volatile to explicitly thread into a function. Much of real-world functional programming is not about eliminating all state, but about managing what state is intrinsic, necessary or appropriate.

For a concrete example, let’s say I have a blackboard, face down, with a number (state) on it, and I ask someone to read that (call it n) and erase the number, then write n+1 on the blackboard. I’m assuming he won’t lie to me, and that he’s physically capable of lifting the board; I want to determine if he can carry out this operation. If I don’t know n, and only see what is written after he is done, I have no hope of knowing whether the person carried out my order correctly. Of course, I could control the testing enviroment and write 5 on the blackboard before asking him to do this. If he writes 6, then I know he did what I asked him to do. At that point, though, the blackboard isn’t necessary. It’s more lightweight to just ask him, “what is 5 + 1?” I’ve moved from an imperative style of testing to a functional one: I’m determining whether his model of the addition function gives the right answer, rather than putting him through an exercise (action) and checking the state after it is done. The functional alternative is a unit test. I’m not trying to assess whether he knows how to turn over a blackboard, read it, erase it, and write a new number on it, because I only care about whether he can add. If I want to assess all of those as well, then I need to make an integration test of it. Both types of test are necessary in real-world software engineering, but the advantage of unit tests is that they make it easy to determine exactly what went wrong, facilitating faster debugging.

Testing, debugging, and maintenance are a major component of real-world software engineering, and functional programming gives us the tools to tackle these problems in a tractable way. Functions should be referentially transparent and, ideally, small (under 20 lines when reasonable). Large functions should be broken up, hierarchically, into smaller ones, noting that often these small components can be used in other systems. Why is this desirable? Because modularity makes code reuse easier, it makes debugging and refactoring much simpler (fixes only need to be made in one place) and, in the long term, it makes code more comprehensible to those who will have to modify and maintain it. People simply can’t hold a 500-line object method in their heads at one time, so why write these if we can avoid doing so?

The reality, for those of us who call ourselves functional programmers, is that we don’t always write stateless programs, but we aim for referential transparency or for obvious state effects in interfaces that other programmers (including ourselves, months later) will have to use. When we write C programs, for example, we write imperative code because it’s an imperative language, but we aim to make the behavior of that program as predictable and reasonable as we possibly can.

Functional programming, in the real world, doesn’t eschew mutable state outright. It requires mindfulness about it. So why is functional programming, despite its virtues, a ghetto? The answer is that we tend to insist on good design, to such a degree that we avoid taking jobs where we’re at risk of having to deal with bad designs. This isn’t a vice on our part; it’s a learned necessity not to waste one’s time or risk one’s career trying to “fix” hopeless systems or collapsing companies. Generally, we’ve come to know the signs of necrosis. We like the JVM languages Clojure and Scala, and we might use Java-the-language when needed, but we hate “Java shops” (i.e. Java-the-culture) with a passion, because we know that they generate intractable legacy messes in spite of their best efforts. Say “POJO” or “Visitor pattern”, and you’ve lost us. This seems like arrogance, but for a person with a long-term view, it’s necessary. There are a million “new new things” being developed at any given time, and 995,000 of them are reincarnations of failed old things that are going to crash and burn. If I could characterize the mindset of a functional programmer, it’s that we’re conservative. We don’t trust methodologies or “design patterns” or all-purpose frameworks promising to save the world, we don’t believe in “silver bullets” because we know that software is intrinsically difficult, and we generally believe that the shiniest IDE provides us enough to compensate for its shortfalls and false comforts. For example, an IDE is useless for resolving a production crisis occurring on a server 3,000 miles away. We’d rather use vim or emacs, and the command line, because we know they work pretty much everywhere, and because they give us enough power in editing to be productive.

From a functional programmer’s perspective, it’s easy to mistake the rest of the software industry for “the ghetto” (especially considering the pejorative association, which I am trying to disavow, with that word). Our constructions are stable and attractive, and we do such a good job of cleaning up after ourselves that there’s not much horseshit on our streets. Outside our walls are slums with rickety, fifteen-story tenements that are already starting to lean. The city without is sloppy and disease-ridden and everything built in out there will be burned down, to kill the plague rats, in ten years. We don’t like to go there, but sometimes there are advantages of doing so– for one thing, it’s fifty times larger. If we lose awareness of size and scale and what this means, we can forget that we are in the ghetto. That’s not to say we shouldn’t live in one, for it’s a prosperous and intellectually rich ghetto we inhabit, but a ghetto it is.

I think most functional programmers only get a full awareness of this when we’re job searching, and thanks to most of us being in the top 5% of programmers, our job searches tend to be short. Still, I’ve lost count of the number of times over the past five years that I’ve found a job listing that looked interesting, except for its choice of language. “5 years of experience in Java, including knowledge of design-pattern best practices.” Nope. It might be a good company writing bad copy, but its technical choices look exactly the same as those of the bad firms, so how can I be sure? The process quickly becomes depressing. It’s not that Java or C++ are “dirty” languages that I would never use. It’s that any job that involves using these languages full-time is so likely to suck that it’s hardly worth investigating. Occasionally, C++ and Java are the right tools for the job, but no one should try to build a company on these languages. Not in 2012. Java isn’t a language that people choose to use, not for primary development. Not if they’ve used three or four languages in their career. It’s a language that people make other people use. Usually, it’s risk-averse and non-technical managers making that call. A Java Shop is almost always a company in which non-engineers call the shots.

What we call functional programming is somewhat of a shibboleth for good-taste programming. We prefer the best programming languages, like Ocaml and Clojure, but we don’t actually restrict ourselves to writing functional programs. Do we use C when it’s the right tool for the job? Hell yeah. Do we put mutable state into a program when it makes it simpler (as is sometimes the case)? Hell yeah. On the other hand, we trust the aesthetic and architectural decisions made by brilliant, experienced, gray-bearded engineers far more than we trust business fads. We have a conservative faith in simplicity and ease-of-use over the shifting tastes of mainstream managerial types and the superficial attractiveness of silver bullets and “methodologies”. We roll our eyes when some fresh-faced MBA tells us that structuring our calendar around two-week “iterations” will solve every software problem known to humankind. Unfortunately, this insistence (often in the face of managerial authority) on good taste makes us somewhat unusual. It stands out, it can be unpopular, and it’s not always good for one’s career. Few stand with us. Most leave our camp, either to become managers (in which case, even a Java Shop is a plausible employer) or to accept defeat and let bad taste win. It’s hard to live in a ghetto.

Now, I have little faith in the stereotypical average programmer, the one who never thinks a technical thought after 5:01 pm, and who doesn’t mind using Java full-time because the inevitable slop is the problem of some “maintenance guy”. That person probably shouldn’t be programming. On the other hand, we’re about 2 percent of the software industry, if that, right now. We can reach out. We can do better. We’re not so brilliant that the other 98% of programmers have no hope of joining us. Not even close. There are many IDE-using, Java-hacking, semi-bored developers who are just as smart as we are but haven’t seen the light yet. It’s our job to show it to them, and if we fail to convince them that they could become 2 to 10 times more productive than they ever dreamed of being, and that programming can become fun again, then we’re the ones to blame. We must reach out, and we can probably bring 10, 20, maybe even 30 percent of programmers over to the light side, bringing about dramatic changes in the software industry.

I think the integrity of our industry depends on our ability and willingness to figure out how to do this.

Radical Transparency #1: who gets fired, how, and why.

Personal note: I wrote the bulk of this material in February 2011. I recently left a job (from which I was not fired) and admit the timing is less than perfect. There’s no connection between this essay and my personal situation, although its insights are derived from dozens of observations over the past 6 years.

I’m writing this post for a political purpose, not a personal one. Namely, I wish to provoke an epic showdown between the cognitive “1 percent” and the socioeconomic “1 percent”– a pleasant euphemism in the latter case, since it’s really 0.1% of Americans who are in access to make major decisions. I wish to force this conflict as fast as I can I believe technological trends are in our favor and we will win. It is not enough to defeat the socioeconomic “(0.)1 percent” as a class or as individuals. It is far more important that we dismantle the ugly world that they have created. If we destroy them personally but not the execrable processes they’ve set in motion, others will step up to replace each one we remove from power. We need to change the processes, which involves changing the discussion. Radical transparency is the first step toward doing this. 

I’m going to shed some light on the processes by which people are separated from companies. Involuntary termination. Layoffs. Getting fired. Shit-canned. Pink-slipped. There’s obviously a lot of nastiness surrounding this function, despite its absolute necessity to the health of an organization. I don’t intend to say that firing people is mean, or wrong, or unethical. Far from it, it’s something that all organizations will need to do from time to time. Some people are frankly toxic or unethical, and others are just not able to perform well within the organization, and must be let go.

In fact, I’d argue that the results of what happens when incompetent people are rarely fired can be seen in U.S. politics, where we face one of the most disastrously ineffective legislatures in human history because voters are so bad at firing idiots. Political officials are subjected to periodic elections (360-degree performance reviews) which are supposed to be intensely competitive. Yet the incumbent-victory rate is astonishingly high: over 95 percent in American political elections. The job-loss rate of political officials is less than 2 percent per year, despite their poor performance. In finance and technology, even people who are extremely good at their jobs have higher risk of involuntary termination than that. That contrast, I consider illustrative. For reasons like this, I will never say it is wrong to fire people, although it’s important to be decent about it. More on that later.

Two years ago, one of my friends was served with a Performance Improvement Plan (PIP) issued to him from a large company. If there is a TPS Report Museum, there must be an entire wing dedicated to PIPs. I’ll say one thing about these: they should never be used, and they don’t work. The first way in which PIPs fail is that they don’t work at improving performance. A manager who genuinely wishes to improve an employee’s performance will address the matter, one-on-one, with the employee in a verbal meeting (or series of meetings) where the intention is discovering the cause (“blocker”) of low performance, and decide either (a) to resolve this problem, if it can be done, (b) to accept transient low performance if the cause is a temporary one such as a health problem, or (c) to determine that the problem is irresolvable and terminate the employee (preferably in a decent way). On the other hand, written negative communication about performance (formalized most finally in a PIP) is universally interpreted as an aggressive move and will lead to distrust of the manager, if not outright contempt toward him. As soon as a manager is “documenting” negative performance, the relationship between him and his report has failed and he should progress immediately to a decent termination. Never PIP. Fire, but do it right.

What’s a decent termination? It’s one in which the employee is allowed to resign, provided with a positive reference (both in writing, to guarantee it, and verbally when needed) and a severance package equal to between 1 and 1.5 times the average duration of a typical search for jobs of that kind (between 2 and 3 months for junior-level positions, and 6 to 9 months for executives, and with an additional multiplier to 1.5-2 granted to those who are “overexperienced”) to compensate the employee for the unexpected job search. Of course, companies have no legal obligation to any of this, but it’s the right way of doing things. In fact, because financial pressures can result in a suboptimal job search, I’d argue that severance (when the cause issue is fit or performance rather than unethical behavior) is an ethical obligation.

Speaking of this, a lot of extremely unethical things happen in American workplaces, and that is a result not of “bad people”, but the bulk of this behavior comes from morally-average people who are scared. One of the things people fear most at work is a sudden, unjust, badly-structured termination that leads to a long-term career problem. This fear motivates a lot of the bad activities that occur in workplaces that lead to unsafe products and defrauded customers. The best thing a company can do for its culture, and for its macroscopically visible good citizenship, is to establish a policy of managing terminations in a proper way– to say that, yes, we’ll fire people whose poor performance constitutes an unacceptable cost to us, but we’ll always do so in a way that ensures that good-faith low performers (i.e. decent people who are a bad fit for their role) move on to more appropriate jobs.

How does a PIP actually affect performance? First, it destroys the relationship between the manager and the employee, who now feels “sold out”. If claims in the PIP suggest that others contributed to it, it may destroy the working relationship between the employee and his colleagues, causing isolation. PIPs usually carry a biased or even inaccurate summary of the employee’s work as the motivation for the Plan. Second, PIPs often generate a lot of additional work for the employee, making it harder to perform. A PIP usually contains deadlines equivalent to a 40-hour per week work schedule. This seems reasonable, except for the fact that many work demands are unplanned. An employee who faces responsibility for an emergent production crisis during a PIP will be forced to choose between the PIP work or the emergent crisis. A PIP’d employee ends up actually ends up with four conflicting jobs. The first is the work outlined in the PIP, which is already a 40-hour obligation. The second is any emergent work that occurs during the PIP period (which is usually unspecified in the PIP, but claimed to be covered by a vague “catch-all” clause). The third is the legalistic fighting of the PIP– the employee must contest false claims about his performance or the company will represent him as having agreed with them, which damages his position if he ends up in severance negotation. The fourth is the job search process, which must be commenced right away because the PIP usually ends in termination without severance.

This brings us to the real purpose of PIPs, which is to legally justify cold-firing employees. The PIP is the severance package. Defenders of PIPs represent it as cheaper (and it’s not, because of contagious morale effects) to keep a burned-out, miserable employee in the office for a month or two than to fire him and pay a 3-month severance package. Employees who are terminated under a PIP are usually granted no severance, because their termination is “for cause”. There’s a lot of intimidation that usually goes on here. Because the PIP is confidential, employees are usually told they will be breaking the law if they show the PIP to anyone, even an attorney. (Not true.) They’re also told that they can be terminated “for cause” (read: without severance) and that it will damage their ability to get a reference if they don’t sign the PIP, damaging their ability to contest it later. This one is true (employment is at will, and severance is not a legal obligation) but a company that gives a bad reference, especially for retaliatory reasons, will not fare well in court.

I was once privy to managerial statistics at a large company. Over a 12-month window during a time of average job fluidity, 82 out of 100 of PIP’d employees left their jobs within the duration (30-60 days) of the PIP. These might be judged to be “success” cases (they left) but if this was the desired outcome, it could have been suggested in an in-person discussion without involving nasty paperwork. In another 10 cases, the employee fails the PIP and is usually fired. In 6 more cases, the PIP is ruled “inconclusive” because of factual inaccuracies, or because the PIP’d employee is able to garner support from elsewhere in the organization, and transfer to a more sympathetic manager, who kills the PIP. Only 2 actually “pass” the PIP. Also, a PIP’d employee is likely to face another PIP within six months if he stays with the same manager, while most managers don’t want to take a chance on someone who faced a PIP (even successfully) because, statistically speaking, an unblemished new hire is a better bet.

My advice to PIP’d employees is as follows. As I said, you’ve got 4 jobs now: (a) the PIP work, (b) emergent work, (c) legalistic fighting, and (d) finding another job in the event of failing the PIP. Regarding the first, don’t blow off the PIP work entirely, or at least not openly. It may seem like it’s pointless to do this work (and it is, because your future at that company has ended) but one needs to show a good-faith effort in any case. PIPs are not guarantees of employment during the specified duration; they can be “closed” early if it is judged as obvious that the employee will not meet the PIP’s deadlines. However, in the rare case that the PIP is actually fairly structured and you can complete it in time, do not use the “surplus” to move forward on the PIP work. Instead, use this time to learn new technologies (for the job search) and contribute to open-source projects (only on your own resources). Even if you pass the PIP, your days are numbered. Regarding the second, request in writing that the manager place any emergent responsibilities on the PIP calendar. Now is the time to document everything and be a pain in the ass. If he ignores these requests, you can make a case to HR to rule the PIP inconclusive, buying you time until the manager’s next opportunity to be PIP (usually a quarter or two later) you comes along. You should have a new job before that happens. On the third of your four jobs– the legalistic wrangling– document everything, but keep communication terse, objective, and professional. Don’t spend more than an hour per day on it, but if you can waste your manager’s time with HR paperwork in the hope of getting the PIP ruled inconclusive, do it. The fourth of these– the job search– is where one’s highest priority should be focused. Represent interviews (in email; again, document everything) as doctors’ appointments. This is also a great opportunity to disclose health issues, which reduces the probability of being fired without severance.

My advice for companies or managers considering the use of PIPs. Don’t. Sure, you’ve got version control and backups, but do you really want an essentially fired employee in the office for a month? It’s amazing to me that companies have security escort employees out of the building when they are officially terminated (at which point the shock has passed and emotions are likely to be resolved) but keep them in the office for months on PIPs, when they are effectively fired. PIP’d walking dead are far more damaging to morale than terminations.

Finally, my advice for employees who actually get fired is simple. Don’t be a dick. Disparaging firm-wide emails will ruin your reputation, and property damage is illegal and fucked-up and wrong, and therefore not in your interest at all. Depart cordially. On references, there are two kinds: unofficial (which can be a peer you select) and official (manager or HR). Usually official references say very little, but have your reference from that company checked by a professional service and get an attorney involved if anything negative is being said about you. Cultivate positive relationships with colleagues after your exit. Most importantly, move on quickly. When asked about the previous job, stay professional and represent the termination as one that you initiated. If your ex-employer ever contradicts you and says you were fired, then congratulations: you get a huge settlement for a negative reference.

Also, don’t let it hurt your self-esteem. There are two rites of passage in the modern corporate world. The first is getting fired. The second is learning that getting fired is not usually a reflection on the employee (more on this later, on who gets fired). Your self-esteem will benefit if you experience the second before the first.

Ok, so that’s a bit about the process. It’s important for people to know this stuff, so they know how to box if the fight comes to them. Now, I’d like to shed some light on who gets fired. Low performers, right? Well, not always. Companies initiating impersonal layoffs for financial reasons will usually drop low-performing teams and businesses, but not all those people are individually low performers. Personal firings are a different story. These don’t occur because “companies” choose to fire these people; people do. So what are some of the motivations that lead to this? First, let me say that political (non-necessary) firings usually occur in the middle-bottom and middle-top of an organization’s performance spectrum. The bottom 10% are usually so used to lifelong, serial incompetence that they’ve mastered office politics as a survival skill. The top 10% are objective, visible high-performers who tend to have political clout on account of the value they add to the organization. The middle 60 percent rarely stick out enough to get into trouble. This leaves two deciles– the 2nd and the 9th– that are most exposed to personal terminations.

Second-decile firings tend to be “trophy firings”, initiated by managers who wish to impress upon higher-ups that they “take action” on low performers. They aren’t the worst people in the organization, and whether it’s of economic benefit to retain them is not strongly resolved either way, but they’re weak enough that if they leave the organization, they won’t be missed.

Ninth-decile firings are “shot-from-the-bridge” firings, initiated against a high performer by a jealous peer before she develops the credibility and capability to become a threat (i.e. graduate into the objectively and visibly high-performing 10th decile). These people are usually fired because they’re creative, passionate, and opinionated. Now, managers don’t initiate ninth-decile firings, at least not willingly. Managers want more people to be more like their 9th- and 10th-decile reports. Rather, those firings to be initiated by “young wolves”, who are typically same-rank co-workers, but sometimes insecure middle managers, who sabotage the target’s ability to perform. The textbook young wolf is Pete Campbell in the early seasons of Mad Men, the slimy and machiavellian junior account executive who lies and threatens his way to the top. (Disclaimer: I wish the term for such people wasn’t young wolves. I like wolves. I like them better than most people.)

A direct confrontation with a manager about a 9th-decile report is not a politically wise idea, because managers know that 9th-decile employees are highly valuable. So young wolves focus on giving the target some disadvantage in project allocation. Most technical teams have a “people manager” (who is savvy to political machinations) and a technical lead (who is not usually politically savvy, but has authority to allocate projects, make technical decisions that affect the team, and delegate work). Tech leadership is a hard role because (a) it involves a lot of responsibility with minimal glory, and often no additional compensation, and (b) tech leads are responsible as individual contributors in addition to their leadership role. Thus, the tech lead is usually overburdened, and young wolves usually cozy up to him, with the intention of “helping out” on work allocation. Then they start delegating impossible, discouraging, or unsupported work to the target, the 9th-decile employee they want to shoot from the bridge before they become 10th-decile. When the target’s performance drops (transiently) to about the 7th or 8th decile, they begin to speak about how she “does good work, but isn’t as good as we thought she’d be”. As she sees leadership opportunities and the most interesting projects receding from view, her motivation declines further. By the time she’s at the 5th or 6th decile (because she’s now beginning to look for other opportunities, and prioritizing work that benefits her external career) the party line is that she’s “not giving her best”, which is, by this point, entirely true and equally irrelevant to whether it’s economically beneficial to employer to retain her. Social rejection of her is encouraged through gossip about her being “not a team player” and whispered suspicions that she’s looking for employment elsewhere. When she enters the 3rd or 4th decile, the young wolves engage her manager and– if the manager is unaware of what’s happened (a young-wolf situation)– she receives the first official warning signs, such as negative feedback delivered over email instead of in spoken form. This is an aggressive move that confirms this now-declining employee’s suspicions that her future there is over. If she’s still working there (unlikely, in a decent economy) by the time she falls into the 2nd decile, she’s now a trophy-fire.

This process of targeted, intentional demotivation, prosecuted not by managers (whose interests are aligned with the success of the whole team) but by young wolves, can occur over years or over days, but it’s a discernable pattern that I’ve seen play out dozens of times. The problem is that “flat” managerial arrangements don’t work. I don’t like the alternative (official hierarchy) much, so I wish they did work, but they almost always fail. A “flat” organization means “we’re large enough that we need hierarchy but we’re too socially inept to have hard conversations”. In most cases, I actually think that large companies (for post-industrial knowledge work) are inherently broken, because neither the hierarchical model nor the flat model results in good products. Why is “flat” so bad? It means that a manager has a ridiculous number of reports (sometimes over 50) and that it’s impossible for him to be on the lookout for young-wolf behavior, which is immensely damaging to the company because it tends to drive out the most creative individuals. Young wolves thrive in environments of managerial inattention.

How does one combat a young wolf within an organization? As with any progressive, but curable, parasitic infection, early detection is the key. A decent manager who identifies a young wolf will send it to the pound and have it put down, but effective young wolves evade managerial detection for a long time. Defining the term helps, but there’s a danger here. If “young wolf” enters the business lexicon, it will just become a term of disparagement that young wolves use for their targets. The most effective young wolves, instead of declaring their targets “not a team player”, will instead attempt to paint their targets as young wolves. So here, I don’t have the answer, but I have a thought.

I’ve been coding for five years, and I’ve made a serious mistake. At this point, I have no open source contributions. Two and a half years I put into a failed startup that should have allowed me to open-source my work (30,000 lines of Clojure code to build a custom graph-based database, and some of the best technical work of my life) but because the CEO of that failed startup is a vindictive asshole who wishes to punish me for “leaving him”, so that work is gone. For this entire five years, I was a “company man”, pumping 50 to 60 hours per week of software effort into companies with which I no longer have a relationship. I have a “great resume”, but nothing in the open-source world to show that I can actually code. I’ll be just fine, and I’m going to take some time to write some open-source work, but that’s a serious regret I have over how I’ve spent the last half-decade. Why’s open-source software so important? Radical transparency. It provides a socially beneficial mechanisms for programmers to demonstrate, objectively, high levels of skill. It provides sufficient job security to the best programmers that they can fearlessly avoid most office politics and focus on work, which generally makes the quality of that work a lot higher.

I think radical transparency should apply within large workplaces as well. Many middle managers discourage line (i.e. non-managerial) employees from taking initiatives that would make them visible to other departments, for the fear that it would make them a “flight risk”. Managers of undesirable projects are aware that external visibility for their reports will lead to transfers, and managers of average projects are often insecure that their projects might be undesirable. Moreover, many companies discourage unmetered work, such as education, career development and extra-hierarchical collaboration (i.e. helping out other teams) which further isolates line employees. The result of this is that, for many employees, the manager is literally the only person who knows what the employee is doing. This gives the manager free rein to represent his opinion of that employee (which, if the manager has 25 reports, is very probably unreliable) as objective fact. Human Resources (HR) is supposed to ameliorate conflicts that occur when this leverage is abused, but the truth about HR representatives involved in manager-employee disputes (and terminations) is that they work for the company’s lawyers (and that’s why they think PIPs are a good idea) and no one else. At any rate, this pattern of dysfunction and outsized managerial power is sometimes called “manager-as-SPOF”, where SPOF is a technical abbreviation for “single point of failure” (the weak point of a system).

Manager-as-SPOF, isolation, and a lack of transparency about who is doing what allow socially manipulative twerps (young wolves) to succeed. In this kind of environment, 9th- and 8th- and 7th-decile performers get almost no visibility and are defenseless against a political campaign against them. The solution is to make the contributions of these employees visible. It’s to make everyone’s contributions visible and to foster an environment that makes many of the worst manifestations of office politics untenable. It’s to encourage employees to share their contributions with the whole company, and to work on things that benefit all of their colleagues– not just their managers.

A future RT post will focus on this: how to construct a radically transparent workplace, and what it would look like.

Taxation without representation: why code maintenance fails– and how to fix it.

The software industry, as a whole, has fallen into a dismal state. Software development is supposed to be a creative endeavor, but often it’s not. An overwhelming portion (more than 80% on mature codebases) of software engineering time is spent on software “maintenance” instead of new invention, a morale-killer that contributes to the high burnout rate in software engineering. In spite of these sacrifices on the part of developers, code quality seems nonetheless to decline inexorably, to the point where many software projects become intractably unfixable after a certain amount of time. Not only is maintenance painful, but it doesn’t seem to be working very well. What causes this? How does it happen? There are many factors, but I think the biggest problem the software industry faces is organizational: by failing to allocate maintenance responsibilities properly, the result is that these jobs are done poorly and the legacy/maintenance load becomes enormous.

Before reading further, I ask the reader to peruse Jeff Atwood’s excellent blog post, a header for what I’m about to write. In my mind, the most important point he makes is this one: We should probably have our best developers doing software maintenance, not whoever draws the shortest straw. 

Code maintenance, as usually practiced, is dull and lacks creativity. It’s also usually a career tarpit, because new invention is generally superior (from a developer’s perspective) in terms of visibility, education and advancement. Maintenance work rarely delivers on any of these. Maintaining one’s own code is a crucial educational experience, in which the developer sees first-hand the consequences of her architectural decisions. Maintaining the monstrosities created by others? It’s not much of a learning experience after the first time it’s done, and it’s rarely visible either. This incentive structure is perverse, because a working, healthy system that fulfills known, existing needs (i.e. the outcome of a successful maintenance project) is of more value than a new one that may or may not prove to be useful.

Maintenance is extremely important work; it’s the upkeep of one of a software company’s most important assets. Unfortunately, certain systemic aspects of organizational cultures make it unlikely to be rewarded in accord with its importance. The consequence is that maintenance work tends to “roll downhill” until it reaches junior programmers who lack the leverage to work on anything else. The result of this is that maintenance projects are usually done by people who don’t have the organizational knowledge, level of skill, or, most importantly, the political pull to do the job properly.

Inept or even average-case software maintenance fails to improve the general health of the code. A few bugs may be fixed or reactive features added, but the overall health of the software tends to decline. Methods become longer, dependencies are added, junk parameters are added to existing functions, and the architectural intentions behind the original system are buried under thousands of hacks. Code maintenance as practiced tends to beget an even greater maintenance burden in the future.

What needs to change? There’s a simple answer. Give maintainers power. The reason why the vast majority of software maintenance projects fail to improve code health whatsoever is that the developers in a maintenance role (usually junior programmers with no clout) lack the political pull to do their jobs properly. Telling a more senior developer, who was “promoted away” from maintenance duties years ago, to drop everything and document prior work is generally not an option for the maintenance programmer. Nor is removing ill-advised functionality from interfaces (someone more senior is guaranteed to “need” that function) or declaring a project in need of a rewrite. This is the sort of clout that a maintainer needs to have in order to have any shot whatsoever at succeeding.

There’s a flip side of this. If code maintenance work implies power, it absolutely shouldn’t be delegated to junior developers who can’t yet be trusted with that kind of power.

Giving software maintainers power will remove most of the stigma associated with such projects, seeing as they are hated because of the lack of power they often entail. It’s taxation without representation. Often, the maintainer’s job is specified in uninspiring and untenable terms like, “Make it work exactly like the old system, but faster”. Bug-for-bug replications, as well as faithful reproductions of any software without a spec, are doomed to failure. Improving code health necessitates imposing change on client developers. Rarely are such costs considered tolerable for the sake of “mere” upkeep, and maintenance projects usually, for that reason, fail to provide lasting improvement. The low rate of success observed on such projects makes them undesirable, and therefore they are avoided by the most skilled developers.

On the other hand, if maintenance responsibilities were coupled with sufficient power to restore the health of the codebase, such projects could become attractive enough that those with skill and seniority would want to do them. People who work on a daily basis with software generally develop a deep interest in the system’s health, and this alone can be motivation to commit time to its upkeep. Software maintenance has the reputation of grunt work, but it really isn’t intrinsically unpleasant for all developers; there are people who would enjoy such projects immensely if they were properly structured.

On structure, it’s critical that a company of any size separate the “software engineer” job description (which currently entails creative, new invention as well as maintenance) into two separate job descriptions. One (I’ll call it the “software architect” track) exists for the creative engineers who tend to fail at maintenance efforts because of their inability to tolerate ugliness. The other (I’ll call it the “maintenance engineer” track) exists for the “Sherlock Holmes” sort who enjoy deciphering ineptly- or even hostilely-written code. Since qualified maintenance engineers are uncommon, and since most developers find that work undesirable, maintenance engineers should be paid highly. I would argue that a 30% premium is not unreasonable. Moreover, they must be given social status and clout, since their job is to be pains in the asses of developers who are potentially more senior than they are. If maintenance projects are structured and allocated in this way, it’s very unlikely that they will dominate engineering time as they do now.

One thing I would argue against, however, is any suggestion that the maintenance engineer’s existence should absolve developers of the responsibility for maintaining their own code. It doesn’t. Maintenance engineers may be given a supervisory role in code and design review, but maintenance of one’s own work should remain each developer’s own responsibility; otherwise, “moral hazard” kicks in. I will also note that it is a terrible practice for developers to be “promoted away” from maintenance responsibilities, those being delegated to less important folk, as a reward for a successful launch. If the launch’s success came at the expense of enormous “technical debt” (the interest on which is usurious) then a maintenance time-bomb has been built.

I’ve addressed the social changes required in order to allow successful software maintenance. What will this entail in terms of project structure? In-place maintenance of defective systems– any software module that requires full-time maintenance effort fits the bill for “defective”– will become rarer, with outright replacement more common. That’s a good thing, because software either enervates or empowers and the former kind can rarely be upgraded to the latter through incremental changes. Note here that I’m advocating replacement, not “complete rewrites”. In my opinion, total rewrites (unless specific efforts are made to avoid prior mistakes) are often as inadvisable as in-place maintenance; the latter keeps defective software on life support, but the former practice often generates new defects– often more flaws, on account of the time pressure that complete-rewrite projects usually face.

What is gained in doing all this? First of all, job satisfaction among software engineers improves dramatically. Instead of the industry being characterized by a gloomy, never-ending maintenance slog, the proper allocation of such work ensures that it’s done properly and quickly rather than being passed from one hapless junior developer to another. Second, engineer productivity goes through the roof, because engineers with creative and architectural talents can unlock them instead of being forced to contend with legacy misery. Third, employee retention is improved not only for individual software companies but for the industry as a whole. Instead of architecturally-inclined engineers being burned out in droves (I’ll note that people with strong architectural talents tend to be notoriously intolerant of ugliness, to a point that can be emotionally and intellectually crippling in software maintenance) by the legacy burden that is the current reality of the industry, they’re encouraged to make a career of it, the result being better software architecture across the board. The result of these improvements in architectural know-how will be better products– more comprehensible, more attractive, easier to improve and keep up, and more profitable.