Street fighting, HR, and the importance of collective action.

Street fights

Street fighting is a topic on which a large number of men hold strong opinions, although very few have been in a true fight. I haven’t, and I hope never to be in one. Playground fisticuffs are one thing, but for adults, street fighting is far too dangerous to be considered anything but a terrible idea. It’s not likely to happen, but a punch from an adult can kill. The exact mechanics by which punches to the face incapacitate aren’t fully understood, but rotation in the neck (constricting blood vessels) seems to be a culprit, and the forces involved in a proper face punch are enough, with the right conditions, rupture blood vessels in the neck and head and end a life. A person falling and hitting the ground can do it, too.

Anything before age 16 doesn’t count. Martial arts sparring doesn’t count. Boxing doesn’t count. Fight clubs don’t count. Those are a lot safer. In a real fight, you typically don’t know your opponent. If he wins, he might kill you (intentionally or otherwise) once you are on the ground. He may have a weapon and, even if he doesn’t, he can kill you with a few kicks or punches. Losing means you’re likely to end up in the hospital (and possibly dead). Winning means you have to appear in court, possibly for murder. Either way, it’s not a good place to be. If there are bystanders, the loser faces humiliation but probably won’t die. If there are none, it is up to the one who wins the fight, and unintentional (or intentional) deaths from punches and kicks happen all the time.

The moral bikeshedding around fistfighting (and the actual physical risks of it) are discussed in this analysis of the Trayvon Martin case. Is George Zimmerman a huge asshole and a probable racist? Absolutely. Should he have followed and harassed a 17-year-old? No, not at all. Is George Zimmerman a good or admirable person? Clearly not. Was Zimmermann in mortal danger during that fistfight? Yes. He was on the ground, being punched, which alone puts his life in danger. (“Ground and pound” constitutes attempted murder in most jurisdictions.) He had a lethal weapon and, if he lost capacity, he’d also lose control of the weapon. Zimmerman’s wrongdoing was following a stranger in the first place– but not firing a gun in “just a fistfight”. When the opponent is a stranger and there is no one to break up the fight, there is no such thing as “just a fistfight”.

Fistfights (and armed confrontations) aren’t like what people see in movies or on television. Some notable differences are:

  • in most, the loser is incapacitated within a few seconds. One punch can do it, and the element of surprise (that is, being the one to start the fight) confers a huge advantage.
  • fighting is exhausting and most people will be winded (and vulnerable) inside of 15 seconds. Fights become remarkably more dangerous at this point, a common cause of unexpected death being falls after a decisive punch.
  • it’s very difficult to get up when grounded, especially if one has taken several blows to the face or head.
  • an incapacitated opponent is easy to kill, either with continual punching, kicks to the head, or a stomp to the head or neck. Most fights are broken up before that can happen, but unexpected deaths (even from a single punch) occur, sometimes hours after the fight.
  • if there are two or more opponents, fighting skill and strength typically won’t prevent a very bad outcome. Two-on-one is dangerous.
  • most untrained people, when attacked, will panic. Between the short duration of typical fights, the incapacitating nature of blows to the face or head, and the effect of surprise, that initial second or two of shock is enough to cause them to lose the fight.

Knowing all this, I also know to avoid getting into fistfights. I might know more than most people about them (because I’ve done my research) but I won’t pretend to have any idea how I’d perform. It’s not worth finding out.

Enough about fistfighting, though. It’s not that interesting. I just wanted to bring it up, because it’s something that men think they know a lot about, and they tend to underestimate the risks. There are a lot of people who think they know what they’d do in George Zimmerman’s position (panicked, mounted, having sustained several punches). Most of them don’t. They haven’t been in a fight since junior high, and those don’t count.

“I would…” Stop right there. You don’t know.

Let’s talk about organizational life, on which I’ve shared a lot of opinions.

Ask a software engineer in Silicon Valley for his opinion on collective bargaining, and you’re likely to hear a negative response. Unions have been targeted for negative publicity for decades and have a reputation for mediocrity and corruption. “I don’t need no stinkin’ union!” Your typical 25-year-old engineer probably hasn’t been fired from a job he wanted or needed to keep (a summer job doesn’t count). He’s probably never faced financial pain or stress on account of a bad job outcome. If he needs to negotiate a severance (or, a more delicate task, an agreed-upon reference) from an employer unwilling to budge, he probably has no idea what to do.

As with fistfighting, most people have an alarmingly precise image of how they’d get out of a bad situation and, for the most part, they don’t know.

“If I were in that grapple, I’d grab his wrist and bite down, then drive my elbow into his balls, then knee him in the head until he gave.” (No, you wouldn’t. You’d be panicked or confused, having taken several punches, and struggling to stay conscious.)

“If I were in a two-on-one fight like that, I would…” (Sorry, but in a two-on-one fight against capable, adult opponents, you’re fucked.)

“If I were put on a PIP, I would…” Same principle. If you’ve never been in a real fight, you don’t know.

Companies have trained fighters on staff, between their attorneys and HR. They have hard fighters who’ll insult and belittle you, and soft fighters who’ll pretend to take your side, while taking notes that can be used against you. If the company wants you in a two-on-one, you’ll face that: your manager and HR, or your manager and his manager. You basically can’t say anything in a two-on-one meeting. They will plan the meeting beforehand together, decide on “the story” afterward together, and (most likely) you’ll have no credibility in court or internally. If the meeting is adversarial and two-on-one, say as little as you possibly can.

In a police interrogation, you’d have the right to have your attorney present, and the right to remain silent, and you really want to exercise those rights. Watch this incredibly convincing video, backed by decades of experience, if you have doubt on that matter. What about a PIP meeting, which comes unannounced? On Tuesday, your boss was congratulating you on your work. You were an employee in good standing. On Wednesday, you’re told in the middle of a 1-on-1 “not performing at your level”. You were ambushed; there was no sight that this was coming. That corner you cut because your boss told you to do so is now being cited as an example of the low quality of your work. Most people think they would immediately fight intelligently, with the same confidence as applied to fistfights. Not so. Between the initial panic (especially if the person has financial needs) and intense emotions related to betrayal, it’s going to be really hard not to shit the bed when under a PIP ambush. And, as with a fistfight, it’s really easy to lose quickly. Raise your voice? You’ll probably be fired “for cause” because of “threatening behavior”. Attempt to amend the PIP’s factual inaccuracies? (There will always be inaccuracies in a PIP and they’re intentional. Those documents are supposed to piss you off because about 1 in 5 will lose his cool, thereby firing himself on the spot.) That might be construed as insubordination.

HR is probably present in meetings with management, from that point forward. The reason put forward to the PIPee is to protect the employee against managerial retaliation, and that’s complete bullshit. HR is there to corroborate management’s version of the story (even if management lies). Nothing that the employee says can improve his case. As with a fistfight, two-on-one means you’re probably done. Avoid it if you can.

In one way, it’s more perilous to be under PIP ambush than in a police interrogation, though. In the latter, one has the right to be silent and to call in a lawyer. In a PIP meeting, the rules aren’t clearly defined. Is an employee who refuses to answer a manager’s question guilty of “insubordination”, allowing termination on the spot? Can the employee be fired for refusing to sign the PIP “until I talk to a lawyer”? (That will be noticed.) Can the employee turn down an “independent” HR party’s meeting request “to hear your side of the story”? (Don’t fall for this.) Is the employee guilty of insubordination if he shares, with other employees, that he’s under a PIP, and voices his supposition that the decision to place him on it was political? At-will employment is complicated and undefined in many cases, and the answers to these questions are, “no one knows”. The company will terminate (once a PIP is shown, the decision has been made and only a change in management can reverse it) when it feels it can do so with minimal risk and cost. Some companies are aggressive and will use anything that is said in a PIP or HR meeting as cause to fire without severance. Now, at-will employment has a lot of serviceable corner cases (despite what employers claim) but is it worth it to launch a lawsuit against a deep-pocketed adversary? For most people, probably not. And lawsuits (again, like fistfights) are another form of combat in which an untrained person has no idea what to expect, and is at a disadvantage. Say the wrong thing, and it will be used against you. Even if you “should” win because you have the space or superior ability, you can lose in seconds.

With PIP meetings, management has had time to prepare, and the employee is ambushed. That unfairness is intentional. It’s part of the design. Same with the two-on-one intimidation. In a PIP meeting, silence will be presented as a non-option. Even though this is illegal, most employers will claim that not signing the PIP will lead to termination (record this conversation on a cell phone if you can; but if you’re not unusually calm, there’s a low chance that you’ll remember to do this). They can bring anyone they want into the room, but you almost certainly won’t be allowed to have an attorney or representative present. It’s unfair, it’s shitty, and you need to have a plan before it ever gets to that. Your best option is to say, “I need to go to the bathroom”. Take time, calm down, and possibly call a lawyer. If you return to the PIP meeting 20 minutes later, so be it. They can wait.

Macho self-reliance

No amount of self-defense training will make it safe to get into a street fight. The odds may improve, but it’s still an inherently dangerous thing to do. Knives aren’t good protection, being much harder to control in a fight than one might think, and inviting other weapons such as guns. Speaking of guns, most people wouldn’t know what to do with a gun in a situation of actual danger. Many would think that, with a gun against four unarmed assailants, they’d be able to emerge the winner. I wouldn’t count on that.

All that said, there are plenty of people (mostly men) who believe that, because of their martial arts training or because they possess a weapon, they can safely go to places, and engage in behaviors, that make fistfights common. They’re wrong. The issue is that people wish to identify with the winner in any conflict. “I would never end up on the ground like that.” They overestimate their performance in the face of fatigue, panic, confusion, or under time pressure. Until something bad happens to that person, many people assume it never will. “I’d never be put on a PIP, because I’m good at my job”. That’s wrong, too. True low performers are usually eased out through other means. PIPs are generally used to sucker-punch a politically-targeted high performer, and they work. Plenty of people shit the bed the first time they see one.

In the world of work, this macho self-reliance is seen in the resistance of many workers to any sort of protection. You hear this one a lot from software engineers: “I wouldn’t want to work somewhere where it’s hard to fire people.” I’m sorry, but that’s a bit ludicrous. It should be hard enough to fire people that the politically unlucky still end up with a decent severance, and hard enough that people who fit poorly with their first project are given at least one more chance.

Let’s talk about the “10x engineer”. That effect is real. It’s not only about natural ability, so it’s not the same people who’ll be “10x” in every single context. Motivation, project/person fit, political position (those who write code and make decisions will always be more productive than those who read code and are downwind of decisions) and domain-specific expertise all apply. To understand why, consider the Gaussian bell curve, which emerges as random variables compound additively. In most human affairs, though, they compound multiplicatively, producing the “fat-tailed” lognormal distribution. Sometimes, there are knock-on effects that produce an even fatter-tailed power law distribution.  (If this all sounds like jargon, the takeaway is, “outliers are more common than one would think.”) Consider a world in which a new employee’s productivity is a function of the outcome of a 6-sided die, like so:


| Die |  A  |  B  |
-------------------
|  1  |   0 |   4 |
|  2  |   9 |   6 |
|  3  |  10 |  10 |
|  4  |  10 |  20 |
|  5  |  11 |  40 |
|  6  |  11 | 100 |

In scenario A, average employees (10 points) have job security, because even at that point, they’re above the mean productivity (8.5 points) of a new employee. Even the noticeably below average employees are OK. On the other hand, in Scenario B, the mean is 30 points, which means that the mediocre ’3′s and ’4′s (10 and 20 points, respectively) should be fired and will be, even if it’s not their fault. In practice, it doesn’t workout quite that badly. Firing average employees is bad for morale, and the “10x” effect is as much about motivation and purpose as it is about the individual. But we can see, from a theoretical basis and considering the convex nature of software development, why tech companies tend to have such mean-spirited management practices. From their perspective, they’re in the business of finding and exploiting 10x-ers (who won’t be paid anything remotely close to their value, because they’re lucky to still be employed) and anyone who seems not to be a 10x-er should be tossed back into the sea.

Closed allocation tech companies, one notes, are notoriously awful when it comes to internal mobility. This is a severe ethical issue because internal mobility is promised almost inherently by big companies. It’s the one advantage large companies have over small ones. Yet, most technology companies are set up so that median performers are practically immobile (unless their projects are canceled and they must be put somewhere). Headcount limitations and political intricacies (one manager not wanting to upset the other) arrange it so that middling and even average-plus performers can’t really move anywhere. The fact that there are political externalities to deal with, alone, makes them less desirable to the target team’s hiring manager than a fresh recruit from outside the company. Stars generally are worth fighting a broken transfer system to attain, but they generally don’t want to move, because they have a gravy train they’d rather not stop.

Most software engineers think too logically and will take basically reasonable premises (“low performers should be fired”) to unreasonable conclusions (“performance reviews should be part of the transfer packet”). This allows technology companies to abuse them, taking advantage of their just world delusion and rugged individualism. When very real abuses are targeted toward those around them, they fail to perceive that such things could ever happen to them. “I’m a rock star coder. I’ll never be put on a PIP! If I were, I’d just work weekends and show the boss he’s wrong!” This is like claiming that knowing a couple jiujitsu throws makes it safe to get into bar brawls. It’s just not true.

Do we need unions?

Originally, I said no. When I said that, to be frank about it, I didn’t know what the fuck I was talking about. I got professions right, but I had a defective understanding of what labor unions actually do and why they are important.

I’m not ready to come down on either side of that question. We also don’t want to see the seniority-bound nightmare of, say, the airline pilots’ unions. Also, uniformity in compensation begets mediocrity on both sides and we don’t want that; the good news is that writers’ and actors’ unions seem not to destroy the upside in pay for those people. Hollywood’s labor pool, including celebrity actors, is unionized and hasn’t suffered in quality or compensation for it. Professional structure doesn’t necessarily lead to mediocrity.

However, we need collective bargaining. We need the right of a software engineer to have an independent representative, as well as legal counsel, in the room when in hot water with management (to prevent those “shit the bed” PIP ambush scenarios). We need protection against political terminations, such as for revealing salary information (the general prohibition offices have against that isn’t for anyone’s benefit but their own) or committing the overperformance offenses (doing a job too well, and embarrassing others) typical of engineers. We need to be part of a political body that corporate management actually fears. When we act in good faith against the interests of our employers (say, by revealing abuses of power) we shouldn’t have to do it alone. We also need to have the right to have those contractual issues that are embarrassing to negotiate for oneself (severance, health accommodations, privacy) negotiated for us by more experienced parties. We need automatic legal assistance in questionable terminations, and automatic work-stopping shutdowns (strikes) against any employer who resorts to a negative reference. We need sleazy and unreliable “back channel” reference checking to be abolished, with employers who engage in or allow it being sued or struck into oblivion.

Right now, we get none of this, and it’s a total fucking crime.

That delusional programmer self-reliance (“I know how to handle myself in a fight”) allows employers to isolate “troublemakers” and pick them off, one by one. It allows them to negotiate salaries and conditions down, brutally, due to asymmetric information and broken, reputation-driven power structures in which even six months out of traditional employment can ruin a person’s life. It allows stack ranking and closed allocation, which any labor body with self-respect would fight back against.

Because of our nonsensical self-reliance, when we do fight, it is usually an individual acting alone. Employers can easily ruin the reputations of isolated individuals, and few will protect a person thus disparaged. Consequently, those battles are typically ineffective and humiliating to the fighter. The conclusion of the workers is that management simply can’t be fought. Yet those same people believe that, through hard work or some notion of personal “merit”, they’ll never face political adversity, unfair terminations, or humiliating, destabilizing processes such as PIPs. Like armchair street fighters, they don’t know what the fuck they are talking about.

Collective strength

Street and bar fights are depressing, illegal, stupid, and dangerous. Your average person would have no idea how to respond in a sudden, unexpected assault, and be completely at the mercy of the attacker. Others who could assess the situation would have a chance to interpose, but the victim would have lost the fight by the time the initial shock wore off. The good news is that such things are uncommon. Most people haven’t been in a fight since high school, if ever. Why? In public spaces, attacking another person is dangerous and generally a terrible idea. One might get beaten up oneself, especially if others interpose, and jail time is likely. When it comes to physical violence, there’s an understanding that, while rare, bad actors do exist and that it’s worth it to protect a stranger against an attack, if one can. People will often defend each other, which makes the costs and risks of assault sufficiently high that it’s a rarity.

In the workplace, we don’t see this. When a good employee is served with a PIP or fired, nothing really happens. People live in such fear of management that word of the unjust termination is unlikely to travel far, and even more unlikely to have any effect on the company. Those who oppose a company’s management in any way, no matter how trivial and even if it is accidental, are isolated and embarrassed by the organization. If it really cannot afford the image issues associated with a clearly political demotion or termination, it will simply assign him untenable work, interfere with performance, and render it impossible for him to do a good job, making the eventual outcome appear deserved.

The result is that sudden, unpublished assaults are common in the corporate world, sometimes because a person falls into an unknown political trap and winds up in a management-level grudge, sometimes because the organization must protect unrelated interests, and sometimes for no reason at all. They are rarely talked about. The person is isolated and either goes silently, or is humiliated. Opposition to the organization resulting from his treatment will never happen. In light of the severe (and unreasonable) stigma placed on employees who “bad-mouth” ex-employers, he can’t sue or publicly disparage that company without risking far more than he should ever have to put on the line. No one has the individual employee’s back, and that’s a shame.

Software engineers might be the worst in this regard, because not needing support is taken as a badge of pride, as seen in tech’s celebration of reckless firing, punishing work conditions, and psychotic micromanagement in the name of “agile”. “I don’t need no stinkin’ rights. Two fists are all I need.” That just makes no sense. It renders the individual programmer (a person with skills for which demand is so high that, if we acted collectively, we could really improve things for ourselves) easy to isolate, exploit, and humiliate if he even attempts to get a fairer deal.

The enemy, and this is especially a problem for software engineers, is delusional self-confidence. People feel a lot safer than they actually are, and to communicate the truth of one’s lack of safety is taken as admitting weakness, which allows the bad actors to isolate their targets easily. This prevents non-managerial professional groups (such as software engineers) from acting collectively and overcoming their current disadvantage.

Meritocracy is the software engineer’s Prince Charming (and why that’s harmful).

One of the more harmful ideas peddled to women by romance novels and the older Disney movies is the concept of “Prince Charming”, the man who finds a young girl, sweeps her off her feet, and takes care of her for the rest of her days. It’s not a healthy concept, insofar as it encourages passivity as well as perfectionism in mates. But it also encourages women to make excuses for bad (and often abusive) men. Because the archetype is so unreasonable, men who can make themselves seem to fulfill it are the manipulative and sometimes abusive ones, not genuine good (but flawed) men. I’d argue that software engineers have a similar Prince Charming.

It might begin as a search for “a mentor”. Savvy software engineers take knowledge and favor from multiple people, but every Wall Street or Silicon Valley movie showcases a mentor/protege relationship as the path to success. Meet this magical person, and he’ll take care of your career from there on out. That doesn’t exist for most people, either, and most software engineers learn that around age 25. Their counterreaction is to develop a bizarre self-reliance in which they start refusing help, wanting to work alone, and denigrating those who advance their careers based on “politics” or “connections”. Having too much dignity to wait for a magical mentor to rescue them from mediocrity, they insist on their new Prince Charming, an interpersonal force that will recognize and elevate talent: meritocracy.

The problem with meritocracy is that every organization claims to be one, yet almost all organizations are deeply political. Software engineers are not a subtle breed, so I must imagine that they imagine most non-meritocracies perceive themselves as such, and admit so much, and that’s clearly not true. Oil companies, banks, startups and dysfunctional academic bureaucracies all have this in common: they believe in their own meritocracy. Otherwise, they wouldn’t be self-consistent and stable. “We’re a meritocracy” means nothing. And what is “merit”? Organizations make promotion decisions not to recognize some abstract principle of “merit”, but on what is perceived to be in the short-term, narrow interest of the organization. It’s not what software engineers mean when they use the term merit, but one could argue that political acumen is organizational merit. The people who are promoted in and end up dominating organizations are… those most able to convince organizations to promote them, whether through delivering objective value or by trickery and intimidation. It’s a self-referential, Darwinian sense of “merit” akin to “fitness”. Darwinian fitness is neither a matter of good, bad, or anything other than the ability to self-replicate.

Of course, I know what software engineers mean when they say they want to live in a “meritocracy”. They want important decisions that affect their working lives to be made by the right people. The problem is that the ability to make good executive decisions is almost impossible to measure, reliably, especially on a timeframe that businesses would consider acceptable. Political machinations can happen, on the other hand, in split seconds. Saying something stupid in a meeting can end someone’s career, even if that person is, in general, a good decision-maker. It takes too long to select leaders based on the quality of their decisions, so organizations develop political side games that end up consuming more energy, time and attention (especially at high levels) than the actual work or purpose of the organization. Generally, this side game takes on the feeling of a war of attrition. Nonsensical pressures and busywork are added until people embarrass themselves out of contention, or their health fails, or they leave to pursue better options, leaving one person standing. Software isn’t different from that, with the long hours and posturing machismo and general disregard for health.

By believing in meritocracy, software engineers trick themselves into making excuses for awful companies and bad bosses that hurt their careers, destroy their confidence, and unapologetically exploit them. When they enter organizations, they tend (at least, when young) to want to believe in the self-professed “meritocracy”, and it’s hard to let such an idea go even in the face of adverse evidence. When these engineers are betrayed, it’s practically an ambush.

Older, savvier engineers know that few workplaces are meritocracies. In general, the claim of “meritocracy” is nothing more than a referendum on the leadership of the company. For this reason, it’s only in the midst of an open morale crisis (in which firing the obviously unhappy people isn’t viable because almost everyone is obviously unhappy) that one can admit to the organization’s non-meritocracy.

The expensiveness of it all

Software engineers’ belief in meritocracy costs them money and career advancement. By conflating their organizational position (low, usually) with personal merit, their confidence falls to zero. Computer programming, if marketed properly, ought to be “the golden skill” that allows a person unlimited mobility within industry. However, we’ve allowed the businessmen who’ve colonized us to siloize us with terms like DBA, operations, data scientist, etc., and use those to deny opportunities, e.g. “you can’t take on that project, you’re not a real NLP programmer”. As a class, we’ve let these assholes whittle our confidence down to such a low level that our professional aura is one either of clueless youth or depressive resignation. When they beat us down, we tend to blame ourselves.

Our belief in meritocracy hurts us in another way, in that we justify things being unduly hard on us. We hate the idea of political promotion. Perhaps, on first principles, we should. What this means is that engineers are promoted “confirmationally” rather than “aspirationally”. In HR-speak, confirmational promotion means that they’re given formal recognition (and the organizational permission to operate at the level they have been) once they’re already working at the level signified by the title. Aspirational promotion means that people are promoted based on potential, but this opens the door for a host of clearly political promotions. On paper, confirmational promotion is superior, if infuriatingly slow. (It requires people to blow off their assigned duties and to take unrequested risks.) Engineers, of course, prefer confirmational regimes. And what’s wrong with that?

Engineers don’t like to negotiate, they don’t like politics, and they’re against favoritism. Most have a proud self-reliance that would leave them uncomfortable even if personally favored. They’re also, in general, irreverent toward title as long as they believe they’re fairly paid. To them, confirmational promotion is right. The problem? Everyone but engineers is promoted aspirationally. Engineers need long, completed, successful projects to get bumped to the next level. What, pray tell, does it take to become VP of Product or Senior Manager as opposed to Manager, or to rise on just about any of the nontechnical tracks, in most tech companies? Absolutely nothing. There is no fucking magic there. You have to convince someone to “see something” in you. That is, you have to play politics.

To the engineer’s chagrin, playing politics comes easily for most ambitious people. It sure isn’t rocket science. Getting over one’s own moral objections is, for most people, the hardest part. The result of this is that nontechnical tracks, including management tracks that often cross over engineers, are characterized by easy aspirational promotion driven by favoritism and politics. The “meritocratic” engineering track is clearly much more difficult. There are people over 35, with IQs over 140, who haven’t made “senior engineer”, for fuck’s sake. (At a “mere” 125 IQ, you’re smarter than 97% of the nontechnical VPs at most tech companies.) It’s characterized by confirmational promotion, instead. And this is a point of pride for software engineers: it’s really hard to climb the ladder, because one is competing with the smartest people in the organization, and because while favoritism exists, political promotions are much rarer on the engineering track than on non-technical tracks (largely because promotions in general are rarer).

This is something that software engineers don’t really get. What do job titles actually mean in organizations? Companies will say that “Director” means one thing and “VP” means another, with some jargon about “the big picture” and a person’s responsibilities within the organization. The truth is that they mean very little, other than serving as political tokens that prove the person was able to get them. “Director” means, “he was able to negotiate a salary between $X and $Y from HR”. Not more.

Where it leads

If you ask an engineer whether he thinks he’s ready to be VP of Engineering or CTO, you’ll get a half-hearted, self-deprecating answer. “You know, I might be ready to lead a small team, but I’m not sure I’m at the VP/Eng level yet.” Cluelessly, he believes that “the VP/Eng level” exists objectively rather than politically. On the other hand, if you ask a nontech the same question, he’ll take it without hesitation. Even if he’s terrible at the job, he gets a generous severance (he’s a VP) and will fail up into a better job. The relevant concept here is the effort thermocline, or the level in an organization where jobs stop being harder with increasing rank, but become easier (although, more political). It can be politically difficult to get a job above the effort thermocline, but it’s ridiculously easy to keep it. At that point, one has power and credibility within the organization sufficient that one cannot, personally, fail due to a lack of effort.

Nontechs, except for clueless people in their 20s who haven’t figured out what they want to do, go into work with one purpose: to get promoted beyond the effort thermocline. That’s not to say that they’re all unambitious or lazy. They’re just realistic about how the game works. Even if you want to work hard, you don’t want hard work to be expected of you. If you’re an SVP and you show up for work every day and put in an honest effort, you get credit for it. If you’re a worker bee, you get nothing for your 8-or-more hours per day. It’s just what you’re expected to do.

Above the effort thermocline, promotion is political, and people stop pretending otherwise. When you get “into the club”, you’re permitted to speak frankly (and hear frank speech) about how the organization actually works. The issue with the engineer’s mind is that it clings to a belief in right and wrong. It’s moralistic. It struggles to accept what people really are. Engineers don’t want to contend with the basic fact of most organizations, which is that they’re politically corrupt and dysfunctional, because most people are lazy, greedy, and weak. I’d likewise argue that this is connected to the low levels of acquired social skills in people like software engineers. It’s not a neurological disability for most. They never learn to read cues beyond a subconscious and juvenile level, because they hate what they see, which is that humans are mostly defective and that many are horrible.

Engineers don’t like the concept of the effort thermocline, or of political promotion in general. As much as they can, they’d refuse to have it within their ranks. I’d tend to side with the engineers. Who wouldn’t, from first principles, prefer a meritocracy over a political rat’s nest? The business responds by turning off political promotions for most engineers– while the rest of the organization continues to get them. The result is that, while they start off well in terms of pay and occupational dignity, engineers are being surpassed by the nontechs (who gleefully accept political promotions and feel none the worse for it) by age 30 and, by 40, are undervalued and way underpaid relative to their worth to their companies.

Engineering tracks in organizations are notoriously title-deflating, in comparison to the rest of the business world. Most software engineers would be appalled by how little talent and work ethic are required to become a non-technical VP at even the most esteemed tech companies. Many of these people are lazy (11-to-3 with 90-minute lunches) and just plain dumb. And by dumb, I don’t mean programmer dumb (understands the theory behind neural networks, but has never put one in production) but actual shame-to-the-family, village-idiot stupid. You know how towns in the Midwest used to bus their “defectives” to San Francisco in the mid-20th century? Well, so does the corporate world, and they end up as nontechs and upper management in tech companies.

Conclusion?

Meritocracy is the Prince Charming of the software engineer. It doesn’t exist. It never has, and it never will. Some have asked me to comment on recent HR issues occurring at open-allocation technology companies. The only thing I can say is that, yes, open-allocation companies have serious political issues; but closed-allocation companies have those same issues and more. Open allocation is strictly superior, but not a panacea. When there are people, there is politics. The best an organization can do is to be fair and open about what is going on, and hope to achieve eventual consistency.

Every organization defines itself as a meritocracy, and most engineers (at first, until they are disillusioned with a company) will tend to believe it. They aren’t stupid, so they don’t believe their companies to be perfect in that regard, but they (cluelessly) tend to believe that meritocracy is a core value of the leadership. Almost never is that the case. “We’re a meritocracy” is code for, “don’t question promotions around here”.

The Prince Charming delusion of meritocracy is dangerous because it leads people to make excuses for bad actors. Every company has to lay off or fire people, and frequently these choices are made with imperfect information and under time pressure (one large layoff is less damaging to morale than several small, measured, layoffs) so often the wrong people are let go. A self-aware organization understands this and lets them go gracefully: with severance, outplacement assistance, and positive reference. A delusional “meritocracy” has to cook the books, create psychotic policies that impede internal mobility for everyone, and generate useless process in order to build phony performance cases. In practice, just as many people are let go as in established (and less delusional companies) but their reputations have to be demolished first, with bad project assignments and hilariously disingenuous “performance improvement plans“. Personally, I’d rather see the honest, times-are-tough, layoff than the tech company’s dishonest “low performer initiatives”, much less the permanent (and destructive) rolling layoff of stack ranking.

The biggest casualty, however, of the typical engineer’s head-in-sand attitude toward political promotion is that they never stop happening to everyone else. Engineers just make themselves ineligible. Engineers want promotion to be confirmational (that is, resulting from demonstrated merit) rather than aspirational (that is, based on potential and, therefore, subjective, personal, and political). The problem with this is that, after 10 to 20 years, most engineers haven’t been able to demonstrate even 20% of what they’re capable of. They kept getting crappy projects, were never allowed to finish anything, were rushed to produce work that broke under strain, and their lack of finished accomplishment (due to political forces often not their fault) left them ineligible for promotion to more senior roles, but too old to even pretend in the junior roles (hence, the age discrimination problem). After that gauntlet of false starts and misery, they’re still answering to nontechnical people and executives who had the benefit of aspirational, political promotion. By refusing to play politics and believing in the false god of meritocracy, they deprived themselves of the full spectrum of causes for advancement. Politics, however, went on regardless of whether they believed in it.

This false meritocracy is very clever when it comes to reinventing itself. Few expect a large company like, say, Alcoa or Exxon-Mobil to be a meritocracy. Engineers have figured out, as a group, that “big companies” become political. The response? Startups! Venture capital! The actual result of this has been to replace well-oiled and stable (if inefficient) corporate non-meritocracies with the mean-spirited and psychotic non-meritocracy of the VC-funded ecosystem and the feudalistic reputation economy that the leading investors, through collusion, self-dealing, and note-sharing, have created. The cheerleading of intellectually charismatic figures like Paul Graham and Marc Andreessen has managed to create a sense of meritocracy in that world, but belief in those idols also seems to be waning, and I’m proud to say that I contributed to that loss of faith.

If meritocracy is impossible, what should we do? As individuals, we need to learn to fight for ourselves. It’s not undignified or desperate or “pushy” to look out for our own interests. It’s what everyone else is doing, and we should get on board. As a collective, we need to have honest introspection on what we value and how best to achieve it. Perfect meritocracy within any organization is impossible. It is good to strive for that, but bad to believe it has been achieved anywhere. Eventual consistency and technical excellence are achievable, and we should aim for those.

Before we do anything, though, we need to learn how to fight for ourselves. Bringing frank knowledge to the good, in that fight, is what I’ve been striving to do all along.

VC-istan 8: the Damaso Effect

Padre Damaso, one of the villains of the Filipino national novel, Noli me Tangere, is one of the most detestable literary characters, as a symbol of both colonial arrogance and severe theological incompetence. One of the novel’s remarks about colonialism is that it’s worsened by the specific types of people who implement colonial rule: those who failed in their mother country, and are taking part in a dangerous, isolating, and morally questionable project that is their last hope at acquiring authority. Colonizers tend to be people who have no justification for superior social status left but their national identity. One of the great and probably intractable tensions within the colonization process is that it forces the best (along with the rest) of the conquered society to subordinate to the worst of the conquering society. The total incompetence of the corrupt Spanish friars in Noli is just one example of this.

In 2014, the private-sector technology world is in a state of crisis, and it’s easy to see why. For all our purported progressivism and meritocracy, the reality of our industry is that it’s sliding backward into feudalism. Age discrimination, sexism, and classism are returning, undermining our claims of being a merit-based economy. Thanks to the clubby, collusive nature of venture capital, to secure financing for a new technology business requires tapping into a feudal reputation economy that funds people like Lucas Duplan, while almost no one backs anything truly ambitious. Finally, there’s the pernicious resurgence of location (thanks to VCs’ disinterest in funding anything more than 30 miles away from them) as a career-dominating factor, driving housing prices in the few still-viable metropolitan areas into the stratosphere. In so many ways, American society is going back in time, and private-sector technology is a driving force rather than a counterweight. What the fuck, pray tell, is going on? And how does this relate to the Damaso Effect?

Lawyers and doctors did something, purely out of self-interest, to prevent their work from being commoditized as American culture became increasingly commercial in the late 19th century. They professionalized. They invented ethical rules and processes that allowed them work for businessmen (and the public) without subordinating. How this all works is covered in another essay, but it served a few purposes. First, the profession could maintain standards of education so as to keep membership in the profession as a form of credibility that is independent of managerial or client review. Second, by ensuring a basic credibility (and, much more important, employability) for good-faith members, it enabled professionals to meet ethical obligations (i.e. don’t kill patients) that supersede managerial or corporate authority. Third, it ensured some control over wages, although that was not its entire goal. In fact, the difference between unionization and professionalization seems to be as follows. Unions are employed when the labor is a commodity, but ensure that the commoditization happens in a fair way (without collective bargaining, and in the absence of a society-wide basic income, that never occurs). Unions accept that the labor is a commodity, but demand a fair rate of exchange. Professionalization exists when there is some prevailing reason (usually an ethical one, such as in medicine) to prevent full commoditization. If it seems like I’m whitewashing history here, let me point out that the American Medical Association, to name one example, has done some atrocious things in its history. It originally opposed universal healthcare; it has received some karma, insofar as the inventively mean-spirited U.S. health insurance system has not only commoditized medical services, but done so on terms that are unfavorable to physician and patient both. I don’t mean to say that the professions have always been on the right side of history, because that’s clearly not the case; professionalization is a good idea, often poorly realized.

The ideal behind professionalization is to separate two senses of what it means to “work for” someone: (1) to provide services, versus (2) to subordinate fully. Its goal is to allow a set of highly intelligent, skilled people to deliver services on a fair market without having to subordinate inappropriately (such as providing personal services unrelated to the work, because of the power relationship that exists) as is the norm in mainstream business culture.

As a tribe, software professionals failed in this. We did not professionalize, nor did we unionize. In the Silicon Valley of the 1960s and ’70s, it was probably impossible to see the need for doing so: technologists were fully off the radar of the mainstream business culture, mostly lived on cheap land no one cared about, and had the autonomy to manage themselves and answer to their own. Hewlett-Packard, back in its heyday, was run by engineers, and for the benefit of engineers. Over time, that changed in the Valley. Technologists and mainstream, corporate businessmen were forced to come together. It became a colonial relationship quickly; the technologists, by failing to fight for themselves and their independence, became the conquered tribe.

Now it’s 2014, and the common sentiment is that software engineers are overpaid, entitled crybabies. I demolished this perception here. Mostly, that “software engineers are overpaid” whining is propaganda from those who pay software engineers, and who have a vested interest. It has been joined lately by leftist agitators, angry at the harmful effects of technology wealth in the Bay Area, who have failed thus far to grasp that the housing problem has more to do with $3-million-per-year, 11-to-3 product executives (and their trophy spouses who have nothing to do but fight for the NIMBY regulations that keep housing overpriced) than $120,000-per-year software engineers. There are good software jobs out there (I have one, for now) but, if anything, relative to the negatives of the software industry in general (low autonomy relative to intellectual ability, frequent job changes necessitated by low concern of employers for employee career needs, bad management) the vast majority of software engineers are underpaid. Unless they move into management, their incomes plateau at a level far below the cost of a house in the Bay Area. The truth is that almost none of the economic value created in the recent technology bubble has gone to software engineers or lifelong technologists. Almost all has gone to investors, well-connected do-nothings able to win sinecures from reputable investors and “advisors”, and management. This should surprise no one. Technology professionals and software engineers are, in general, a conquered tribe and the great social resource that is their brains is being mined for someone else’s benefit.

Here’s the Damaso Effect. Where do those Silicon Valley elites come from? I nailed this in this Quora answer. They come from the colonizing power, which is the mainstream business culture. This is the society that favors pedigree over (dangerous, subversive) creativity and true intellect, the one whose narcissism brought back age discrimination and makes sexism so hard to kick, even in software which should, by rights, be a meritocracy. That mainstream business world is the one where Work isn’t about building things or adding value to the world, but purely an avenue through which to dominate others. Ok, now I’ll admit that that’s an uncharitable depiction. In fact, corporate capitalism and its massive companies have solved quite a few problems well. And Wall Street, the capital of that world, is morally quite a bit better than its (execrable) reputation might suggest. It may seem very un-me-like to say this, but there are a lot of intelligent, forward-thinking, very good people in the mainstream business culture (“MBA culture”). However, those are not the ones who get sent to Silicon Valley by our colonial masters. The failures are the ones sent into VC firms and TechCrunch-approved startups to manage nerds. Not only are they the ones who failed out of the MBA culture, but they’re bitter as hell about it, too. MBA school told them that they’d be working on $50-billion private-equity deals and buying Manhattan penthouses, and they’re stuck bossing nerds around in Mountain View. They’re pissed.

Let me bring Zed Shaw in on this. His essay on NYC’s startup scene (and the inability thereof to get off the ground) is brilliant and should be read in full (seriously, go read it and come back to me when you’re done) but the basic point is that, compared to the sums of money that real financiers encounter, startups are puny and meaningless. A couple quotes I’ll pull in:

During the course of our meetings I asked him how much his “small” hedge fund was worth.

He told me:

30 BILLION DOLLARS

That’s right. His little hedge fund was worth more money than thousands of Silicon Valley startups combined on a good day. (Emphasis mine.) He wasn’t being modest either. It was “only” worth 30 billion dollars.

Zed has a strong point. The startup scene has the feeling of academic politics: vicious intrigue, because the stakes are so small. The complete lack of ethics seen in current-day technology executives is also a result of this. It’s the False Poverty Effect. When people feel poor, despite objective privilege and power, they’re more inclined to do unethical things because, goddammit, life owes them a break. That startup CEO whose investor buddies allowed him to pay himself $200,000 per year is probably the poorest person in his Harvard Business School class, and feels deeply inferior to the hedge-fund guys and MD-level bankers he drank with in MBA school.

This also gets into why hedge funds get better people (even, in NYC, for pure programming roles) than technology startups. Venture capitalists give you $5 million and manage you; they pay to manage. Hedge fund investors pay you to manage (their money). As long as you’re delivering returns, they stay out of your hair. It seems obvious that this would push the best business people into high finance, not VC-funded technology.

The lack of high-quality businessmen in the VC-funded tech scene hurts all of us. For all my railing against that ecosystem, I’d consider doing a technology startup (as a founder) if I could find a business co-founder who was genuinely at my level. For founders, it’s got to be code (tech co-founder) or contacts (business co-founder) and I bring the code. At my current age and level of development, I’m a Tech 8. A typical graduate from Harvard Business School might be a Biz 5. (I’m a harsh grader, that’s why I gave myself an 8.) Biz 6 means that a person comes with connections to partners at top VC firms and resources (namely, funding) in hand. The Biz 7′s go skiing at Tahoe with the top kingmakers in the Valley, and count a billionaire or two in their social circle. If I were to take a business co-founder (noting that he’d become CEO and my boss) I’d be inclined to hold out for an 8 or 9, but (at least, in New York) I never seemed to meet Biz 8′s or 9′s in VC-funded technology, and I think I’ve got a grasp on why. Business 8′s just aren’t interested in asking some 33-year-old California man-child for a piddling few million bucks (that comes along with nasty strings, like counterproductive upper management). They have better options. To the Business 8+ out there, whatever the VCs are doing in Silicon Valley is a miserable sideshow.

It’s actually weird and jarring to see how bad the “dating scene”, in the startup world, is between technical and business people. Lifelong technologists, who are deeply passionate about building great technology, don’t have many places elsewhere to go. So a lot of the Tech 9s and 10s stick around, while their business counterparts leave and a Biz 7 is the darling at the ball. I’m not a fan of Peter Shih, but I must thank him for giving us the term “49ers” (4′s who act like 9′s). The “soft” side, the business world of investors and well-connected people who think their modest connections deserve to trade at an exorbitant price against your talent, is full of 49ers– because Business 9′s know to go nowhere near the piddling stakes of the VC-funded world. Like a Midwestern town bussing its criminal element to San Francisco (yes, that actually happened) the mainstream business culture sends its worst and its failures into the VC-funded tech. Have an MBA, but not smart enough for statistical arbitrage? Your lack of mathematical intelligence means you must have “soft skills” and be a whiz at evaluating companies; Sand Hill Road is hiring!

The venture-funded startup world, then, has the best of one world (passionate lifelong technologists) answering to the people who failed out of their mother country: mainstream corporate culture.

The question is: what should be done about this? Is there a solution? Since the Tech 8′s and 9′s and 10′s can’t find appropriate matches in the VC-funded world (and, for their part, most Tech 8+ go into hedge funds or large companies– not bad places, but far away from new-business formation– by their mid-30s) where ought they to go? Is there a more natural home for Tech 8+? What might it look like? The answer is surprising, but it’s the mid-risk / mid-growth business that venture capitalists have been decrying for years as “lifestyle businesses”. The natural home of the top-tier technologist is not in the flash-in-the-pan world of VC, but the get-rich-slowly world of steady, 20 to 40 percent per year growth due to technical enhancement (not rapid personnel growth and creepy publicity plays, as the VCs prefer).

Is there a way to reliably institutionalize that mid-risk / mid-growth space, that currently must resort (“bootstrapping”) to personal savings (a scarce resource, given that engineers are systematically underpaid) just as venture capital has done to the high-risk /get-big-or-die region of the risk/growth spectrum? Can it be done with a K-strategic emphasis that forges high-quality businesses in addition to high-value ones? Well, the answer to that one is: I’m not sure. I think so. It’s certainly worth trying out. Doing so would be good for technology, good for the world, and quite possibly very lucrative. The real birth of the future is going to come from a fleet of a few thousand highly autonomous “lifestyle” businesses– and not from VC-managed get-huge-or-die gambits.

VC-istan 7: solving the wrong problem

I’ve written at length about VC-istan, its poor performance and its bigotries. What, however, is VC-istan’s “original sin”? Why is it so dysfunctional? Is there a foundational reason for its pattern of across-the-board moral and financial failure? I think the answer is obviously, “yes”. There’s a simple root cause: it’s solving the wrong problem. This requires two investigations: what problem should venture capital be solving, and what is it actually doing?

What’s the purpose of venture capital?

This is an easy one. The purpose of venture capital is to finance endeavors that require substantial backing in an all-or-nothing transaction. A biotechnology firm that requires $100 million to develop, and put into clinical trial, a new drug or device would be one example of this. With $10 million, it produces nothing salable; with ten times that, it has a chance. Others exist around infrastructure and in more deeply industrial pursuits like clean energy. Venture capitalists do invest in these spaces, and that’s outside of what I’d call “VC-istan”. Not everything that venture capitalists do is ugly, of course, and not all of it is VC-istan.

Venture capital, in a way, was originally intended as the “capital of last resort” for high-risk, capital-intensive businesses that would never qualify for more traditional financing. Why? Because when the proper way to invest is all-or-nothing, that has (unavoidable) negative consequences for all sides. It means that most people won’t get funded, and it’ll be extremely competitive to get capital, and dilution of founder equity will be severe. It’s not ideal, but if all you have is an idea, your product is 3 years away from the market in the best-case scenario, and you’re asking for $125 million to get started, those are the terms you have to take. It is, of course, quite a noisy process. The best ideas might not get funded, because there is literally no one able to assess what the best ideas are.

Venture capital for biotechnology and infrastructure has its own rules and culture. I’m not an expert on that, but it’s not what I consider “VC-istan”. From my perspective, which may be limited, venture capitalists in that space are trying to act in good faith and invest in viable businesses. To be blunt, I don’t think the “cool kids” nonsense (see: TechCrunch) matters so much in those sectors, because the science has to be sound. If you’re trying to turn algae into diesel fuel, Mike Arrington’s half-baked opinion of you matters a billion times less than the chemistry inside your lab.

What problem is VC-istan solving?

VC-istan is a subset of “all venture capital”, and focused on the “hip” stuff that can be likened to “reality TV”. To explain this analogy, ask this question: why are “reality TV” shows so prolific? It’s not about their quality. Cheap production is often cited, but it’s not just amount the numbers in the accounting ledger. The reality show formula is one that admits perfect commoditization. Writers and actors, at high levels of talent, resist commoditization. They won’t work on shows that are bad for their careers, and they have agents whose full-time job is to represent their interests. This makes them non-fungible. At the highest level of talent, labor may be able to push back against commoditization of itself, because there are few enough people at the highest levels to make the market discrete rather than continuous– or, in other words, illiquid. Reality TV does away with those “prima donna” creatives and celebrities: the writing demands are minimal and can be fulfilled with a mediocre staff, and the actors are nonentities. This enables the production studio to iterate quickly with half-baked concepts without needing to concern itself with the career needs of the parties involved.

VC-istan loves social media, and consumer-web marketing experiments, which are like reality television in that they can be produced with mediocre, “commodity-grade” inputs. To launch a biotech firm, one actually needs to have a strong grounding in science. Assessing founders for scientific literacy is hard, and private equity people are rarely up to the task. But any idiot can come up with, and hire someone good enough to implement, a Snapchat or a Clinkle. In the soft space of marketing experiments using technology, as opposed to the much harder sector that is technology proper, commodity founders and engineers suffice, and because the space is a gigantic bike shed, every investor feels entitled to have strong opinions. If genuine technical talent is needed for “scaling” down the road, it can be hired once the company has been covered by TechCrunch and appears legitimate.

Ultimately, the purpose of VC-istan’s “tech” companies is not to innovate or to solve hard problems. It’s to flip teams that have been validated by three to six rounds of venture funding, and possibly by success in the market (but that’s optional). Occasionally there’s an IPO, but those are about as common as big-company spinoffs. More typical is the “acqui-hire”, whose purpose can only be understood in the broader context of corporate dysfunction.

M&A has replaced R&D

A company’s need for top talent tends to be intermittent or subtle, and most often both. An example of the first (intermittent need) is around a short-term crisis that only a small percentage of people will have the insight, creativity, experience, or work ethic that is necessary to surmount it. The second pertains to the long-term existential need for innovation; if the company doesn’t have some engine that produces an occasional positive-impact black swan, it will be torn to shreds by the bad kind: crises that no amount of talent or effort can resolve. While every company pays lip service to its need for top talent, the truth is that most companies don’t need top talent for their day-to-day operations. If they did, that would be irresponsible design: a dependency on something that is somewhere between a highly volatile commodity and an outright non-commodity. The need for top talent tends to be a long-term issue.

Top talent is difficult to truly employ; one merely sponsors it. Old-style corporations understood that and invested in R&D. When the rare crisis that was truly existential would emerge, talent could be borrowed from the R&D pool. Additionally, while R&D could focus on basic research that was of general benefit to the world, and not necessarily in the firm’s immediate, parochial interests, the proximity to that research the corporation enjoyed gave it such an edge in practical innovation to pay for itself several times over.

Unfortunately, basic research was one of the first casualties of the private equity invasion that began in the 1980s. The old R&D labs that built C, Unix, Smalltalk and the internet weren’t scrapped outright, but reduced to a fraction of their former size, and forced to take a next-quarter focus. Conditions weren’t actually made so bad as to flush existing talent out, but positions became scarce enough that new talent couldn’t get in. The executives of those companies weren’t all short-sighted idiots, though. They knew that the high-autonomy, R&D-oriented work was the only thing keeping top talent in place. With corporate R&D near obliteration, that was threatened. So they knew they needed a solution that talent-intake problem. What did private iniquity propose as a solution? More private equity.

Enter venture capital, formerly a subsector of private equity that was generally avoided by those with other career options, due to its infuriatingly intermittent performance. What would it mean, however, if venture capital could be made less “venture”, by filling a need created by the disintegration of another part of the economy? Companies shutting down the blue-sky, high-autonomy R&D work had to get talent somehow. Explicitly paying for it proved to be too expensive, except in investment banking, due to hedonic adaptation– people who are performing at a high level, if their needs for autonomy are not met, require 25-50% per year raises to be content. Tapping high-talent people for managerial ranks proved fruitless as well, because many of these people (while exceptional as individual contributors) had neither the desire nor the ability to manage (and, additionally, middle-management positions were also cut during the private equity invasion). The remaining solution to the talent problem became one that private equity men found extremely attractive, given the premium they collect on deals– companies must buy it.

I don’t intend to insult the low-level employees of the Googles and Yahoos of the world by saying that those companies have “no talent” at the bottom. That’s clearly untrue. Companies don’t acqui-hire (which is far more expensive than internal promotion) because they have no top talent in their ranks. They have plenty, but they acqui-hire because they have lost the ability to discover what they have. It’s a malfunction of the middle-management layer. These companies are like hoarders that buy new coats every winter not for a lack of coats, but because their houses are so out of order that a new purchase is preferable to sorting the old place out.

Moreover, a company cannot, in general, adequately commoditize its highest levels of talent. The best will always seek their own career goals foremost, and perform at their highest only when there is coherency between their long-term personal goals and the work assigned to them. There are also, to put it bluntly, not enough such people to merit any explicit managerial correction to this problem. An executive focused on the career-coherency issues coming out of the most talented 5% is ignoring the day-to-day work completed by the other 95%. Two (problematic) solutions end up emerging. The first is for the company to ignore the high-talent problem and treat its top 5% like everyone else: closed allocation, low autonomy, etc. Then it loses them, plain and simple, and becomes dysfunctional after a few years of brain drain. The second is to leave them alone and effectively let them work on whatever they want. That’s great, in the short term, but it can be politically messy; others (who may belong in the top 5%, but haven’t been recognized) may resent them for their higher level of autonomy, or that set of people may lose sight of their need to continually market themselves and justify their favorable conditions, and then be crushed (not for a lack of value to the organization, but because it fails to market itself) when there is a management or market change.

So what is the “problem” that VC-istan exists to solve? It’s there to commoditize top talent. Although a specific company cannot commoditize its top 5%, the conceit is that an army of dedicated specialists– a mix of investors, corporate biz-dev executives, and “tech press”– can do so. In the consumer web space, venture capitalists have become a sort of high-end headhunter, but one that follows different rules.

For one major difference between the old corporate ladder and the acqui-hire system, employers are not allowed to explicitly discriminate on age, pregnancy status, health issues, race or gender. Investors can. Middle managers are too busy to conduct invasive “back channel” reference checks that, in truth, constitute civil harassment and would admit blacklisting charges if they ever interfered with employment (thus, risk-averse companies prefer not to do so). Investors can do so, and in such a way as to work through people who will keep their secrets (preventing lawsuits). This is a wet dream of the new right wing, an Uberization of executive hiring. The old system, with decades of regulation thrown into it because those rules were actually necessary, has been supplanted by a premium, rule-breaking, and vicious new one. The people who need the regulations imposed by the old system (i.e. women, minorities, people with health problems, people over 40, people with kids) are simply judged unfit to compete.

Here’s a question: how well is VC-istan actually doing, on its own terms? The first question is: what does it mean to “commoditize top talent”? While that sounds like something I might be against, I can’t actually say it’s a bad thing– not even for top-talent people. When something is commoditized, a fair price (that may fluctuate, but is fair relative to published market conditions) is established and it’s very easy to buy or sell it near that price. Currently, the compensation for top (2.0-level) engineering talent swings between about $75,000 and $10+ million per year– there is a clear uncertainty about what it is worth– with a median around $150,000. If that level of talent were adequately and fairly commoditized, that range would be more like $300,000 to $500,000– which would give most of them a hefty pay bump. The truth about the commoditization of labor is that labor generally finds it unobjectionable when the terms are fair. In fact, one effect of labor unions is to explicitly commoditize labor while attempting to ensure fairness (while professions, in general, oppose the commoditization regardless of terms). The murky issue in technology is that “top talent” is very hard to detect, because the people with the requisite skill have better things to do. Those who can, do; those who can’t, evaluate others’ work.

VC-istan, then, is built on the record-company model. Founders and engineers are treated as commodities (and generally, for reasons I won’t get into here, don’t get fair terms) but there is a hope that, thanks to the law of large numbers, top talent will be detected and validated by the outside market.

Where VC-istan went wrong is that it never figured out what top talent might look like, so the resources were thrown behind those who were either best at self-promotion or (increasingly, over time) those who could pull inherited connections. As a mechanism for detecting the rising generation’s top marketing talent, it might not be doing so bad. For picking out the best technical talent, especially as pertains to long-term R&D, it’s worse than abysmal. It’s doubtful that it’s picking up any signal at all. Companies that have a genuine need for R&D talent will be poorly served if they source it through acqui-hires.

VC-istan exists to commoditize top talent, but it has also erected a feudalistic reputation economy in which investors hold the cards. Founders hold few, engineers hold none. The highest levels of technical talent have been rendered, by this new economy, effectively irrelevant, depriving it of any leverage whatsoever. So, the terms are made bad– so bad that top engineering talent is rarely delivered. Whether this will strip VC-istan of credibility in the long run is something that remains to be seen.

The point I’ve made here is that it’s “solving” an ugly problem in a bad way.

What can venture capital do for technology?

Venture capital’s purpose is to build companies that, if successful, will become massive corporate behemoths. On a fundamental level, it’s stuck in the 20th-century mentality where a gigantic organization is the only acceptable prize for winning. Startup life is sold (by founders, and rarely by investors directly) to talented, usually clueless, engineers as an antidote to the ills of “MegaCorp” when, in truth, the explicit purpose of the VC-funded startup is to become exactly that: a new MegaCorp, but usually with crappier health benefits, longer hours, and faster firing.

What the best engineers actually tend to want is high autonomy so they can deliver exceptional work. They’d prefer ownership over it, all else being equal, but as long as they’re fairly compensated, they’re generally happy whether they work for a 20,000-person company or for themselves. When corporate R&D was sold for parts, venture-funded startups were proposed as the solution, the new way forward. Don’t like what happened to your old job? Create a new job for yourself! The lie here is that founding a VC-funded company provides the autonomy associated with true ownership. In truth, venture capitalists become full owners (de facto, if not de jure, due to the power afforded them by VC’s feudal reputation economy) of the company even when they hold a minority stake. Working for VCs is not fundamentally better than working for a boss; in many ways, it’s worse because the social distance is greater. Most bosses don’t consider themselves inherently superior based on favorable birth;

There are several critical misses that have become evident as venture capital has attempted to replace more traditional venues for innovation. One is that it has proven not to be a valid replacement for internal R&D. Nothing that VC-istan has coughed up is anywhere near the order of magnitude of Bell Labs or Microsoft Research. The second is that it has failed to be an engine of small-business generation, which is necessary for economic growth. It hasn’t connected top talent with the autonomy that comes from ownership. Rather, it has abandoned top talent in the pursuit of commodity startups run by commodity founders and commodity engineers. Over time, one might hope top talent to abandon it. That trend seems to be emerging, but I have no idea when or how (or, even at this stage, if) it will mature.

There is a fundamental technical flaw with VC-istan, additionally. That I’ll focus on, because it might lead us in the direction of a solution. If we consider the risk/reward profile of businesses, we see an underserved middle of the spectrum. Low-risk businesses can take bank loans, but those require personal liability, so it’s not wise to use them for anything that might actually fail. High-risk gambits with above-90% chances of failure, but that are capable of returning 20-50x on success, are what VCs love. The mid-risk/mid-growth space– targeting 15 to 50% annual growth, with a low but nonzero chance of business failure– is inappropriate for bank loans (too risky) but unpalatable to venture capitalists (not risky enough). Unfortunately, I don’t see an easy fix for that. Venture capital could become very profitable by funding the 15-50% range, but investment decisions aren’t driven by profits so much as the career needs of the investors. Returning a steady profit (say, 25% per year, with a bit of variance) by investing in a number of solid but moderately-sized businesses is not career-making; having been in on Facebook (even as a minor and late investor) is. The name-dropping world of Sand Hill Road cannot be expected to change, and if it does not, the focus will be less on building quality businesses and more on taking insane risks in the hope of hitting a career-making blockbuster.

This is problematic because the mid-growth/mid-risk space is exactly where true technologists live. They do not become machine learning experts or compiler gurus in an overnight episode of “virality”, and whether Mike Arrington or Paul Graham owes them a favor is irrelevant to whether they can actually code. They get good (and, if possible, rich) slowly. In terms of abstract value-added capacity, 15 to 50% per year seems to be about the natural rate (although most engineers would be thrilled to have salary increases at even half that rate). Technologists are extremely good at delivering these 20 and 40 percent per year improvements. What lies outside their interest (and, usually, their ability) is engineering the social conditions that admit 100x “viral” growth (or, far more often, abysmal failure). It’s just not where they live; they weren’t born in casinos.

The future

VC-istan is not about to die, any more than recording labels have ceased to exist. As a method of shaving 15 years off a rich kid’s corporate-ladder climb via “acqui-hire”, it will persist. As a machine that produces commodity startups run by commodity entrepreneurs, it will persist and probably be profitable for quite some time. As a way of enabling companies to discriminate on age, health, pregnancy status, and other illegal factors at upper levels (filled through acqui-hires, while rendering internal promotion rare) while keeping the discrimination off their books, it will hold that niche quite well. How relevant will VC-istan remain to true top talent? On that one, VC-istan’s lifespan may be limited. In that territory, it’s “ripe for disruption”.

So what shall be built to bring the disruption?

Software engineer salaries aren’t inflated– at least, not for the 99%

It’s autumn 2013, and there’s a lot of discussion around the current bubble (now obviously one) in the VC-funded technology world and how it will end. Business Insider acknowledges that a bubble exists, but gets some crucial details wrong. Let’s talk about one that most of us actually care about. Business Insider claims: “It’s not just tech asset prices that are high. Salaries are high, too.” Them’s fighting words. Is it true? Well, sort-of. Here’s the evidence, from tech recruiter Matt Allen:

Instead, we’re seeing sign-on bonuses for individuals five-years out of school in the $60,000 range. Candidates queuing-up six, eight or more offers and haggling over a few thousand-dollar differences among the offers. Engineers accepting offers and then fifteen minutes before they’re supposed to start on a Monday, emailing (not calling) to explain they found something better elsewhere.

Ok, let’s dissect this. One: a few people (and it’s not clear that they’re engineers) are getting huge signing bonuses. $60,000 isn’t a number to sneeze at, but it’s not that extreme. Management-level hires typically get signing/relocation bonuses that cover the cost of a cross-country move (easily over $20,000, for people with families) and there’s no reason software engineers shouldn’t get the same. Additionally, signing bonuses usually have clawback provisions if the employee leaves (even involuntarily) in the first year, penalizing the job-hopping for which the worst of our generation is known. Given the tax penalty associated with receiving a bonus and risking having to pay it back, I’m not sure I’d want a $60,000 bonus under typical terms. Two: some candidates are queuing up 6 to 8 job offers. I call bullshit on that one, if only because of the scheduling difficulties in a startup ecosystem where 7-day exploding offers are the norm. I’m sure there are people getting 6-8 offers in the course of an entire job search (I’ve had that) and that people are claiming to have portfolios of excellent offers in negotiation, but the logistics of getting 6 active, credible job offers at one time are unfavorable, to say the least. Three: people are being unprofessional dickbags, pulling out of accepted offers on their start date. I’m sure that that is happening, but how is an occasional episode in which a privileged young hotshot acts like a jackass newsworthy, much less the sign of a bubble? It’s not.

Managers and product executives are making a killing in the present-day startup economy, no doubt, and some of those people might be able to pass as programmers due to some PHP scripts they wrote in their teens, but when one actually studies the contemporary startup economy, there are not a lot of software engineers making over $200,000 per year outside of finance– and those who are tend to be either very good or unusually lucky. For a VC-funded startup to offer $200,000 to an engineer would be incredibly rare, even in the Bay Area, and equity allotments after VCs are involved are notoriously stingy.

Twenty years ago, when startups were underdogs almost by definition, the scene had a “Revenge of the Nerds” feel. A bunch of ragtag computer aficionados, typically from middle-class backgrounds and far away from the East Coast’s financial and corporate elite, were showing up the old guard. New, powerful technologies were being developed, and power shifted (temporarily, perhaps) to those few who understood them at a deep level. There was slight subversion of the 1%; they weren’t destroyed or even harmed, but they were visibly outperformed. For a contrast, the hot properties of the current VC-funded world almost entirely come from the 1%. Behind almost every single one of the VC darlings, there’s a series of strings pulled by powerful people repaying favors to the rich daddies of the founders. There’s no meritocracy in it. It’s not a challenge to the established and rich; it’s a sideshow for the supercapitalists. In a surprising reversal, the old-style corporate world (and the enterprise companies existing and being formed to serve their needs) has a much more middle-class culture, because the current-day rich find it boring.

Software engineer salaries in the VC-funded world are not especially low (nor are they high). They’re usually 75 to 95 percent of what more typical employers are offering. Equity distributions, on the other hand, are extremely lopsided. I worked for a company once where the board refused to allow more than 0.04% to go to an engineer. (Why? Because fuck the people doing the work, that’s why.) There’s something that needs to be discussed here, because it applies to the age-old question of why people who do actual work are modestly compensated, while vacuous celebrity types take the lion’s share. It’s the Teacher-Executive Problem.

The Teacher-Executive Problem

As a society, we need teachers, police officers, park rangers, and other such people who are modestly compensated. We don’t need celebrities, business executives, or professional athletes. I’m not going to argue that the latter are overpaid, insofar as it’s difficult and competitive to get to the top ranks in any field. That would be a subjective argument; all I intend to say is that, objectively, the need for the latter class of labor is smaller. If we didn’t have teachers or police, society would fall apart. If we didn’t have corporate executives, companies would find other ways to survive. So why are the more necessary people paid less? Because being necessary means that more workers will be drawn into the field, and that limits individual compensation. We probably pay more, as a society, for teachers and police than we do for corporate executives (as we should) but the individual slices for the larger, more necessary, job categories are smaller.

We have 3 million teachers in the US, and we need that large number of them, because individual attention per student is important. The functioning of society would be greatly impaired if that number dropped to 2 or 1 million. One might argue that competent teachers are “worth” $200,000 (or much more) per year– and I’d say that the best are worth several times that– but can society afford to pay that much for teaching? Three million $200,000 paychecks is a $600-billion annual liability. Taxes would go up substantially– in a time when the base of political power is (unfortunately) divided between a structurally disadvantaged (read: mostly fucked) emerging-adult cohort and retiring Boomers whose children are out of school– and society would likely determine that $200,000 annual paychecks for teachers “can’t be afforded” (especially given the claim that “they get off work at 3:00″). $200,000 isn’t a large amount of money for a single person, but for people who are actually needed in significant numbers, the multiplier of 3 million makes it seem unacceptable. (I am not arguing that teachers don’t deserve $200,000 salaries; only that it would be politically impossible to get them there.)

For a contrast, the social need for corporate executives (excluding entrepreneurs) is pretty minimal, and society recognizes this in a rational way: there aren’t a large number of slots: title inflation aside, there might be ten thousand truly executive roles in powerful companies. However, when the number of people performing a job is low, gigantic salaries (if those people control the distribution of resources) become socially affordable. Three million somewhat high salaries is a problem, ten thousand enormous ones is not. This is paradoxical because the middle-class conceit is that the way to become wealthy is to make oneself valuable (or, better yet, necessary) to society. What the Teacher-Executive Problem shows us is that there’s more potential for outlier compensation in doing things that aren’t necessary, because asking for more compensation doesn’t carry the implicit multiplier based on the size of the labor base. Society “can’t afford” to pay the 3 million teachers such high salaries, but it can afford the huge salaries of corporate executives, and the $850-million acquisition that enriches the top executives of IUsedThisToilet.com.

Why do so few software engineers get a fair shake in the VC-funded world? They’re on the wrong side of the Teacher-Executive Problem. They’re actually necessary. They’re required in order for technology firms to function.

What about 10X?

The generally accepted consensus (even among software engineers) is that average programmers aren’t very valuable. They write all that buggy, hideous legacy code. There’s little that software engineers and business executives agree upon, but the low status of the average programmer is probably not a point of disagreement. I don’t care to speculate on what the “average” software engineer is like, because while I have seen a ton of incompetents (and a smaller number of good engineers) out there in the world, I don’t have a representative sample. I also think that most of the engineering incompetence comes not from a lack of ability, but from an anti-intellectual culture originating in business culture at large, as well as nonexistent mentoring, so it’s not programmers who are mostly at fault. However, I will agree readily that the bulk of software engineers don’t deserve high ($200,000) salaries. They might have the talent, but few have that level of skill.

However, there is the concept of the “10x” software engineer, one who is 10 times as productive as an average engineer. It reflects a truth of software engineering, which is that excellence and peak productivity are tens to hundreds of times more powerful than the average-case output. (In fact, often that ratio is infinite because there are problems that require top talent to solve it.) Moreover, groups of engineers often scale poorly, so a team of 10 isn’t really (most of the time) 10 times as productive, but maybe 2 or 3 times as strong, as an individual. So it’s not surprising that a great engineer would be 10 times as valuable. The degree to which “10x” is real depends on the type of work, the context, project-person fit, and the competence of the engineer. It’s highly context-dependent, it’s not always the same people, and it’s quite unpredictable. The national average salary for a software engineer is about $90,000. The 10x-ers are not earning 10 times that and, to be honest about it, they probably shouldn’t. You can’t know, when hiring someone, whether the context that supports 10x output for that person is going to exist in the role. The bona fide 10x engineers typically earn 1.5 to 2 times that amount ($135,000 to $180,000) in the U.S. I’m not going to argue that they’re underpaid at this level– although, at least in comparison to MBA graduates earning twice that before age 30, I think they clearly are– but they’re far from overpaid at that level.

Why don’t 10x engineers get paid astronomical sums? For a large part, I think it’s because of the context-dependent nature of “10x”. It doesn’t require only a good engineer, but a good engineer connected with the right kind of work. Companies can’t afford (obviously) to pay $900,000 salaries to senior engineers just on the hunch that those (typically highly specialized) talents will find a use. When engineers do find environments in which they can deliver 10x output, they’re happy– and they’re not liable to clamor for huge raises, or to move quickly and risk starting over in a defective environment. This isn’t especially wrong; engineers would rather have interesting work at “merely” 1.5x salaries than risk happiness and growth for a chance at more. It’s just worth pointing out to establish that, in general, software engineers (and especially the most capable ones) are not overpaid. Moreover, the people commanding $500,000+ salaries in technology are typically not real engineers, but managers who might occasionally “drop down” and hack on one of the sexier projects to keep their skills sharp. Finally, the few (very few) software engineers making that kind of money are generally worth it: we’re not talking about top-1% talent at that level, but top-0.05% (and a level almost never achieved before age 40). There are plenty of people drawing undeserved high salaries in the Valley, but they aren’t the ones writing code.

Why must I point this out?

This (bubble) too, shall pass. The era when a well-connected rich kid can raise a $1-million seed round rather than eating his lumps in an investment bank’s analyst program will end. I don’t think that that’s a controversial assumption. Timing the end of a bubble is nearly impossible, and I don’t think anyone has shown reliability in that particular skill; but predicting that it will die off is trivial– they always do. When this happens, there will be a lot of job losses and belt-tightening. There always is. It’ll get ugly, and that’s fine. Most of these businesses getting funded and acquired don’t deserve to exist, and the economy will inevitably purge them. What I don’t want to see is the bubble made into an argument against the middle-class, hard-working software engineers. When the bubble ends, there will be losses to eat and austerity to go around, and it ought to go right to the people who reaped the benefits while the bubble existed. The end of the bubble should not be used to reduce the compensation of software engineers as a whole, whose pay is currently (I would argue) not quite unfair, but on the low side of the fair interval.

For the 99 percent, there is no software engineer salary bubble.

Dimensionality– and also, a theory about the rarity of female programmers.

People have invented a number of theories regarding the relative lack of women in the software industries. I don’t intend to opine on each one of those theories. I’m pretty sure that it’s not a lack of ability, because the women who are in software tend to be quite good, and seem to be more capable on average than men in the industry. I’ve met very good male and female programmers, but the bad ones tend to be men. Now, some people attribute the rarity of female programmers to the macho cultures existing in badly managed firms, and I think that that’s part of it, but I also see the negative cultural aspects of this field resulting from the gender imbalance, rather than the reverse. After all, women had to overcome cultures that were just as bigoted and consumptive (if not moreso) as the sort found in the worst software environments, in order to break into medicine and law; but eventually they did so and now are doing quite well. What’s different about software? I offer a partial explanation: social and industrial dimensionality. Perhaps it is a new explanation, and perhaps it’s a spin on an old one, but let’s dive into it.

In machine learning, dimensionality refers to a problem where there are a large number of factors that might be considered, making the problem a lot more complicated and difficult, because it’s usually impossible to tell which dimensions are relevant. Many problems can be framed as attempting to estimate a function (predicted value according to the pattern the machine is trained to learn) over an N-dimensional space, where N might be very large. For example, Bayesian spam filtering implements a K-dimensional regression (specifically, a logistic regression) where K is the number of text strings considered words (usually about 50,000) and the inputs are the counts of each.

A 10- or even 100-dimensional much space is larger than any human can visualize– we struggle with three or four, and more than 5 is impossible– but algebraic methods (e.g. linear regression) still work; at a million dimensions, almost all of our traditional learning methods fail us. Straightforward linear algebra can become computationally intractable, even with the most powerful computers available. Also, we have to be much more careful about what we consider to be true signal (worth including in a model) because the probability of false positive findings becomes very high. For some intuition on this, let’s say that we have a million predictors for a binary (yes/no) response (output) variable and we’re trying to model it as a function of those. What we don’t know is that the data is actually pure noise: there is no pattern, the predictors have no relationship to the output. However, just by chance, a few thousand of those million meaningless predictors will seem very highly correlated to the response. Learning in a space of extremely high dimensionality is an art with a lot of false positives; even the conservative mainstay of least-squares linear regression will massively overfit (that is, mistake random artifacts in the data for true signal) the data, unless there is a very large amount of it.

We see this problem (of overfitting in a space of high dimension) in business and technology. We have an extreme scarcity of data. There are perhaps five or six important, feedback-providing, events per year and there are a much larger number of potentially relevant decisions that lead to them, making it very hard to tell which of those contributed to the result. If 20 decisions were made– choice of programming language, product direction, company location– and the result was business failure, people are quick to conclude that all 20 of those decisions played some role in the failure and were bad, just because it’s impossible to tell, in a data-driven way, which of those decisions were responsible. However, it could be that many of those decisions were good ones. Maybe 19 were good and one was terrible. Or, possibly, all 20 decisions were good ones in isolation but had some interaction that led to catastrophe. It’s impossible to know given the extreme lack of data.

Dimensionality has another statistical effect, which is that it makes each point an outlier, an “edge case”, or “special” in a way. Let’s start with the example of a circular disk and define “the edge” to mean all points that are more than 0.9 radii from the center. Interior points, generally, aren’t special. Most models assume them to be similar to neighbors in expected behavior, and their territory is likely to be well-behaved from a modeling perspective. In two dimensions– the circular disk– 81% of the area is in the interior, and 19% is on the edge. Most of the activity is away from the edge. That changes as the number of dimensions grows. For a three-dimensional sphere, those numbers are 72.9% in the interior, and 27.1% at the edge. However, for a 100-dimensional sphere, well over 99% of the volume is on the edge, and only 0.9^100, or approximately 0.0027%, is in the interior. At 1000 dimensions, for all practical purposes, nearly every point (randomly sampled) will be at the edge. Each data point is unique or “special” insofar as almost no other will be similar to it in each dimension.

Dimensionality comes into play when people are discussing performance at a task, but also social status. What constitutes a good performance versus a bad one? What does a group value, and to what extent? Who is the leader, the second in charge, all the way down to the last? Sometimes, there’s only one dimension of assessment (e.g. a standardized test). Other times, there can be more dimensions than there are people, making it possible for each individual to be superior in at least one. Dimensionality isn’t, I should note, the same thing as subjectivity. For example, figure-skating performance is subjective, but it’s not (in practice) highly dimensional. The judges are largely in agreement regarding what characterizes a good performance, and disagree largely on (subjective) assessments of how each individual matched expectations. But there aren’t (to my knowledge, at least) thousands of credible, independent definitions of what makes a good figure skater. Dimensionality invariably begets subjectivity regarding which dimensions are important, i.e. on the matter of what should be the relative weights for each, but not all subjective matters are highly dimensional, just as perceived color is technically subjective but generally held to have only three dimensions (in one model; red, green, and blue).

Social organizations also can have low or high dimensionality. The lowest-dimensional organization is one with a strict linear ordering over the people. There’s a chief, a second-in-command, a third and fourth, all the way down to the bottom. If you’re 8th out of 18 people, you know that you’re not 7th or 9th. Social status has one legible dimension. Status is determined by one legible fact about a person: the specific assigned rank. Typical hierarchies aren’t that way; while there is a rigid status pyramid, same-rank people are not comparable against one another. In most organizations and groups, leaders are visible, and omegas might be legible because they serve a social purpose, too; but in the middle, it’s almost impossible to tell. Venkatesh Rao goes into a lot of detail on this, but the general rule is that every social group will have a defined alpha and omega, but members 2 to N-1 are incomparable, and the cohesion of the group and the alpha’s positional stability often require this. An independent #2, after all, would represent eventual danger to the chief, which is why proteges are only selected by alphas who plan on graduating to a higher-status group. What is it that keeps social status illegible within the group? Dimensionality. People have been comparing themselves against each other forever; what prevents the emergence of a total linear ordering is the fact that different dimensions of capability or status will produce different rankings, and there’s uncertainty about which matter more.

The group might have one main social status variable, and will usually arrange it so that only one person (or, at scale, a few) have truly elevated status in that dimension, because that’s necessary for stability and morale. Fights over who is #2 vs. #3 are an unwanted distraction. This leaves it to the people within the group to define social status how they will, and the good news for them is that most people can find things or combinations of things at which they’re uniquely good. People find niches. In The Office, people who will never be managers take solace in joining clubs like the Party Planning Committee and the Finer Things Club. People like to become “the <X> guy” (or gal) for some X that makes them important to the group, e.g. “I’m the Friday Cupcake Guy”. It gives each person an additional dimension of social status at which he or she can be an undisputed local chieftain– possibly of territory no one wants, but still the owner of something.

What might this have to do with programming? Well, I’ve often asked (myself, and others) what makes a great programmer, and the conclusion that I’ve come to is that it’s very hard, at an individual level, to tell. Speaking broadly, I can say that Clojure (or Lisp) programmers are better than Java programmers, who are in turn better than VB programmers. I know the patterns and the space, and that’s clearly true (in the aggregate). Better programmers like more challenging, but ultimately more powerful, languages. But there are excellent Java programmers and bad Lisp hackers. Also, if you bring a crack Clojure or Haskell developer into a typical enterprise Java environment where things are done in a wrong but common way, she’ll struggle, just because she’s not familiar with the design patterns. Moreover, a person’s reputation in a new job (and, in the long term, status and trajectory) depends heavily on his performance in the first few months, during which familiarity with the existing technology practices and tools (“tech stack”) have more of an impact than general ability. In the short term, it can be very hard to tell who the good and bad programmers are, because so much is project-specific. People are often judged and slotted into the typical software company’s pseudo-meritocracy before sufficient data can be collected about their actual abilities.

Assessing programmers is, to put it simply, a high-dimensional problem. There are more important and possibly powerful technologies out there (and plenty of duds, as well) than there is time to learn even a small fraction of them, and a lot of the know-how is specific to subdomains of “programming” in which one can have a long, fruitful, and deep career. Machine learning requires a dramatically different skill set from compiler design or web development; a top machine-learning engineer might have no idea, for example, how to build a webpage. People in business are judged shallowly (indeed, 95% of success in “business”, in the U.S., is figuring out how to come out on top of others’ shallow– but usually predictably so– judgements) and programming is rarely an exception, so when a person tries something new, there will be “embarrassing” gaps in his or her knowledge, no matter how capable that person is on his or her own territory. There might be 1000 dimensions that one could use to define what a good vs. bad programmer is, and no one excels at all of them.

Given the extreme dimensionality of assessing programmers, I also contend that self-assessment is very difficult. Good programmers don’t always know that they’re good (it can be frustrating and difficult even for the best) and many bad ones certainly think that they’re good. I don’t think that there are huge differences is self-confidence between men and women, individually speaking. Differences between groups are often smaller than those within groups, and I think that this applies to self-efficacy. However, I think the effect of dimensionality is that it can create a powerful feedback loop out of small personal biases over self-efficacy– and I do believe that men are slightly more inclined to overconfidence while women, in the aggregate, are slightly biased in the other direction. Dimensionality gives leverage to these tendencies. A person slightly inclined to view herself as competent will find reasons to believe she’s superior by selecting her strongest dimensions as important; one inclined the other way will emphasize her shortcomings. Dimensionality admits such a large number of ways to define the rules that a person’s view of him- or herself as a competent programmer is extremely flexible and can be volatile. It’s very easy to convince oneself that one is a master of the craft, or simply no good.

When starting in the software career, women (and men) have to deal with, for just one irritating example, socially undeveloped men who display obnoxious surprise when they’re new to programming trivia. (“You don’t know what xargs is?”) They also had to deal with those assholes when breaking into medicine and law as well, but there was a difference. The outstanding female doctors, for the most part, knew that they were competent and, often, better than the jerks hazing them. That was made obvious by their superior grades in medical school. In software, newcomers deal with environments where the dimensions of assessment often change, can sometimes (e.g. in “design pattern” happy Java shops) even be negatively correlated with actual ability, and are far outside of their control. Dimensionality is benevolent insofar as it gives people multiple avenues toward success and excellence, but can also be used against a person; those in which the person is weak might be selected as the “important” ones.

A piece of ancient wisdom, sometimes attributed to Eleanor Roosevelt although it seems to pre-date her, is: great minds discuss ideas, middling minds discuss events, and small minds discuss people. This is extremely true of programming, and it relates to dimensionality quite strongly.

Weak-minded programmers are actually the most politically dangerous; they don’t understand fuck-all, so they fall back to gossip about who’s “delivering” and who’s not, and hope to use others’ fear of them to extort their way into credibility, then garner a lateral-promote into a soft role before their incompetence is discovered. As expected, the weak-minded ones discuss people.

The great programmers tend to be more interested in the ideas (concurrency, artificial intelligence, algorithms, mathematical reasoning, technology) than in the tools themselves, which is why they’re often well-versed in many languages and technologies. Through experience, they know that it’s impossible to deal with all of computer science while limited to one language or toolset. Anyway, Hadoop isn’t, on its own, that interesting; distributed programming is. It’s what you can do with these tools, and what it feels like to use them on real problems, that’s interesting. Great minds in programming are more attracted to the fundamentals of what they are doing and how they do it than the transient specifics.

However, most minds are middling; and most programmers are the middling kind. Here, discussion of “events” refers to industry trends and the more parochial trivia. Now, great programmers want to narrow in on the core ideas behind the technologies they use and generally aren’t interested in measuring themselves in relation to other people. They just want to understand more and get better. Bad programmers (who usually engineer a transition to an important and highly compensated, but not technically demanding, soft role) play politics in a way that is clearly separate from the competence of the people involved; because they are too limited to grapple with abstract ideas, they focus on people, which often serves them surprisingly well. In other words, the strongest minds believe in the competition of ideas, the eventual consistency thereof, but stay away in general from the messy, ugly process of evaluating people. They shy away from that game, knowing it’s too highly dimensional for anyone to do it adequately. The weak minds, on the other hand, don’t give fuck-all about meritocracy, or might be averse to it, since they aren’t long in merit. They charge in to the people-evaluating parts (“Bob’s commit of April 6, 2012 was in direct violation of the Style Guide”) without heeding the dimensionality, because getting these judgments right just isn’t important to them; participating in that discussion is just a means to power.

Middling programmers, however, understand meritocracy as a concept and are trying to figure out who’s worth listening to and who’s not (the “bozo bit”). They genuinely want the most competent people to rise, but they get hung up on superficial details. Oh, this guy used Java at his last job. He must be a moron. Or: he fucking wrote a git commit line over 65 characters. Has he never worked as a programmer before? They get tricked by the low-signal dimensions and spurious correlations, and conclude people to be completely inexperienced, or even outright morons, when their skill sets and stylistic choices don’t match their particular expectations. These middling minds are the ones who get tripped up by dimensionality. Let’s say, for the sake of argument, that a problem domain has exactly 4 relevant concepts. Then there might be 25 roughly equivalent, but superficially different, technologies or methods or resume buzzwords that have been credibly proposed, as some time, as solutions. Each class of mind ends up in a separate space with differing dimensionalities. Great minds apprehend the 4 core concepts that really matter and focus on the tradeoffs between those. That means there are 4 dimensions. Low minds (the ones that discuss and focus on people) have a natural affinity for political machinations and dominance-submission narratives, which are primordial and very low in dimensionality (probably 1 or 2 dimensions of status and well-liked-ness). The middling minds, however? Remember I said that there are 25 slightly different tools for which familiarity can be used as a credible assessor of competence, and so we end up with a 25-dimensional space! Of course, those middling minds are no more agile in 25 dimensions than anyone else– we just can’t visualize more than two or three at a given time– which is why they tend to narrow in on a few of them, resulting in tool zealotry as they zero in on their local high grounds as representing the important dimensions. (“<X> is the only web framework worth using; you’re a moron if you use anything else.”)

I’ve been abstract and theoretical for the most part, but I think I’m hitting on a real problem. The mediocre programmers– who are capable of banging out code, but not insightful enough to be great at it, and far from deserving any credibility in the evaluation of others– are often the most judgmental. These are the ones who cling to tightly-defined Right Ways of, for example, using version control or handling tabs in source code. One who deviates even slightly from their parochial expectations is instantly judged to be incompetent. Since those expectations are the result of emotionally-laden overfitting (“that project failed because Bob insisted on using underscores instead of camelCase!”) they are stances formed essentially from random processes– often pure noise. But as I said before, it’s easy with a high-dimensional problem to mistake sparse-data artifacts (noise) for signal.

In other words, if you go into programming as a career, you’ll probably encounter at least one person who thinks of you as an idiot (and makes the sentiment clear) for no reason other than the fact that specific dimensions of competence (out of thousands of candidate dimensions) that he’s pinned his identity on happen to be the ones in which you’re weak. It’s shitty, it’s “random”, but in the high-dimensional space of software it’s almost guaranteed to happen at least once– especially when you’re starting out and making a lot of genuine mistakes– for everyone. This isn’t gendered. Men and women both deal with this. Obnoxious people, especially early in one’s career, are just an occupational annoyance in software.

Where I think there is a gendered difference is in the willingness to accept that kind of social disapproval. Whether this is innate or a product of culture, I have no idea, and I don’t care to speculate on that. But the people who will tolerate frank social disapproval (e.g. being looked down upon for doing a garage startup instead of having a paying corporate job) for long periods of time seem to be men. I would argue that most people can’t deal with the false starts, confusing and extreme dimensionality, and improbability of fair evaluation in most companies that characterize the software economy. This is becoming even more true as volatile startup methodologies displace the (deeply flawed, but stable) larger corporate regime, and as tools like programming languages and databases diversify and proliferate– a great thing on the whole, but a contributor to dimensionality. People who can go through that wringer, keep their own sense of self-efficacy strong in spite of all the noise, and do all this over the several years that it actually takes to become a highly competent programmer, are very uncommon. Sociopathy is too strong a word for it (although I would argue that many such people fall under the less pejorative “MacLeod Sociopath” category) but it takes an anti-authoritarian “fuck all y’all” flair to not only keep going, but to gain drive and actually heighten performance, amid negative social signals. Like full-on sociopathy, that zero-fucks-itude exists in both genders, but seems to be more common in men. It’s a major part of why most inventors, but also most criminals, are men.

Women have, in general, equal intellectual talents to ours and I would argue that (on average) they have superior abilities in terms of empathy and judgment of character; but they don’t seem as able to tolerate long-term social disapproval. For the record, I don’t mean to imply that this is a virtue of men. Quixotry, also more often a male trait, is the dangerous and often self-destructive flip side of this. Many things bring social disapproval (e.g. producing methamphetamine) because they deserve condemnation. I’m only making an observation, and I may not be right, but I think that the essentially random social disapproval that programmers endure in their early years (and the fact that it is hard, in a massively high-dimensional space, to gain objective proof of performance that might support a person against that) is a major part of what pushes a large number of talented women to leave, or even to avoid getting involved in the first place. I also think that it is dimensionality, especially of the kind that emerges when middling programmers define assessment and parochial trivia become more important than fundamental understanding, that creates that cacophony of random snark and judgment.

Don Draper’s firing and Silicon Valley

Spoiler Warning: Stop here if you watch Mad Men and aren’t caught up.

At the end of Season 6, Don Draper is fired, a move that many found surprising. One might argue that he has been “CTD” (circling the drain) for some time by that point, as the alcoholic dull spells that punctuate his flashes of brilliance have grown longer and deeper, and his necessity to the firm has declined in the wake of the merger that he (as it were) engineered. Still, the change is surprising at first, even if analysis shows it to be inevitable. (I will write, for this essay, as if Don’s termination were a fait accompli; I do it fully knowing that Season 7 will quite likely involve Don’s return to advertising, possibly in that firm. Don is not a usual person; he will bounce back, in some way, in the next season.)

Don used to be a creative artist. Toward the end, though, he hasn’t created a new ad for years. Instead, he comes up with inventive (but often sociopathic) solutions to business problems, establishing the reputation of a loose cannon. As we see in the mutiny at the end of Season 3, the anti-tobacco letter published after Lucky Strike fires the struggling new firm, and the complicated merger between his firm and that of his rival, Ted Chaough; he shines best, of late, when he’s moving people rather than product. He’s turned from a cynical, “black hat” intellectual who justifies hawking tobacco with jaded, post-beatnik nihilism, into a highly effective and manipulative businessman.

By the time he gets fired, he’s checked out from the daily work of the firm, so many predicted his professional demise (although I didn’t). He hasn’t been earning his keep. Then again, most of the other partners haven’t been earning theirs for years, if the truth’s to be told. This is still a time when partnership is to be a member of “the club” and enjoy the fruits of others’ labor. Sterling, Cooper has been quite tolerant of low performance in its partner-level ranks in the past. One of the perks of being in that club is a long professional audit cycle (generations). You get the benefit of the doubt for as long as you need it– unless you break the rules in a major way, which Don did, in Season 6′s finale.

How did Don get himself fired? Why did it happen now? It might seem to be an consequence of his horrendous pitch in front of Hershey, but he’s performed even more awfully in other pitches (see: Life cereal) and not come even close to that. His erratic performance isn’t a new problem. There’s something different about Season 6′s meltdown. This firm, after all, has tolerated his drunkenness, womanizing, lateness and absenteeism for some time. So what is it that has changed?

For one note, I’d like to call attention to a major player who’s not in the firing meeting: Ted Chaough. Ted and Don are rivals, and Don becomes nasty when they’re too close, but they still like each other on a personal level. They both suffer the same fate, which is to be permanently a junior partner on account of what they do. Account men are revenue, creative is cost; it was as true then as it is now. Ted would not try to push Don out; in fact, earlier on he implores him to become more involved in the firm (“join this company and read a memo!”). It’s Bert, Roger, and Jim who work him out of the firm, and I would argue that it’s largely because of the revelations pertaining to his background. Poor performance was forgivable when he was part of the club, but now that he’s told the truth about himself, he no longer belongs.

They’re not disgusted because Don lost Hershey’s, because they only had (from the sounds of it) a 1-in-30 chance anyway, and they seem to be cavalier about losing clients (as with Manischewitz, a couple episodes back). That firm– hardly a paragon of professionalism or optimal behavior– has had far more severe bungles than that. The disgust is with him, this time around. It’s not that the partners, individually, care about Don’s background. Bert knew most of Don’s secrets already. In fact, what’s worth keeping in mind about Roger and Bert is their flawlessly-played double-nature; their (admittedly, severe) character flaws come entirely out of their born social class; minus that, they’re decent human beings. As humans, they respect and like Don. As guardians of an upper class, however, they can no longer participate with him in that game of relationship-trading. Now that the beans (yes, pun intended) have been spilled in front of Hershey, it’s impossible to know for sure how far Don’s disclosures will travel. The gate-crasher must be tossed out, for fear of how others in the client sphere would receive his retention.

The commonality between Don Draper and Ted Chaough– the aspiration-driven, hopeful world they inhabit– shows the class-driven subordinate role of “creative”. Creative is almost always sourced from the lower, middle-and upper-middle classes (all lumped together as “Not Our Kind, Darling” by the upper class). Even if there weren’t a class-based lack of creativity in the elite– and I’d argue that there is one, due to risk aversion, entitlement, and inbreeding of both literal and figurative sorts– those bottom classes, taken together, are just two to three orders of magnitude larger. Relationship work can be done by any idiot with the right breeding, but creative talent is distributed by nature without regard to social class, and its attendant exertion has to come from the “hungry” outsiders. This also means that when the creative executives run out of ideas– and that’s inevitable, because the executive lifestyle is even more of a creativity-killer as the upper-class one– they’re tossed back where they came from. That’s Don’s future, as made clear even as the series began.

Don himself says that people tell us what they are, and we refuse to listen; in Season 1, he predicts exactly what will happen to him as well as to Pete Campbell, even though these predictions are so negative that one wishes not to believe them. For Pete, he predicts that his lack of interpersonal charisma and character will top him out as a mid-level executive no one likes. Check. Don predicts that he’ll age and run out of ideas, and then be devoured by younger, hungrier executives. Check, sort-of. Don seems to be his own worst enemy, not done in by others. Perhaps it is his past (nostalgia) that is the hungry, young executive to slay him. Since Season 1 he has understood, intuitively, that Bert and Roger (and even Pete) are the natural inhabitants of his world, while he’s just a passer-through. Ted and Peggy, like Harry Crane, will find their ways to other pastures. He won’t. Dick Whitman might remain alive, biologically, for another thirty years, but Don Draper is that job and when it ends, so does he.

Dick Whitman knows how to kill Don Draper, and lose his toehold in the New York advertising world. He tells the truth about himself. He does it in an embarrassing and costly way, but it’s what he reveals– not the cost to the firm of the revelation– that gets him shown the door. Don could run horrible lies and get away with almost anything; but once he lets the truth out, his professional life is over.

This is where it gets personal.

I don’t post about my own life that much. I’ve mentioned negative experiences at Google, neutral-to-positive experiences in finance, and let on that I’m a damn good programmer with strong mathematical skills. That I’m 30 years old and live in New York and like functional programming is not hidden, either. Still, people probably have a sketchy view of what I’ve done professionally and where I’m trying to go. About myself, I’m less comfortable sharing than most people are on the Internet. Yet, on the deeper, society-wide issues, I’ve also spent a lot of time in 2012 and 2013 trying to do something that few people have: in the public, find the truth. That’s why I’ve gone farther into the rabbit hole of the software world’s sociology than most people have the courage to go (in the public, by their real names). Boy, has that led to some real pain. I haven’t even begun to describe some of the things that have happened to me once I started doing that: unreasonable professional losses, threats, and inexplicable behavior around me. I’ve been through hell and I’m still burning; but, from a distance, to burn is to shine.

Despite my interest in public truth, I keep it hard for the Internet to know much about me. That’s because my own personality is not what’s interesting. I’m actually a run-of-the-mill, fairly typical guy in most ways. I come from an average background, look like an average person, et cetera. As a person, I’m not that interesting, and I don’t wish to be. What is interesting is the underlying and general truth that we need to discover to move society forward, and it’s an accident of fate that has left me extremely well-equipped to do it. I wasn’t born to be the one who’d coin the term “open allocation” and thus become the savior (if I am effective, that is) of technology. There’s no prophecy behind it and certainly no genetic superiority (trust me, I have none). That was just luck. Yet, here I am with a unique array of experiences that has left me able to pose (and sometimes, to answer) some of the most important questions of the technological economy? For example, why has the formerly most creative industry out there (small-company technology) fallen so quickly into decline, and how do we fix that? Can we do it? What types of structures (financial, cultural, and technological) will we need to invent to solve the problem?

For a second, I will get into some personal stuff, even at risk of embarrassment. It doesn’t take much work to figure out that I was a gray-hat troll in the 2000s. It was a hard habit to kick, because I have intermittent hypergraphia (compulsive writing). When I was an angry, broken person, that led to some angry and broken writing. I created some bizarre, fictional internet characters, many of which (and I’m thankful for this) have never been connected to me. However, some of that stuff was easy to find, at least at one time, and trace to me. Yet it never interfered with my career, at least not to my knowledge. Gray-hat trolling is seen (correctly) as a weird often-public hobby. It has never “caught up” with me or done any professional harm, but I’m actually pretty embarrassed by it. If ultimately harmless, it’s still fundamentally dishonest to create weird fictional personalities and convince others that they’re real. Worst of all, it was a gigantic waste of time. Yet, I “got away” with it completely. It probably made me seem a little bit strange and, at the time, would have precluded leadership opportunities if found– but I was also in my mid-20s, so that wasn’t an issue– in its time.

What has hurt my career is the white-hat stuff: truthful revelations about organizational behavior, specific companies and what they did wrong, and general willingness to state obvious but undesired truths about the software career and the sociological forces preventing technological progress. We now lose hundreds of billions, if not trillions, of dollars per year to bad software management. Yet the consequences of simply revealing such things have been, in my experience, severe.

My trolling past has shown that one can be offensive and horrible and flagrant as long and get away from it as long as one lies; because lies from non-credible sources (and let’s be honest, that’s 99.9% of us) are ineffectual and harmless. They’re entertainment. Truth, on the other hand, is a deadly weapon and a feared one, because it doesn’t matter who holds it. The power of a lie is directly proportionate to the social status of the liar, and most of us have such minimal social status as to be harmless to those in charge. Truth, on the other hand, has eternal and status-independent power, which makes it dangerous.

Tell a lie as a non-credible agent, and you’ll be cancelled out by the rest of the world, as it generates a noise haze of counter-lies and oblique lies and inept supporting lies and uncomfortable humor that weaken your case and render what you said irrelevant. The people with stakes in reputations can count on the lie having no long-term net effect. Tell the truth, on the other hand, and there’s a risk (albeit a small one, moral courage often being thin on the ground) that the world will move with that revelation, as more people come out to confirm it. Truth is gunpowder (an equalizer) and the upper-caste sword-wielders (credible liars) can’t stand for anyone to have it.

Now, I’ll return to Mad Men. Don Draper’s truth isn’t just that he grew up in a coal-country whorehouse. There’s a lot more to it than that. To start, he establishes his pitch as a lie, pointing out that advertisement exploits untrue stories and what people want to believe. That’s not a truth to share with a client. Another unstated truth he brings out is that no one will accept his true upbringing, because there is supposed to be no one like him in a high position in that sort of firm. He duped the whole firm into buying his prestige. Account men come from the generational upper class, and creative comes from the Ivy upper-middle, and people like Dick Whitman shouldn’t even be on the floor, unless running the elevator. The humiliation of the firm comes from the revelation– in front of Hershey’s upper management, since the partners would probably accept the fact if known only to them– that they let him in.

Bert Cooper and Roger Sterling were duped, and they were happy to be duped when he was a brilliant creative executive. Being relatively progressive by upper class standards, they recognize prestige as an elaborate lie, so learning about Don’s charlatanry made no difference so long as he delivered. However, when he makes Hershey aware of his gate-crashing, the damage is so severe that he must be thrown out with the trash.

Finally, there’s a major truth, revealed throughout the show, about advertisement’s self-selling as a “creative” industry. Creativity (in advertising) takes a second-class standing relative to the corrosive politicking for which Pete and Bob Benson are so well-known. Peggy and Don take pride in their creative work, and Ted additionally takes pride (MacLeod Clueless?) in his leadership ability– he’s the only decent boss on the show. Yet none of the stuff matters. Not one of the creatives is present in (or even aware of) the decision to fire Don, because their opinions don’t merit concern. Creatives are just high-end light bulbs to be used, burned out, and discarded. That’s a truth that’s relevant to this day. Recall what I said about creative being cost centers and account men being revenue-producers. (Of course, creative work drives the long-term health of the firm, but that’s irrelevant to the year-by-year decisions around promotions and firings.)  The upper class naturally gravitate toward the unsexy but leverage-providing revenue-center roles, and leave creative cost centers for the marginal people who need to prove themselves to stay in the game.

Replace “Madison Avenue” with “VC-funded Silicon Valley” and “copywriters” with “software engineers”. Conclude your own about that ecosystem and its fate, noting that the glory days of Madison Avenue ended a couple years after the events of the recent series. VC-istan is the successor to the world of Mad Men, with the same soul-devouring politics, except it’s even more of a sausage fest.

Don– to exploit the metaphor, the most honest whore in the brothel– has just burned out of an industry founded upon lies. He’s brought in truth. He didn’t belong, he got in anyway; and about a decade later, he fired himself on his own terms. A few stray flames are smoldering and truth is breaking through, but the total conflagration is yet to come. (Advertising will be just fine, but its “white-shoe” reputation and cachet will fall away in less than a decade.) Don has angered Power enough for it to come in to an office at 9:00 on Thanksgiving morning. Old lies are breaking down– that process started long before Don got in– but the consequences for revealing truth, in such a time, can be severe.

So it is, too, with VC-istan. Truth is breaking through. It has had, over the past two months, some excruciating revelations, between Sean Parker’s wedding, the revelation of inappropriate data usage, FWD.us, and various morale crises among software engineers that will not go away until terms improve. The exceptionalist acceptance once applied to technology’s new barons is waning. The edifice of lies is inflamed, and no one knows what will happen next. The only sure thing is that it will be fun to watch. As with Mad Men, all that most of us can do is wait, stay alive, and then enjoy the next season.

Gervais / MacLeod 9: Convexity

Originally, I had intended the 9th part to be the one in which I solve this damn thing for good. Unfortunately, as that “9th post” crossed into 12 kiloword territory, I realized that I needed to break it up a bit. Even for me, that’s long. So I had to tear some stuff out and split this “final post” yet again. So here’s the 9th chapter in this ongoing lurid saga. (See: Part 1Part 2Part 3Part 4Part 5Part 6Part 7, Part 8). I am really trying to wrap this fucker up so I can get my life back, but I’m not going to solve it just yet… there are a few more core concepts I must address. Today’s topic, however, is convexity.

What is convexity?

Convexity pertains to the question: which is more important, the difference between excellent work and mediocre work, or that between mediocre work and noncompliance (zero)? For concave work, the latter is more important: just getting the job done is what matters, and qualitative concerns are minimal. For convex work, the difference between excellence and mediocrity is substantial and that between mediocrity and failure is small.

The theoretical basis for this is the logistic function, or “S-curve”, which is convex at its left side (looking exponential at y << 0.5) and concave on the right, as it approaches a horizontal asymptote (saturation point). Model input as a numerical variable i pertaining to resources, skill, talent, or effort. Then model task output as having a maximal yield Y, and the function Y * p(i) where p is a logistic function with range (0, 1) representing the proportion of maximum possible yield that is captured. Now, the inflection point (switch-over from convexity to concavity) is exactly where p(i) = 0.5. Taken in full, this logistic function is neither concave or convex. Yet, for most economic problems, the relevant band of the input range is narrow and is mostly on one side of the inflection point or the other. We can classify tasks as convex or concave based on what we know about the performance of the average.

To get concrete about it, consider exams in most schools. A failing student might be able to answer 60% of the questions right; an average one gets 80%, and the good ones get 90%. That’s a concave world. The questions are easy, and one needs to get almost all of them right to distinguish oneself. On the other hand, a math researcher would be thrilled to solve 50% of the (much harder) problems she confronts. With concave work, the success or completion rates tend to be high because the tasks are easy. With convex work, they’re low because the tasks are hard. What makes convex work worth doing is that, often, the potential yield is much higher. If the task is concave, it’s been commoditized, and it’ll be hard to make a profit by doing it. Even 100% completion will yield only marginal profits. If the work is convex, the most successful players can generate outsized returns. It may not be clear even what the upper limit (the definition of “100%”) is.

Convexity and management

Consider the following payoff structures for two tasks, A and B. A is concave; B is convex, and 0 to 4 represent degrees of success.

| Performance | A Payoff | B Payoff | Q dist | R dist |
-------------------------------------------------------
| 4 (Superb)  |      125 |      500 |  20.0% |   0.0% |
| 3 (Good)    |      120 |      250 |  20.0% |  20.0% |
| 2 (Fair)    |      100 |      100 |  20.0% |  60.0% |
| 1 (Poor)    |       60 |       25 |  20.0% |  20.0% |
| 0 (Awful)   |        0 |        0 |  20.0% |   0.0% |
-------------------------------------------------------

Further, let’s assume there are two management strategies, Q and R. Under Q, the workforce will be uniformly distributed among the five tiers of performance: 20% in each. Under R, 20% each of the workforce will fall into Good and Poor, 60% into the Fair tier, and none into the Superb or Awful tiers. R is variance-reducing managerial strategy. It brings people in toward the middle. The goal, here is to maximize bulk productivity, and we assume we have enough workers that we can use the expected payoff as a proxy for that.

For Job A, which is concave, management strategy Q produces an output-per-worker of 81, while R yields 96. The variance-reducing strategy, R, is the right one, yielding 15 points more. For example, bringing up the worst slackers (from 0 to 60) delivers more benefit than pulling down the top players (from 125 to 120).

For Job B, which is convex, strategy Q gives us an average yield of 165, while R delivers only 105– 60 points less. The variance-reducing strategy fails. We see more of a drop in pulling down the best people (from 500 to 250) than we gain in hauling the laggards (o to 25).

In short, when the work is concave, variance is your enemy and reducing it increases expected yield. When the work is convex, variance is your friend; more risk means more yield. 

The above may seem disconnected from the problems of the MacLeod organization, but it’s not. MacLeod organizations are based on variance-reduction management strategies, which have worked overwhelmingly well over the past 200 years of concave, industrial-era labor. MacLeod Losers naturally desire familiarity, uniformity, and stability. They want variance to be reduced and will give up autonomy to have that. MacLeod Clueless (middle managers) take on the job of reducing variance in conditions for the Losers below them, and reducing volatility in performance for the Sociopaths above. Their job is to homogenize and control, and they do it well. It doesn’t require vision or strategy. MacLeod Sociopaths start out as the “heroic” risk-takers (entrepreneurs) but that caste often evolves (as especially as transplant executives come in) into a cushy rent-seeking class as the organization matures (necessitating the obfuscation enabled by the Clueless and the Effort Thermocline). The Sociopath category itself becomes risk-averse, out of each established individual’s desire to protect organizational position. The result is an organization that is very good at reducing variance and stifling individuality, but incapable of innovation. How do we come back from that?

In the concave world, the failures of the MacLeod organization were tolerable. Businesses didn’t need to generate new ideas. They needed to turn existing, semi-fresh ones into repeatable processes, and motivate large groups of people to carry out difficult but mostly simple functions. Variance-reduction was desired and encouraged. Only in the past few decades, with the industrial era fading and the technological one coming on, has there been a need for business to have in-house creative capacity.

Old-style organizations: the optimization model

The convex/concave discussion above assumes one dimension of input (pertaining to how good an individual is at a job) and one of output (observed productivity). In truth, a more accurate model of an organization’s performance would have a interconnected network of such “S-curve” functions for the relationships between various variables, many of which are hidden. There’d be a few input variables (“business variables”) and the things the company cares about (profit, reputation, organizational health) would be outputs, but most of the cause-and-effect relationships are hidden. Wages affect morale, which affects performance, which affects productivity, which affects the firm’s profits, which is its performance function. With all of the dimensions that could be considered, this function might be very convoluted, and while it is held to exist “platonically” it is not known in its entirety. The actual function relating controllable business variables to performance is illegible (due to hidden variables) and certainly not perfectly concave or convex.

So how does the firm find an optimal solution for a problem it faces?

This gets into an area of math called optimization, and I’m not going to be able to do it justice, so I’ll just address it in a hand-wavy way. First, imagine a two-dimensional space (if only because it’s hard to visualize more) where each point has an associated value, creating a 3-dimensional graph surface. We want to find the “highest point”. If that surface is globally concave, like an inverted bowl, that’s very easy, because there can only be one maximum. We can start from any point and “hill climb”: assess the local gradient, step in the most favorable dimension. We’ll end up at the highest point. However, the more convoluted our surface is, the harder the optimization problem. If we pick a bad starting point on a convoluted surface, we might end up somewhere sub-optimal. Thinking of it in topographical terms, a “hill climb” from most places won’t lead to the top of Mount Everest, but to the neighborhood’s highest hill. In other words, the “starting point” matters if the surface is convoluted.

Real optimization problems usually involve more than two dimensions. It is obviously not the case that organizations perform optimizations over all possible values of all possible business variables (of which there are an infinite number). Additionally, the performance function changes over time. As a metaphor, however, for the profit-maximizing industrial corporation, it’s surprisingly useful. One part of what it must do is pick a good “starting point” for the state of the business, a question of “What should this company be?” That requires non-local insight. Another part is the iterative process of refinement and hill-climbing once that initial point is selected.

This leads to the three-tier organization. People who are needed for commodity labor but not trusted, at all, to affect business variables are mere workers. Managers perform the iterative hill-climb and find the highest point in the neighborhood. In startup terms, they “iterate”. Executives, whose job is to choose starting points, also have the right to make non-local “jumps” in business state if needed. In startup terminology, they “pivot”.

The MacLeod organization gets along well with this computational model of the organization. MacLeod Losers are mediocre in dedication, but that’s fine. That aspect of them is treated as a hidden variable that can be modulated via compensation (carrots) or managerial attention (sticks). In the optimization model, they’re just infrastructure– human resources in the true sense of the word. The fault of MacLeod Clueless is that they aren’t strategic, but they don’t need to be. Since their job is just to climb a hill, they don’t need to worry about non-local “vision” concerns such as whether they’re climbing the right hill. That’s for someone else to worry about. They just assess the local gradient and move in the steepest upward direction. Finally, there are the MacLeod Sociopaths, whose goal is to be strategic and have non-local insight. Being successful at that usually requires a high quality of information, and people don’t get that stuff by following the rules. The source could be illegal (industrial espionage) or chaotic (experimental approaches to social interaction) or merely insubordinate (a programmer learning new technologies on “company time”) but it’s almost always transgressive. The MacLeod Sociopath’s ability to get information confers more benefit, in an executive position, than the negatives associated with that category.

Why the optimization model breaks down

In the model above, there is some finite and well-specified set of business variables. The real world is much more unruly. In truth, there are an infinite number of dimensions. Two things make this more tractable. The first is sparsity. Most dimensions don’t matter. For example, model “product concern” as a vector representing the products that a company might make (1.0 meaning “it’s the only thing we care about”, 0 meaning “not interested at all”). Assume there are 387 trillion possible conceivable products that a firm could create. That’s 387 trillion business variables; 386.99999…+ trillion of those entries are always going to be zero (excepting a major pivot) and can be thrown out of the analysis. Second is aggregation. For personnel, one could have a variable for each of the world’s 7.1 billion people (again, most being zero for ‘not working here’) but most companies just care about a few things, like how many people they employ and how much they cost. Headcount and budget are the important business variables. Whether John A. Smith, 35, of Flint, Michigan is employed at the company (i.e. one of those 7.1 billion personal variables) isn’t that relevant for most values of John A. Smith, so executives need not concern themselves with it.

Even still, modern companies have thousands to millions of business variables that matter to them. That’s more information than a single person can process. Then there is the matter of what variables might matter (unknown unknowns). If the optimization problem were simple, the company would only need one executive to call out starting points, but these information issues mandate a larger team. The computation that the organization exists to perform must be distributed. It can’t fit on one “node” (i.e. one person’s head). That also mandates that this massively high-dimensional optimization problem be broken down as well. (I’m ignoring the reality, which is that most people in business don’t “compute” at all, and that many decisions are made on hunches rather than data.)

As far as many dimensions are separable (that is, they aren’t expected to interact, so the best values for each can be found in separation) the problem can be decomposed by splitting it into subproblems and solving each in isolation. Executives take the most important business variables where it is most likely that non-local jumps will be needed, such as whether to lay off 15% of the workforce. The less important ones (like whether to fire John A. Smith) are tackled by managers. Workers don’t participate in the problem-solving; they’re just machines.

This evolves from an optimization model where business variables and performance functions are presumed to exist platonically, to a distributed agent-based model operated by local problem-solving agents. This is a more accurate model of what actually happens in the corporation, made further amusing by the fact that the agents often have diverging personal objective functions. Centralized computation is no longer the most important force in the company; it’s communication between the nodes (people).

Here’s where MacLeod comes in to the agent-based model. MacLeod Losers consume information and turn it into work. That’s all they’re expected to do, and ideally the only thing that should be coming back is one word: done. MacLeod Clueless furnish information up and down the food chain, non-editorially because they aren’t strategic enough to turn it into power. They tell the Losers what to do, and the Sociopaths what was done, and they aren’t much of a filter. The only information they take in, in general, is what information others want from them. The MacLeod Sociopaths are strategic givers and takers of information, and (having their own agendas) they are selective in what they transmit. Organizations actually need such people in order to protect the top decision-makers from “information overload”. It’s largely the bottom-tier Sociopaths who participate in dimensionality reduction and aggregation, so they’re absolutely vital; however, they make sure to use whatever editorial sway they have toward their own benefit.

Optimization and convexity

The actual performance function of a company, in terms of its business variables, is quite convoluted. It’s generally concave in a neighborhood (enabling managers to find the “local hill”) but its global structure is not, necessitating the non-local jumping afforded to executives. The underlying structure, as I said earlier, is driven by an inordinate number of hidden variables. It might be best thought of as a neural network of S-curve functions (“perceptrons”) wherein there are elements of concavity and convexity, often interacting in strange ways. It’s not possible for anyone to ascertain what a specific organization’s underlying network looks like exactly. The overall relationship between business variables (inputs) and performance (output) is not going to be purely concave or convex. The best one can hope for is a well-chosen initial point in which the neighborhood is concave.

For typical organizations and most people in them, concavity has a lot of nice properties. For one thing, it tends toward fairness. Allocation of resources to varying parties, if the input/output relationships are concave, is likely to favor an “interior” solution– everyone gets some– because marginal returns diminish as the resource is allocated to richer people. If the input/output relationship is convex, resource allocation can favor an “edge” solution where one party gets everything and the rest get none, which tends to exacerbate the “power law” distributions associated with social inequalities: a few (who seem the most capable of turning those resources into value) get much, most get little or nothing. Another benefit of concavity is that performance relative to a standard can be measured. At concave work, the maximum sustainable output is typically well-studied and known, and acceptable error rates can be defined. With convex work, no one knows what’s possible. Once a maximum is established and can be reliably attained, the task is likely to become concave (as people develop the skills to perform it successfully more than 50% of the time). Research is inherently convex: most things explored don’t pan out, but those that do deliver major benefit. When those explorations lead to repeatable processes that can be carried out by people of average motivation and talent, that’s concave, commodity work.

MacLeod organizations exist as a risk trade between those who want to be protected from the vagaries of the market, so creating islands of concavity and easiness is kind of what they do. The Big World Out There is a place with many pockets of convexity, plenty of bad local maxima, and a difficult and mostly unexplored landscape. The MacLeod organization provides its lower-level workers with access to an already-explored and safe concave hill neighborhood. Executives read maps and find start points. Managers just follow the steepest gradient up to the top, and workers just have to follow and carry the manager’s stuff.

Technology and change

There’s a problem with concave work. It tends to be commoditized, making it hard to get substantial profits on it. If “100% performance” can truly be defined and specified, then it can also be achieved mechanically. Not only are the margins low, but machines are just better at it than humans: they’re cheaper, don’t need breaks, and don’t make as many mistakes. Robots are taking over the concave work, leaving us with convexity.

We cannot compete with robots on concave work. We’ll need to let them have it.

Software engineering is notoriously convex. First of all, excellent software engineers are 5 to 100 times more productive than average ones, a phenomenon known as the “10X engineer”. As is typical of convex projects, most software projects fail– probably over 80 percent deliver negative net value– but the payoffs of the successes are massive. This is even more the case in the VC-funded startup ecosystem, where companies that seem to show potential for runaway success are sped along, the laggards being shot and killed. In a convex world, that’s how you allocate resources and attention: focus on the winners, starve the losers.

Convexity actually makes for a very frustrating ecosystem. While convex work is a lot more “fun” because of the upside potential and the challenge, it’s not a great way to make a living. Most software engineers get to age 30 with no successful projects (most of this being not their fault) under their belt, no credibility on account of that series of ill-fated projects, and mediocre financial results (even if they had successes). Managing convex work, and compensating it fairly, are not things that we have a society have figured out how to do. For the past 200 years– the industrial era, as opposed to the technological one that is coming on– we haven’t needed to do it. Almost all human labor was concave. What little convex work existed was generally oriented toward standardizing processes so as to make 99.9% of the organization’s labor pool concave. We are now moving toward an economy where enormous amounts of work are done by machines, practically for free, leaving only convex, creatively taxing work.

The fate of the MacLeod organization

MacLeod organizations, over the past 200 years, could perform well. They weren’t great at innovation; they didn’t need it. They got the job done. One of the virtues of the corporation was its ability to function as a machine made out of people. It would render services and products not only at more scale, but much more reliably, than individual people could do. The industrial corporation co-evolved with the failings of each tier of the MacLeod organization, hence converging on the “optimization model” above that uses the traits of each to its benefit. Of course, I do not mean to suggest that these “computations” are performed in reality, but the metaphor works quite well. 

The modern technological economy has created problems for that style of organization, however. Microeconomic models tend to focus on a small number of business variables– price points, quantity produced, wages. Current-day challenges require thousands, often ill-defined. What product is one going to build? What kind of people should be hired? What kind of culture should the company strive toward, and how will it enforce those ideals? Those things matter a lot more in the technological economy. Hidden variables that could once be ignored are now crucial, and industrial-era management is falling flat. Combine this with the convexity of input/output relationships regarding individual talent, effort, and motivation, and we have a dramatically different world.

The islands of concavity that MacLeod organizations can create for their Losers and Clueless are getting smaller by the year. The ability to protect the risk-averse from the Big World Out There is diminishing. MacLeod Sociopaths were never especially scrupulous about keeping that implicit promise, but now they can’t.

Even individuals now have to make non-local (formerly executive-level) decisions. For a concrete example, consider education. The generalist education implicitly assumes that most people will have a concave relationship between their amount of knowledge in an area and utility they get from it. It’s vitally important, as most educational institutions see it, for one to first get a mediocre amount of knowledge about a lot of subjects. (I agree, for non-economic reasons. How can a person know what is interesting without having a wide survey of knowledge? A mediocre knowledge gives you enough to determine if you want to know more; with no knowledge, you have no clue.) However, there’s no such thing as a Generalist job description. The market doesn’t reward a breadth of mediocre knowledge. People need to specialize. In 1950, having a college degree bought a person credibility as someone capable of learning quickly, thus entry into the managerial, professional, or academic ranks. (Specialization could begin on the job.) By 1985, one needed a marketable major: preferably, math, CS, or a physical science. In 2013, what classes a person took (compilers? machine learning?) is highly relevant. The convex valuation of a knowledge base makes deep knowledge in one area more valuable than a broader, shallower knowledge. Choosing and changing specialties is also a non-local process. A well-rounded generalist can move about the interior by gradually shifting attention. The changing specialist must jump from one “pointy” position to another– and hope it’s a good place to be.

In technology especially, we’re seeing an explosion of dimensionality. General competence doesn’t cut it anymore. Firms aren’t willing to hire “overall good” people who might take 6 months to learn their technology stacks, and the most credible job candidates don’t want to pin their careers on companies that don’t strongly correspond with their (sometimes idiosyncratic) preferences. When there’s a bilateral matching problem (e.g. dating) it usually has something to do with dimensionality. Both sides of the market are “purple squirrel hunting”.

This proliferation of dimensionality isn’t sustainable, of course. One thing I’ve come to believe is that it has an onerous effect on real estate prices. That might seem bizarre, but the “star cities” are the only places that tolerate purple squirrel hunting. If you’re a startup that wants a Python/Scala/C++ expert with production experience in 4 NoSQL products and two PhDs, you can find her in the Bay Area. For some price, she’s out there. That’s not because the people in the Bay Area are better; it’s that, with more of them, you get a continuous (it’s available at some price) rather than discrete (you might wait intolerably long and not find it) market for talent– and also for jobs, from a candidate’s perspective– even if you’re trying to fill some ridiculous purple squirrel specification. That’s what makes “tech hubs” (e.g. Bay Area, New York, Boston) so attractive– to candidates and companies both– and a major part of what keeps them so expensive. The continuous markets make high-risk business and job-hopping careers– that aren’t viable in smaller cities unless one wants to move or tolerate remote work– possible. Since real estate in these areas is reaching the point of being unaffordable for technology workers, I think it’s a fair call to say that this dimensionality explosion in technology won’t continue forever. However, convexity and high dimensionality in general are here to stay, and about to become the norm for the greater economy. The convexity introduced by an economic arrangement where an increasing bulk of commodity labor is dumped directly on machines has incredible upsides, and is very attractive. Now, in the late-industrial era, global economic growth is about 4-5 percent per year. In the thick of the technological era– a few decades from now– it could be over 10% per year.

If MacLeod rank cultures are going to become obsolete, what will replace them? That I do not know for sure, but I have some thoughts. The “optimization model” paints a world where the relevant business variables are known. Executives call out initial values (based on non-local knowledge) for a gradient ascent performed by managers. As the business world becomes high-dimensional– too many dimensions for any one person to handle them all– it begins to break down the problem and distribute the “computation” (again, solely in metaphor). High-ranking executives handle important dimensions (sub-problems) where tricky non-local jumps might be in order. Managers handle less-important ones where continuous modulation will do. Getting the communication topology right is tricky. Often the conceptual hierarchy that is created will look suspiciously like the organizational hierarchy (Conway’s Law?) This leads to an interesting question: is this hierarchy of people– which will limit the firm’s capacity to form proper conceptual hierarchies and solve its own problems– even necessary? Or is it better to have all eyes open on non-local, “visionary” questions? Is that a good idea? Organizations claim to want their employees to “act like owners”. Is that really true? With the immense complexity of the technological economy, and the increasing inability of centralized management to tackle convexity (one cannot force creative excellence or innovation by managerial fiat) it might have to be true.

Enter the self-executive. Self-executive people don’t think of themselves as subordinate employees, but as free agents. They don’t want to be told what to do. They want to excel. A manager who will guide one (mentorship) gets loyalty. However, typical exploitative managers get ignored, sabotaged, or humiliated. Self-executive employees are the ones who can handle convexity, and enjoy the risk and challenge of hard problems. They strongly toward chaos on the civil alignment spectrum. These are the people one will need in order to navigate a convex technological economy, and the self-executive culture is the one that will unleash their capabilities.

That said, the guild culture has a lot to add as well and should not be ignored. There’s a lot of lost work in exploration that can be eliminated by advice from a wise mentor (although if things change, as they do more rapidly these days, that “don’t go there” advice might sometimes be best discarded). The valuation of knowledge and skill are so strongly convex that there’s immense value generation in teaching. Not only should that not be ignored, but it’s going to become a critical component of the working culture. Companies that want loyalty are going to have to start teaching people again. Self-executives don’t work hard unless they believe they’re learning more on the work given to them than they would on their own– and these people tend to be fiercely autodidactic.

This brings us to the old quip. A VP tells his CEO that the company should invest more in its people, and he says, “What if we spend all that money training them and they leave?” The VP’s response: “what happens if we don’t and they stay?” That ends up looking like MacLeod rank culture over time. There’s a lot to be learned from guild culture, and when I finally Solve This Fucking Thing (Part 11? 12? 5764+23i?) I won’t be able to afford to overlook it.

IDE Culture vs. Unix philosophy

Even more of a hot topic than programming languages is the interactive development environment, or IDE. Personally, I’m not a huge fan of IDEs. As tools, standing alone, I have no problem with them. I’m a software libertarian: do whatever you want, as long as you don’t interfere with my work. However, here are some of the negatives that I’ve observed when IDEs become commonplace or required in a development environment:

  • the “four-wheel drive problem”. This refers to the fact that an unskilled off-road driver, with four-wheel drive, will still get stuck. The more dexterous vehicle will simply have him fail in a more inaccessible place. IDEs pay off when you have to maintain an otherwise unmanageable ball of other people’s terrible code. They make unusable code merely miserable. I don’t think there’s any controversy about this. The problem is that, by providing this power, then enable an activity of dubious value: continual development despite abysmal code quality, when improving or killing the bad code should be a code-red priority. IDEs can delay code-quality problems and defer macroscopic business effects, which is good for manageosaurs who like tight deadlines, but only makes the problem worse at the end stage. 
  • IDE-dependence. Coding practices that require developers to depend on a specific environment are unforgivable. This is true whether the environment is emacs, vi, or Eclipse. The problem with IDEs is that they’re more likely to push people toward doing things in a way that makes use of a different environment impossible. One pernicious example of this is in Java culture’s mutilation of the command-line way of doing things with singleton directories called “src” and “com”, but there are many that are deeper than that. Worse yet, IDEs enable the employment of programmers who don’t even know what build systems or even version control are. Those are things “some smart guy” worries about so the commodity programmer can crank out classes at his boss’s request.
  • spaghettification. I am a major supporter of the read-only IDE, preferably served over the web. I think that code navigation is necessary for anyone who needs to read code, whether it’s crappy corporate code or the best-in-class stuff we actually enjoy reading. When you see a name, you should be able to click on it and see where that name is defined. However, I’m pretty sure that, on balance, automated refactorings are a bad thing. Over time, the abstractions which can easily be “injected” into code using an IDE turn it into “everything is everywhere” spaghetti code. Without an IDE, the only way to do such work is to write a script to do it. There are two effects this has on the development process. One is that it takes time to make the change: maybe 30 minutes. That’s fine, because the conversation that should happen before a change that will affect everyone’s work should take longer than that. The second is that only adept programmers (who understand concepts like scripts and the command line) will be able to do it. That’s a good thing.
  • time spent keeping up the environment. Once a company decides on “One Environment” for development, usually an IDE with various in-house customizations, that IDE begins to accumulate plugins of varying quality. That environment usually has to be kept up, and that generates a lot of crappy work that nobody wants to do.

This is just a start on what’s wrong with IDE culture, but the core point is that it creates some bad code. So, I think I should make it clear that I don’t dislike IDEs. They’re tools that are sometimes useful. If you use an IDE but write good code, I have no problem with you. I can’t stand IDE culture, though, because I hate hate hate hate hate hate hate hate the bad code that it generates.

In my experience, software environments that rely heavily on IDEs tend to be those that produce terrible spaghetti code, “everything is everywhere” object-oriented messes, and other monstrosities that simply could not be written by a sole idiot. He had help. Automated refactorings that injected pointless abstractions? Despondency infarction frameworks? Despise patterns? Those are likely culprits.

In other news, I’m taking some time to learn C at a deeper level, because as I get more into machine learning, I’m realizing the importance of being able to reason about performance, which requires a full-stack knowledge of computing. Basic fluency in C, at a minimum, is requisite. I’m working through Zed Shaw’s Learn C the Hard Way, and he’s got some brilliant insights not only about C (on which I can’t evaluate whether his insights are brilliant) but about programming itself. In his preamble chapter, he makes a valid insight in his warning not to use an IDE for the learning process:

An IDE, or “Integrated Development Environment” will turn you stupid. They are the worst tools if you want to be a good programmer because they hide what’s going on from you, and your job is to know what’s going on. They are useful if you’re trying to get something done and the platform is designed around a particular IDE, but for learning to code C (and many other languages) they are pointless. [...]
Sure, you can code pretty quickly, but you can only code in that one language on that one platform. This is why companies love selling them to you. They know you’re lazy, and since it only works on their platform they’ve got you locked in because you are lazy. The way you break the cycle is you suck it up and finally learn to code without an IDE. A plain editor, or a programmer’s editor like Vim or Emacs, makes you work with the code. It’s a little harder, but the end result is you can work with any code, on any computer, in any language, and you know what’s going on. (Emphasis mine.)

I disagree with him that IDEs will “turn you stupid”. Reliance on one prevents a programmer from ever turning smart, but I don’t see how such a tool would cause a degradation of a software engineer’s ability. Corporate coding (lots of maintenance work, low productivity, half the day lost to meetings, difficulty getting permission to do anything interesting, bad source code) does erode a person’s skills over time, but that can’t be blamed on the IDE itself. However, I think he makes a strong point. Most of the ardent IDE users are the one-language, one-environment commodity programmers who never improve, because they never learn what’s actually going on. Such people are terrible for software, and they should all either improve, or be fired.

The problem with IDEs is that each corporate development culture customizes the environment, to the point that the cushy, easy coding environment can’t be replicated at home. For someone like me, who doesn’t even like that type of environment, that’s no problem because I don’t need that shit in order to program. But someone steeped in cargo cult programming because he started in the wrong place is going to falsely assume that programming requires an IDE, having seen little else, and such novice programmers generally lack the skills necessary to set one up to look like the familiar corporate environment. Instead, he needs to start where every great programmer must learn some basic skills: at the command-line. Otherwise, you get a “programmer” who can’t program outside of a specific corporate context– in other words, a “5:01 developer” not by choice, but by a false understanding of what programming really is.

The worst thing about these superficially enriched corporate environments is their lack of documentation. With Unix and the command-line tools, there are man pages and how-to guides all over the Internet. This creates a culture of solving one’s own problems. Given enough time, you can answer your own questions. That’s where most of the growth happens: you don’t know how something works, you Google an error message, and you get a result. Most of the information coming back in indecipherable to a novice programmer, but with enough searching, the problem is solved, and a few things are learned, including answers to some questions that the novice didn’t yet have the insight (“unknown unknowns”) yet to ask. That knowledge isn’t built in a day, but it’s deep. That process doesn’t exist in an over-complex corporate environment, where the only way to move forward is to go and bug someone, and the time cost of any real learning process is at a level that most managers would consider unacceptable.

On this, I’ll crib from Zed Shaw yet again, in Chapter 3 of Learn C the Hard Way:

In the Extra Credit section of each exercise I may have you go find information on your own and figure things out. This is an important part of being a self-sufficient programmer. If you constantly run to ask someone a question before trying to figure it out first then you never learn to solve problems independently. This leads to you never building confidence in your skills and always needing someone else around to do your work. The way you break this habit is to force yourself to try to answer your own questions first, and to confirm that your answer is right. You do this by trying to break things, experimenting with your possible answer, and doing your own research. (Emphasis mine.)

What Zed is describing here is the learning process that never occurs in the corporate environment, and the lack of it is one of the main reasons why corporate software engineers never improve. In the corporate world, you never find out why the build system is set up in the way that it is. You just go bug the person responsible for it. “My shit depends on your shit, so fix your shit so I can run my shit and my boss doesn’t give me shit over my shit not working for shit.” Corporate development often has to be this way, because learning a typical company’s incoherent in-house systems doesn’t provide a general education. When you’re studying the guts of Linux, you’re learning how a best-in-class product was built. There’s real learning in mucking about in small details. For a typically mediocre corporate environment that was built by engineers trying to appease their managers, one day at a time, the quality of the pieces is often so shoddy that not much is learned in truly comprehending them. It’s just a waste of time to deeply learn such systems. Instead, it’s best to get in, answer your question, and get out. Bugging someone is the most efficient and best way to solve the problem.

It should be clear that what I’m railing against is the commodity developer phenomenon. I wrote about “Java Shop Politics” last April, which covers a similar topic. I’m proud of that essay, but I was wrong to single out Java as opposed to, e.g. C#, VB, or even C++. Actually, I think any company that calls itself an “<X> Shop” for any language X is missing the point. The real evil isn’t Java the language, as limited as it may be, but Big Software and the culture thereof. The true enemy is the commodity developer culture, empowered by the modern bastardization of “object-oriented programming” that looks nothing like Alan Kay’s original vision.

In well-run software companies, programs are build to solve problems, and once the problem is finished, it’s Done. The program might be adapted in the future, and may require maintenance, but that’s not an assumption. There aren’t discussions about how much “headcount” to dedicate to ongoing maintenance after completion, because that would make no sense. If people need to modify or fix the program, they’ll do it. Programs solve well-defined problems, and then their authors move on to other things– no God Programs that accumulate requirements, but simple programs designed to do one thing and do it well. The programmer-to-program relationship must be one-to-many. Programmers write programs that do well-defined, comprehensible things well. They solve problems. Then they move on. This is a great way to build software “as needed”, and the only problem with this style of development is that the importance of small programs is hard to micromanage, so managerial dinosaurs who want to track efforts and “headcount” don’t like it much, because they can never figure out who to scream at when things don’t go their way. It’s hard to commoditize programmers when their individual contributions can only be tracked by their direct clients, and when people can silently be doing work of high importance (such as making small improvements to the efficiencies of core algorithms that reduce server costs). The alternative is to invert the programmer-to-program relationship: make it many-to-one. Then you have multiple programmers (now a commodity) working on Giant Programs that Do Everything. This is a terrible way to build software, but it’s also the one historically favored by IDE culture, because the sheer work of setting up a corporate development environment is enough that it can’t be done too often, and this leads managers to desire Giant Projects and a uniformity (such as a one-language policy, see again why “<X> Shops” suck) that managers like but that often makes no sense.

The right way of doing things– one programmer works on many small, self-contained programs– is the core of the so-called “Unix philosophy“. Big Programs, by contrast, invariably have undocumented communication protocols and consistency requirements whose violation leads not only to bugs, but to pernicious misunderstandings that muddle the original conceptual integrity of the system, resulting in spaghetti code and “mudballs”. The antidote is for single programs themselves to be small, for large problems to be solved with systems that are given the respect (such as attention to fault tolerance) that, as such, they deserve.

Are there successful exceptions to the Unix philosophy? Yes, there are, but they’re rare. One notable example is the database, because these systems often have very strong requirements (transactions, performance,  concurrency, durability, fault-tolerance) that cannot be as easily solved with small programs and organic growth alone. Some degree of top-down orchestration is required if you’re going to have a viable database, because databases have a lot of requirements that aren’t typical business cruft, but are actually critically important. Postgres, probably the best SQL out there, is not a simple beast. Indeed, databases violate one of the core tenets of the Unix philosophy– store data in plain text– and they do so for good reasons (storage usage). Databases also mandate that people be able to use them without having to keep up with the evolution of such a system’s opaque and highly-optimized internal details, which makes the separation of implementation from interface (something that object-oriented programming got right) a necessary virtue. Database connections, like file handles, should be objects (where “object” means “something that can be used with incomplete knowledge of its internals”.) So databases, in some ways, violate the Unix philosophy, and yet are still used by staunch adherents. (We admit that we’re wrong sometimes.) I will also remark that it has taken decades for some extremely intelligent (and very well-compensated) people to get databases right. Big Projects win when no small project or loose federation thereof will do the job.

My personal belief is that almost every software manager thinks he’s overseeing one of the exceptions: a Big System that (like Postgres) will grow to such importance that people will just swallow the complexity and use the thing, because it’s something that will one day be more important than Postgres or the Linux kernel. In almost all cases, they are wrong. Corporate software is an elephant graveyard of such over-ambitious systems. Exceptions to the Unix philosophy are extremely rare. Your ambitious corporate system is almost certainly not one of them. Furthermore, if most of your developers– or even a solid quarter of them– are commodity developers who can’t code outside of an IDE, you haven’t a chance.

Functional programs rarely rot

The way that most of us sell functional programming is all wrong. The first thing we say about it is that the style lacks mutable state (although most functional languages allow it). So we’re selling it by talking about what it doesn’t have. Unfortunately, it’s not until late in a programmer’s career (and some never get there) that he realizes that less power in a language is often better, because “freedom-from” can be more important than “freedom-to” in systems that will require maintenance. This makes for an ineffective sales strategy, because the people we need to reach are going to see the statelessness of the functional style as a deficit. Besides, we need to be honest here: sometimes (but not often) mutable state is the right tool for the job.

Often, functional programming is sold with side-by-side code snippets, the imperative example being about 30 lines long and relatively inelegant, while the functional example requires only six. The intent is to show that functional programming is superior, but the problem is that people are likely to prefer whatever is most familiar to them. A person who is used to the imperative style is more likely to prefer the imperative code, because they are more used to it. What good is a reduction to six lines if they seem impenetrable? Besides, in the real world, we use mutable state all the time: for performance and because the world actually is stateful. It’s just that we’re mindful enough to manage it well.

So what is it that makes functional programming superior? Or what is it that makes imperative code so terrible? The issue isn’t that state is inherently evil, because it’s not. Small programs are often easier to read in an imperative style than a functional one, just as goto improves the control flow of many small procedures. The problem, rather, is that stateful programs evolve in bad ways when programs get large.

A referentially transparent function is one that returns the same output per input. This, and immutable data, are the atoms of functional programming. Such functions actually have precise (and usually, obviously intended) semantics which means that it’s clear what is and what is not a bug, and unit testing is relatively straightforward. Contrast this with typical object-oriented software where an object’s semantics are the code, and it’s easy to appreciate why the functional approach is better. There’s something else that’s nice about referentially transparent functions, which is that they can’t be changed (in a meaningful way) without altering their interfaces.

Object-oriented software development is contrived to make sense to non-technical businessmen, and the vague squishiness associated with “objectness” appeals to that crowd. Functional programming is how mathematicians would prefer to program, and programming is math. When you actually care about correctness, the mathematician’s insistence on integrity even in the smallest details is preferable.

A functional program computes something, and intermediate computations are returned from one function and passed directly into another as a parameter. Behaviors that aren’t reflected in a returned value might matter for performance but not semantics. An imperative or object-oriented program does things, but the intermediate results are thrown away. What this means is that an unbounded amount of intermediate stuff can be shoved into an imperative program, with no change to its interface. Or, to put it another way, with a referentially transparent function, the interface-level activity is all one needs to know about its behavior. On the other hand, this means that for a referentially transparent function to become complex will require an equally complex interface. Sometimes the simplicity provided by an imperative function’s interface is desirable.

When is imperative programming superior? First, when a piece of code is small and guaranteed to stay small, the additional plumbing involved in deciding what-goes-where that functional programming can make the program harder to write.  A common functional pattern when state is needed is to “thread” it through referentially transparent functions by documenting state effects (that may be executed later) in the interface, which can complicate interfaces. It’s more intuitive to do a series of actions than build up a list of actions (the functional style) that is then executed. What if that list is very large? How will this affect performance? That said, my experience is that the crossover point at which functional programming becomes strictly preferable is low: about 100 lines of code at most, and often below 50. The second case of imperative programming’s superiority is when the imperative style is needed for performance reasons, or when an imperative language with manual memory management (such as C or C++) is strictly required for the problem being solved. Sometimes all of those intermediate results must be thrown away because there isn’t space to hold them. Mutable state is an important tool, but it should almost always be construed as a performance-oriented optimization.

The truth is that good programmers mix the styles quite a bit. We program imperatively when needed, and functionally when possible.

Imperative and object-oriented programming are different styles, and the latter is, in my view, more dangerous. Few programmers write imperative code anymore. C is imperative, Java is full-blown object-oriented, and C++ is between the two depending on who wrote the style guide. The problem with imperative code, in most business environments, is that it’s slow to write and not very dense. Extremely high-quality imperative code can be written, but it takes a very long time to do so. Companies writing mission-critical systems can afford it, but most IT managers prefer the fast-and-loose, sloppy vagueness of modern OOP. Functional and object-oriented programming both improve on the imperative style in terms of the ability to write code fast (thanks to abstraction) but object-oriented code is more manager-friendly, favoring factories over monads

Object-oriented programming’s claiming virtue is the ability to encapsulate complexity behind a simpler interface, usually with configurability of the internal complexity as an advanced feature. Some systems reach a state where that is necessary. One example is the SQL database, where the typical user specifies what data she wants but not how to get it. In fact, although relational databases are often thought of in opposition to “object-oriented” style, this is a prime example of an “object-oriented” win. Alan Kay’s original vision with object-oriented programming was not at all a bad one: when you require complexity, encapsulate it behind a simpler interface so that (a) the product is easier to use, and (b) internals can be improved without disruption at the user level. He was not saying, “go out and write a bunch of highly complex objects.” Enterprise Java is not what he had in mind.

So what about code rot? Why is the functional style more robust against software entropy than object-oriented or imperative code? The answer is inherent in functional programming’s visible (and sometimes irritating) limitation: you can’t add direct state effects, but have to change interfaces. What this means is that adding complexity to a function expands its interface, and it quickly reaches a point where it’s visibly ugly. What happens then? A nice thing about functional programming is that programs are build up using function composition, which means that large functions can easily be broken up into smaller ones. It’s rare, in a language like Ocaml or Clojure when the code is written by a competent practitioner, to see functions longer than 25 lines long. People break functions up (encouraging code reuse) when they get to that point. That’s a really great thing! Complexity sprawl still happens, because that’s how business environments are, but it’s horizontal. As functional programs grow in size, there are simply more functions. This can be ugly (often, in a finished product, half of the functions are unused and can be discarded) but it’s better than the alternative, which is the vertical complexity sprawl that can happen within “God methods”, and in which it’s unclear what is essential and what can be discarded. 

With imperative code, the additional complexity is shoved into a series of steps. For-loops get bigger, and branching gets deeper. At some point, procedures reach several hundred lines in length, often with the components being written by different authors and conceptual integrity being lost. Much worse is object-oriented code, with its non-local inheritance behaviors (note: inheritance is the 21st-century goto) and native confusion of data types and namespaces. Here, the total complexity of an object can reach tens of thousands of lines, at which points spaghettification has occurred.

Functional programming is like the sonnet. There’s nothing inherently superior about iambic pentameter and rhyming requirements, but the form tends to elevate one’s use of language. People find a poetic capability that would be much harder (for a non-poet) to find in free verse. Why? Because constraint– the right kind of constraint– breeds creativity. Functional programming is the same way. No, there’s nothing innately superior about it, but the style forces people to deal with the complexity they generate by forcing it to live at the interface level. You can’t, as easily, throw a dead rat in the code to satisfy some dipshit requirement and then forget about it. You have to think about what you’re doing, because it will effect interfaces and likely be visible to other people. The result of this is that the long-term destruction of code integrity that happens in most business environments is a lot less likely to occur.