An alternate theory of shark-jumping

I’ve watched a fair amount of TV in my life, seen quite a few movies, read a large number of books. A theme that becomes common in creative endeavor is “jumping the shark”, or the decline in creative quality that occurs when a series (of TV seasons, or sequential movies) seems to run out of creative steam and begins “grasping” desperately at new ideas– often, ideas that are not necessarily bad but completely incoherent with the flavor of the series– as it tries to stay relevant. I’m going to address shark-jumping: why it happens, and if there is a way to prevent it.

In the abstract

There are a number of reasons why a series might decline in quality with age, a phenomenon most prominently seen in TV series with undefined length. Why does it happen? The most common explanation given is that the show’s originators “run out of ideas”, as if there were a finite supply of them that each person gets for one lifetime. I don’t think this is adequate, for two reasons. The first is that not all creative people “jump”. Some novelists run out of ideas and peak early; others keep getting better into old age. It doesn’t seem to be that common for a person to actually “run out of ideas”; some creative people become complacent once they’re lifted into upper-middle-class social acceptance (which is hard to attain for a creative person!) but that’s a change of context rather than a natural decline, and it doesn’t happen to everyone. The second is that it’s not a sufficient explanation, in light of the first point. Specific creative people can remain fresh for 15 years, no problem. But almost no fictional TV series, no matter how skilled its people, can stay fresh for that long. Most don’t keep quality for a third of that time.

In fact, the more people and money involved in a creative production, the faster the shark-jumping process– which is the opposite of what you’d expect if it were merely a problem of people running out of ideas. Novelists can stay fresh for a lifetime, while TV series tend to jump the shark after 3-6 years on average. Movies tend to jump even more quickly than that– in the first sequel, except in planned series (e.g. those that were designed to be trilogies from the outset). Magic: the Gathering (which requires a large design team) jumped, in terms of thematic quality, when I was half my current age, but Richard Garfield’s new games are still good.

This suggests strongly that shark-jumping is about teams, not individuals. That makes sense. The “idea person” might remain brilliant, but if her team is full of hacks, she’ll be inclined to stick to the tried-and-true. That’s one pattern of shark-jumping, but probably not the most common. Equally or more common is the taking of more risks, but with the new creativity feeling gimmicky and forced. When The Office jumped, it began taking more risks, but was incoherent and haphazard in doing so. When House jumped, the characters’ personal lives became more unusual. Whether more risks or fewer risks are taken, a decline in quality happens either way.

If shark-jumping is about teams, then why not fire the old team and start with an entirely fresh set of people? Most often, that will only make things worse. Even if the people on the new team are paid four times as well, and even if they’re individually quite creative, I maintain that their output will (on average) be worse than if the old team had stayed (in which case decline would still occur). As a TV or movie series matures, the set of constraints laid down upon future creativity increases. That isn’t always bad. More rigid poetic forms like the sonnet (as opposed to free verse) often encourage creativity because the poet has to spend a lot more time thinking about words, and how they sound and flow together, than in typical prose. The same, I think, goes with serial creative work. The increasing constraint load, for some time, actually improves the product. In TV, Season 2 is typically better than Season 1. There is a point, however, when those constraints become a burden. Reasonable avenues of exploration become fewer as the story moves along. That’s not unnatural. In drama, we see that in the tragic arc: eventually, the protagonist reaches a point where the only remaining option is to surrender to the forces that have multiplied against him; the mortal dies, the gods win. In a television series intent on prolonging its life, however, this results in increasingly ridiculous ploys to get the main characters out of whatever final state– whether a positive one like marriage for a lothario, or a negative one like imprisonment or terminal illness– they’ve arrived at. This should also explain why series designed with a finite life in mind (such as Breaking Bad) rarely jump the shark. They’re programmed to end before that would happen.

As much as shark-jumping is about the increasing constraint load and the inverted-U shape of its effect on creative output, it’s also about people. Would the same calibre of people sign up to work on Inception II as worked on the original? I doubt it. It’d be possible to get good people, yes, but the best people would prefer to work on something more original than a sequel. You’d get more people who are there to burnish their resumes and fewer who are there to do the best creative work of their lives. Mature brands tend to draw people in with careerist rather than creative impulses: ability to lead a large group, attach one’s name to a known entity, etc. The average credibility (in terms of on-paper accomplishment and social status) goes up as the brand matures, and this might also improve the mean output, but it reduces variance. Thus, peak creative output is almost always lower in the brand’s later phases.

Therefore, a “fire the old team” strategy is likely to accelerate the shark-jumping problem, which is about the type of team that a series will attract more than the individuals themselves. The old-timers who had the vision are gone, and they’ve been replaced by people who are on the job for careerist reasons. In general, I’d say there’s nothing wrong with this– most people take most jobs for careerist reasons– but it’s not conducive to the highest levels of creative output. If there are still a couple of clever ways, for a series, out of no-credible-options-left shark-jump territory, a fresh team of mercenaries is not likely to find it. They’re likely to barge through walls, strain credibility, and make shark-jumping palpable in the final product.

It’s not that people “run out of ideas”. They don’t. Teams, however, lose members and gain new ones constantly. That’s inevitable. And if there’s one thing I’ll say confidently about creative people as an entire set, it’s that we’re intermittent. Something like a TV series requiring 600 minutes of show time (using the industry-standard 100:1 multiplier, that’s 1000 hours of production time) requires a creative team, because even the best individuals can’t hit every note right over that duration without some help. So, at least in television, even the best of visionary creators needs the support of (and challenges from) a strong team to keep going. And no matter what, that team will evolve in a direction that’s likely to be sharkward. The new team might be paid four times as much as the old one but, by Season 7, almost no one’s focus is on the work in front of them. Rather, they’re more interested in Season 1 of their next project, where they’ll have more input and opportunity to shine. This career incoherency (disparity between what’s good for their jobs vs. their careers) doesn’t actually cause them to “run out of ideas”. More often, it’s the reverse. They (probably subconsciously, for the most part) take risks that may confer personal career benefits, but that go against the grain of what the series is really about.

In software

That this applies to software, also, should not surprise anyone. Like a television series, software is designed with an indefinite lifespan in mind. There is somewhat of a difference, which is that software doesn’t always jump the shark. In the open-source world, it’s far less likely to do so. However, I think that commercial software obeys the same principle of shark gravity. After a certain point, a corporate software module will be in maintenance mode and struggle to attract a high calibre of people.

There are people who will hack the Linux Kernel or Postgres or Clojure because they use those products and care about them deeply. Open-source software is, in truth, a brilliant solution to the career-coherency problem: people can benefit their careers and add value to the world. Such software can jump the shark, but I don’t think it’s guaranteed to do so, and the best software products seem never to jump. There are career benefits to maintaining a respected open-source product, and the fact that the maintainer is also typically a user (and, therefore, aware of existing context) prevents the bad creative risks for which post-shark teams are known.

In-house or commercial software, on the other hand, seems always to jump. Within most companies, however, the career payoff of maintenance work is almost invariably inferior to that of new invention. Open-source software solves the career coherency problem, but internal products almost never become respected enough for that to happen. Software’s shark-jumping dynamic is, in many ways, actually more severe than that of a TV series. In television, the people who join a mature series aren’t necessarily less qualified or worse at their jobs– they have different objectives that are less conducive to doing their best work, but they’re not across-the-board less qualified people. In closed-allocation software companies, however, maintenance usually becomes the ghetto for people who can’t fight their way to something meatier, and very few people who are any good will stay with it for very long.

Rarely, if ever, is a closed-allocation software company able to solve this problem. When the company recognizes that a legacy module is important, it will commit resources to its upkeep, usually in two ways. The first is to increase headcount, and the second is to increase salaries of the people doing the work. On the first, that tends to attract less capable people for two reasons. The incompetent like larger teams because they can “hide within the herd”. But this applies to middle managers as well as front-line workers. Mediocre managers also prefer large teams because it inflates their headcount statistics; they’re more likely to make Director if they can say they had ten reports than if they had three. Good managers generally want to lead teams of high average competence and achieve something tangible; mediocre and bad managers usually want to lead large teams (with minimal concern over whether they get good reports or bad) to improve their stats. So that first solution fails to have the desired effect. What about the second, which is increasing the pay of the maintenance staff? That rarely works, either. The truth is that a savvy, capable software engineer can’t be motivated to do career-incoherent work with a one-time 20 percent– or even 50 percent– bonus. The opportunity cost for her (in not doing work that will advance her career) is too great. She might be appeased with a permanent 30% salary bump, for a year or two, but then that will become “the new normal” for her compensation and she’ll need another bump. But HR is not about to let the salary for a lowly “Software Engineer III” to go that far out of band, and promoting her (to compensate for unpleasant work, regardless of whether she meets the typical criteria for promotion) will often annoy engineers who will (accurately) perceive the promotion as “political”. Even if engineers abstractly agree that undesirable work deserves a reward, they’ll usually oppose a promotion (especially if it is over them) that appears to be given for doing grunt work that is (and because it is) unpleasant rather than technically impressive. So that’s untenable, too. How does the typical closed-allocation software company solve that maintenance problem? The rewards generally all go to the “heroic” middle manager (who usually takes the project on for a second or third chance, after failing at new invention) for “rescuing” the ailing legacy module. In the large closed-allocation software titans, these awards (again, to the managers of the maintenance projects, and rarely to the teams) can reach six or seven figures. The peons get nothing; they’re just doing their jobs and, in the typical closed-allocation hellhole, their managers can easily prevent them from having other options.

That the above doesn’t work, at all, shouldn’t surprise anyone. I’ve already said a lot about that topic here and here, so I won’t lengthen this particular essay by reviewing it.

In sum, shark-jumping (whether in television, or in software) occurs because of two reasons, neither of which requires an individual to “run out of ideas” (we know that that doesn’t always happen). The first pertains to the constraints imposed by the project’s history. At first, constraint is conducive to superior creativity– that’s why most poets are better in rigid forms than in free verse– but, at some point, the complexity load gets to a point where high-quality options have been exhausted. The second, and probably more inexorable, factor is the change in team dynamics. As a brand matures or a software module goes into maintenance mode, the evolution in the motivational profile (that is, why the team is there) is enough to bring on the shark.

What is the solution? For television, the best solution seems to be to let the narrative arc tend toward its natural close– and not to senselessly prolong the life of the series. Breaking Bad did that and never jumped, but with another season, it probably wouldn’t have been as good. Software doesn’t have that option, because it’s designed to be infrastructural in nature. It should mature to a point where it “just works” from the perspective of 99+ percent of those who interact with it. The issue is that someone will have to maintain it. In software, the only incentive system that seems to work– i.e. the only one that can solve the otherwise-crippling career-coherency issues of maintenance work– is the open-source economy.

Creativity and the nautilus

Creativity is one of those undeniably positive words that is reflexively associated with virtue, leadership, intelligence and industry (even though I’ve known some creative people who lack those traits). Yet, 28 years of experience has taught me that people don’t have much patience for creative people. We are like the nautilus, a strange-looking and, some would say, unattractive creature that leaves behind a beautiful shell. Most people are happy to have what we create, but would prefer never to see the marine animal that once lived inside the art. All they want to do is pick it up once left on the shore.

Creativity isn’t in-born and immutable. It’s an attribute of which most people end up using (and, in the long-term, retaining) less than 1 percent of their natural potential. It grows over time as a result of behaviors that most people find strange and potentially unpleasant. Creative people are curious, which means that they often seek information that others consider it inappropriate for them to have. They’re passionate, which means they have strong opinions and will voice them. Neither of these is well-received in the typical conservative corporate environment. Worse yet, creative people are deeply anti-authoritarian, because it’s simply not possible to become and stay creative by just following orders, not when those orders impose compromise over a substantial fraction of one’s waking time. It never has been, and it never will be.

This doesn’t mean that creativity is only about free expression and flouting convention. Creativity has divergent and convergent components. An apprentice painter’s more abstract work may seem like “paint splatter”, but there’s a divergent value in this: she’s getting a sense of what randomness and play “look like”. She might do that in January. In February, she might do a still life, or an imitation of an existing classical piece. (Despite negative connotations of the word, imitation was an essential part of an artist’s education for hundreds of years.) It’s in March that she produces art of the highest value: something original, new, and playful (divergence) that leverages truth and value discovered by her predecessors, trimmed using rules (convergence) they teach. Computationally, one can think of the divergent component as free expansion and the convergent aspect as pruning. The convergent aspect of creativity requires a lot of discipline and work. Novelists refer to this process as “killing your darlings”, because it involves removing from a narrative all of the characters or themes that an artist inserts for personal (often emotional) reasons but that add little merit to the completed work. For technology, Steve Jobs summarized it in three words: “Real artists ship”. It’s intellectually taxing, and many people can’t do it.

Convergent creativity is what our culture’s archetype of “the artist”, for the most part, misses. Youthful “experience chasing”, social climbing, extremes either of fame and wealth or of miserable, impoverished obscurity, all have some divergent value. On the other hand, that type of stock-character artist (or real Williamsburg hipster) has almost no hope of producing anything of artistic value. Divergence alone leads to mutation and insanity, not creation. This also explains why the highest-priced “modern art” is so terrible. We have a culture that worships power, money, and social access for their own sake and so the “brand-name artists” whose connections enable them to be paid for divergence (and divergence only) are treated as high priests by our supposed cultural leaders. The result of this is that terrible art sells for astronomical sums, while real artists often work in obscurity.

Most people, when they think of creative workers, whether we’re talking about writers or game designers or computer programmers, only seem to understand the divergent part. That gives them the impression that we’re a coddled bunch with easy jobs. We “get paid to have fun”. To quote Mad Men from the perspective of an “account man”, we’re “creative crybabies”. Bullshit. Creativity is rewarding and it can be a lot of fun, but it also requires an incredible amount of work, often at inconvenient hours and requiring very high levels of sustained effort. Wake up at 5:00 am with a great idea? Get to work, now. If that idea comes at 10:30 pm instead, stay up. Most of us work a thousand times harder than “executives”, the private-sector bureaucrats whose real jobs are white-collar social climbing.

As with the adiabatic cooling of an expanding gas, and the heating of a compressed one, divergence has a cooling effect while convergence is hot. For the first, free writing tends to calm the nerves and diminish anger. Improvisational art is deeply anxiolytic and invigorating. But if it is taken too far without its counterpart, divergence leads to softness and ennui. A metaphor that programmers will understand is the concept of writing a 100,000-line program on paper, or in a featureless text editor (e.g. NotePad) without an appropriate compiler or interpreter. In the absence of the exacting, brutal convergence imposed by the compiler (which rejects programs that don’t make sense) the programmer is deprived of the rapid and engaging divergence/convergence cycle on which programmers depend. Over time, the hand-written program will accumulate shortcuts and errors, as the programmer loses touch with the substrate on which the program will run. Related to divergence’s cooling effect is convergence’s heating effect. It’s very taxing to hack branches off of one’s “search tree”. It’s painful. As humans, we’re collectors and we don’t like setting things we’ve found out of view, much less discarding them. About two to three hours of this work, per day, is more than most people can reliably perform. Although extremely important, the convergent aspect of creativity is exhausting and leads to frayed nerves and difficult personalities.

Business executives’ jobs are social climbing, internal and external. Their jobs are intentionally made extremely easy to minimize the probability of social hiccups, for which tolerance is often zero. They’re paid five times as much as any person could possibly need, to eliminate the possibility of financial worry and allow them to purchase ample domestic services. They’re given personal assistants to remove pollution and stress from their communication channels. This makes sense in the context of what companies need from their executives. He only needs to have average intelligence and social skill, but he needs to sustain this level reliably, 24/7, under a wide range of unpredictable circumstances. Rare moments (“outlier events”) are where creative people earn their keep, but where executives set themselves on fire. To be an executive, one needs to be reliable. Executives may have the authority to set working hours, but they have to be on time, all the time, every day.

For a contrast, the creative person often “needs a walk” at 3:00 in the afternoon and has no qualms about leaving work for an hour to enjoy an autumn day. After all, he does his best work at 7:00 am (or 11:30 pm) anyway. Creative people earn their keep in their best hours, and are wasting time in the mediocre ones. From an executive perspective, however, this behavior is undesirable. Why’d he up-and-leave at 3:00? There could have been a crisis! What risk! By definition, creative people are full of risk, socially gauche on account of their tendency to mentally exhaust themselves in their desire to produce something great, and therefore not reliable. Clearly “not executive material”, right? I disagree, because I think the discrepancy is more of role than of constitution. Most creative people could become tight-laced executives if they needed to do so. Creatively inclined entrepreneurs often do find themselves in this role, and many play it well. There is, however, an either/or dynamic going on. It’s necessary to choose one role and play it well, rather than attempting both and being very likely to fail. Someone who is exhausted on a daily basis by the demands of high-level creativity (and I don’t mean to say that creative work isn’t rewarding; it is, but it’s hard) can’t be a reliable people-pleaser in the same week, any more than a clay jar that just came out of a 1000-degree oven shouldn’t be used, until it cools, to store ice cream.

For this reason, business executives tend to look at their most creative underlings and see them as “not executive material”. Well, I agree this far: not now. That perception alone wouldn’t be such a problem, except for the fact that people tend to judge others’ merit on a “just like me” basis, especially as they acquire power (ego confirmation). They tend to be blind to the possibility that people very different from them can be just as valuable. Executives tend, for example, to favor mediocre programmers who don’t exert themselves at work, and who therefore retain enough mental energy for social polish. The technical mediocrity that sets in over most software companies is the long-term result of this. It’s not that companies go out of their way to “promote idiots”; it’s that they are designed to favor reliable career politicians over “risky” hard workers whose average performance is much higher, but whose lows are lower.

Would a creative person make a good business executive? I believe so, but with a specific variety of mentoring. Creative people given management positions generally find themselves still wanting to do the non-managerial work they did before the promotion. The problem here is that the manager can easily end up hogging the “fun work” and giving the slag to his subordinates. This will make him despised by those below him, and his team will be ineffective. That’s the major obstacle. Creative people tapped for management need to be given a specific ultimatum: you can go back to [software programming | game design | writing] full-time in 6 months and keep the pay increase and job title, but for now your only job is leadership and the success of your team. Good managers often find themselves taking on the least desired work in order to keep the underlings engaged in their work. It’s unpleasant, but leadership isn’t always fun.

I’ll go further and say that creative discipline is good practice for business leadership. Business leaders can’t afford to be creative at the fringe like artists or computer programmers, because the always-on, reliable social acumen they require precludes this kind of intellectual exhaustion, but the processes they need to manage are like an externalized version of the self-management that creative people need to develop in order to succeed. They must to encourage their subordinates to grow and test new ideas (divergence) but they also need to direct efforts toward useful deliverables (convergence). Many managers who fail do so because they only understand one half of this cycle. Those who focus only on convergence become the micromanagers who complain that their subordinates “can never do anything right” and who rule by intimidation. They fail because the only people who stay with them are those who actually can’t do anything right. The too-nice bosses who focus solely on divergence will either (a) fail to deliver, and be moved aside, or (b) are unprepared when young-wolf conflicts tear apart their organization. So, although the creative personality appears unsuited for managerial roles in large organizations, I’d argue that creative experience and discipline are highly valuable, so long as the manager is prepared to temporarily sacrifice the creative lifestyle hor professional purposes.

The results of creativity are highly valued, although sometimes only after an exhausting campaign of persuasion. The creative process is deeply misunderstood, creative roles are targets of misplaced envy because of their perceived “easiness”, and the people who have the most creativity (the top 0.1%) are rarely well-liked. People want our ideas, but not us and our “attitudes”. The shells, not the gnarly creatures. I invoke Paul Graham’s explanation for why “nerds” are rarely popular in high school. His explanation is that most high schoolers make a full-time job of being popular, while the nerds are too busy learning how to do hard things, such as programming computers or playing musical instruments. We see the same thing (the high-school social dynamic) in the slow-motion cultural and moral terrorist attack that we call “Corporate America”. People can invest themselves into working really hard in pursuit of the chance (and it’s far from a certainty) of producing something truly great. Or they can throw themselves wholesale into office politics and grab the gold. The world is run by those who chose the latter, and the cultural, artistic and moral impoverishment of our society is a direct result of that.