On programmers, deadlines, and “Agile”

One thing programmers are notorious for is their hatred of deadlines. They don’t like making estimates either, and that makes sense, because so many companies use the demand for an estimate as a “keep ’em on their toes” managerial microaggression rather than out of any real scheduling need. “It’ll be done when it’s done” is the programmer’s gruff, self-protecting, and honest reply when asked to estimate a project. Whether this is an extreme of integrity (those estimates are bullshit, we all know it) or a lack of professionalism is hotly debated by some. I know what the right answer is, and it’s the first, most of the time.

Contrary to stereotype, good programmers will work to deadlines (by which, I mean, work extremely hard to complete that project by a certain time) under specific circumstances. First, those deadlines need to exist in some external sense, i.e. a rocket launch whose date has been set in advance and can’t be changed. They can’t be arbitrary milestones set for emotional rather than practical reasons. Second, programmers need to be compensated for the pain and risk. Most deadlines, in business, are completely arbitrary and have more to do with power relationships and anxiety than anything meaningful. Will a good programmer accept business “deadline culture”, and the attendant risks, at a typical programmer’s salary? No, not for long. Even with good programmers and competent project management, there’s always a risk of a deadline miss, and the great programmers tend to be, at least, half-decent negotiators (without negotiation skills, you don’t get good projects and don’t improve). My point is only that it can happen for a good programmer to take on personal deadline responsibility (either in financial or career terms) and not find it unreasonable. That usually involves a consulting rate starting around $250 per hour, though.

Worth noting is that there are two types of deadlines in software: there are “this is important” deadlines (henceforth, “Type I”) and “this is not important” deadlines (“Type II”). Paradoxically, deadlines are assessed to the most urgent (mission critical) projects but also to the least important ones (to limit use of resources) while the projects of middling criticality tend to have relatively open timeframes.

A Type I deadline is one with substantial penalties to the client or the business if the deadline’s missed. Lawyers see a lot of this type of deadline; they tend to come from judges. You can’t miss those. They’re rarer in software, especially because software is no longer sold on disks that come in boxes, and because good software engineers prefer to avoid the structural risk inherent in setting hard deadlines, preferring continual releases. But genuine deadlines exist in some cases, such as in devices that will be sent into space. In those scenarios, however, because the deadlines are so critical, you need professional project-management muscle. When you have that kind of a hard deadline, you can’t trust 24-year-old Ivy grads turned “product managers” or “scrotum masters” to pull it out of a hat. It’s also very expensive. You’ll need, at least, the partial time of some highly competent consultants who will charge upwards of $400 per hour. Remember, we’re not talking about “Eric would like to see a demo by the 6th”; we’re talking about “our CEO loses his job or ends up in jail or the company has to delay a satellite launch by a month if this isn’t done”. True urgency. This is something best avoided in software (because even if everything is done right, deadlines are still a point of risk) but sometimes unavoidable and, yes, competent software engineers will work under such high-pressure conditions. They’re typically consultants starting around $4,000 per day, but they exist. So I can’t say something so simple as “good programmers won’t work to deadlines”, even if it applies to 99 percent of commercial software. They absolutely will– if you pay them 5 to 10 times the normal salary, and can guarantee them the resources they need to do the job. That’s another important note: don’t set a deadline unless you’re going to give the people expected to meet it the power, support, and resources to achieve it if at all possible. Deadlines should be extremely rare in software so that when true, hard deadlines exist for external resources, they’re respected.

Most software “deadlines” are Type II. “QZX needs to be done by Friday” doesn’t mean there’s a real, external deadline. It means, usually, that QZX is not important enough to justify more than a week of a programmer’s time. It’s not an actual deadline but a resource limit. That’s different. Some people enjoy the stress of a legitimate deadline, but no one enjoys an artificial deadline, which exists more to reinforce power relationships and squeeze free extra work out of people than to meet any pressing business need. More savvy people use the latter kind as an excuse to slack: QZX clearly isn’t important enough for them to care if it’s done right, because they won’t budget the time, so slacking will probably be tolerated as long as it’s not egregious. If QZX is so low a concern that the programmer’s only allowed to spend a week of his time on it, then why do it at all? Managers of all stripes seem to think that denying resources and time to a project will encourage those tasked with it to “prove themselves” against adversity (“let’s prove those jerks in management wrong… by working weekends, exceeding expectations and making them a bunch of money”) and work hard to overcome the gap in support and resources between what is given and what is needed. That never happens; not with anyone good, at least. (Clueless 22-year-olds will do it; I did, when I was one. The quality of the code is… suboptimal.) The signal sent by a lack of support and time to do QZX right is: QZX really doesn’t matter. Projects that are genuinely worth doing don’t have artificial deadlines thrown on them. They only have deadlines if there are real, external deadlines imposed by the outside world, and those are usually objectively legible. They aren’t deadlines that come from managerial opinion “somewhere” but real-world events. It’s marginal, crappy pet projects that no one has faith in that have to be delivered quickly in order to stay alive. For those, it’s best to not deliver them and save energy for things that matter– as far as one can get away with it. Why work hard on something the business doesn’t really care about? What is proved, in doing so?

Those artificial deadlines may be necessary for the laziest half of the programming workforce. I’m not really sure. Such people, after all, are only well-suited to unimportant projects, and perhaps they need prodding in order to get anything done. (I’d argue that it’s better not to hire them in the first place, but managers’ CVs and high acquisition prices demand headcount, so it looks like we’re stuck with those.) You’d be able to get a good programmer to work to such deadlines around $400 per hour– but because the project’s not important, management will (rightly) never pay that amount for it. But the good salaried programmers (who are bargains, by the way, if properly assigned or, better yet, allowed to choose their projects) are likely to leave. No one wants to sacrifice weekends and nights on something that isn’t important enough for management to budget a legitimate amount of calendar time.

Am I so bold as to suggest that most work people do, even if it seems urgent, isn’t at all important? Yes, I am. I think a big part of it is that headcount is such an important signaling mechanism in the mainstream business world. Managers want more reports because it makes their CVs look better. “I had a team of 3 star engineers and we were critical to the business” is a subjective and unverifiable claim. “I had 35 direct reports” is objective and, therefore, more valued. When people get into upper-middle management, they start using the terms like “my organization” to describe the little empires they’ve built inside their employers. Companies also like to bulk up in size and I think the reason is signaling. No one can really tell, from the outside, whether a company has hired good people. Hiring lots of people is an objective and aggressive bet on the future, however. The end result is that there are a lot of people hired without real work to do, put on the bench and given projects that are mostly evaluative: make-work that isn’t especially useful, but allows management to see how the workers handle artificial pressure. Savvy programmers hate being assigned unimportant/evaluative projects, because they have all the personal downsides that important ones do (potential for embarrassment, loss of job) but no career upside. Succeeding on such a project gets one nothing more than a grade of “pass“. For a contrast, genuinely important projects can carry some of the same downside penalties if they are failed, obviously, but they also come with legitimate upside for the people doing them: promotions, bonuses, and improved prospects in future employment. The difference between an ‘A’ and a ‘B’ performance on an important project (as opposed to evaluative make-work) actually matters. That distinction is really important to the best people, who equate mediocrity with failure and strive for excellence alone.

All that said, while companies generate loads of unimportant work, for sociological reasons it’s usually very difficult for management to figure out which projects are waste. The people who know that have incentives to hide it. But the executives can’t let those unimportant projects take forever. They have to rein them in and impose deadlines with more scrutiny than is given important work. If it an important project over-runs its timeframe by 50 percent, it’s still going to deliver something massively useful, so that’s tolerated. What tends to happen is that the important projects are, at least over time, given the resources (and extensions) they need to succeed. Unimportant projects have artificial deadlines imposed to prevent waste of time. That being the case, why do them at all? Obviously, no organization intentionally sets out to generate unimportant projects.  The problem, I think, is that when management loses faith in a project, resources and time budget are either reduced, or just not expanded even when necessary. That would be fine, if workers had the same mobility and could also vote with their feet. The unimportant work would just dissipate. It’s political forces that hold the loser project together. The people staffed on it can’t move without risking an angry (and, often, prone to retaliation) manager, and the manager of it isn’t likely to shut it down because he wants to keep headcount, even if nothing is getting done. The result is a project that isn’t important enough to confer the status that would allow the people doing it to say, “It’ll be done when it’s done”. The unimportant project is in a perennial race against loss of faith in it from the business, and it truly doesn’t matter to the company as a whole whether it’s delivered or not, but there’s serious personal embarrassment if it isn’t made.

It’s probably obvious that I’m not anti-deadline. The real world has ’em. They’re best avoided, as points of risk, but they can’t be removed from all kinds of work. As I get older, I’m increasingly anti-“one size fits all”. This, by the way, is why I hate “Agile” so vehemently. It’s all about estimates and deadlines, simply couched in nicer terms (“story points”, “commitments”) and using the psychological manipulation of mandatory consensus. Ultimately, the Scrum process is a well-oiled machine for generating short-term deadlines on atomized microprojects. It also allows management to ask detailed questions about the work, reinforcing the sense of low social status that “conventional wisdom” says will keep the workers on their toes and most willing to work hard, but that actually has the opposite effect: depression and disengagement. (Open-back visibility in one’s office arrangement, which likewise projects low status, is utilized to the same effect and, empirically, it does not work.) It might be great for squeezing some extra productivity out of the bottom-half– how to handle them is just not my expertise– but it demotivates and drains the top half. If you’re on a SCRUM team, you’re probably not doing anything important. Important work is given to trusted individuals, not to SCRUM teams.

Is time estimation on programming projects difficult and error-prone? Yes. Do programming projects have more overtime risk than other endeavors? I don’t have the expertise to answer that, but my guess would be: probably. Of course, no one wants to take on deadline risk personally, which is why savvy programmers (and almost all great programmers are decent negotiators, as discussed) demand compensation and scope assurance. (Seasoned programmers only take personal deadline risk with scope clearly defined and fixed.) However, the major reason for the programmers’ hatred of deadlines and estimates isn’t the complexity and difficulty-of-prediction in this field (although that’s a real issue) but the fact that artificial deadlines are an extremely strong signal of a low-status, unimportant, career-wasting project. Anyone good runs the other way. And that’s why SCRUM shops can’t have nice things.

12 thoughts on “On programmers, deadlines, and “Agile”

  1. Ha I have tried to think of a better term for SCRUM, SCROTUM seems to fit it well. I can’t believe its now in its 10th year? of popularity. My only reasoning for its existence is the complete incompetence of the people that implement it. It gives those abundant but quasi managers something to feel good about since they are doing something even though that something is only inhibiting effective developers.

    No one will ever admit to being blocked since that will be focal point of Mr Incompetence but the best part about that is being blocked never has a chance of it actually occurring. SCROTUM teams are so poorly managed there isn’t much of a chance of cross development and if there is communication will always be done outside of SCROTUM meetings to stay away from the lime light (if they are smart).

    Unfortunately I don’t see SCROTUM going anywhere soon. All the devs I have encountered seem to tolerate it for some reason. I think it because they don’t wanted to be labeled as not a team player or something to that effect. Whatever the reason is I have never encountered a good reason for SCROTOM management. I can only hope calling it for what it is will help.

  2. I’m nowhere near as against Agile as you are, but I think you may be on to the problem. Per the Agile Manifesto, agile is about getting work done and correctly identifying what work needs to be done. I’ve been a Scrum Master within a scrum development environment myself, and deadlines in that environment are in fact completely arbitrary, each sprint is a predefined size and is a deadline in and of itself.

    Estimation on the part of the software engineer then becomes a task of “what can I do in the allotted time.” Breaking down the backlog and tasks into estimable pieces becomes the core and most important estimating skill to develop.

    When you get a team that knows their bounds and can estimate their work accurately, you can really make Scrum work well. Key caveats are that you must honor the estimates and not lobby to pull them in and shorten them. Because, and I’ve even noted this to management, “when you challenge us that something can be done faster, the inclination (even for me) is to take it as a challenge and allow the date to be pulled forward, for no reason except pride.” This pride of course always proceeds a fall. So estimates must not be questioned is a scrum rule that has to hold, or you will spin out of control. But, if you hold the line on lobbying for earlier dates and honor the estimates at all times, eventually software engineers can get very good at measuring their production and output.

    Although I’ve found that only teams of excellent developers can reliably make this work over longish stretches of time and it may be arguable that excellent developers really to not need as many structures governing them as average developers (i.e. they can handle agile scrum because they are excellent and capable and work within the structure just fine, but perhaps the structure isn’t really even necessary).

    One point I like to make often to management is: if you take as a given that everyone is working as hard as they can to get the job done, then what point is having the deadline, it’ll be done when its done.

    Where I currently work we have schedules and milestones, but they are hugely fungible. That is due primarily to the nature of the work. We’d be well capable of working under an agile framework, but what is desired by management is not really applicable to agile, however the date demands are realized as being wishful thinking (or best guesses) and are subject to change when circumstances dictate. You should be able to guess from that what type of work I do in a general sense. Its certainly not the development of packaged salable software, where I do agree all the problems you stated run rampant. When I was younger, I tried that world and found it extremely lacking.

    I did at one point attempt to go into developing software that would have become salable boxed software, but my proposal took into account what you stated about that type of work requiring premium pricing for its development against tight deadlines, and the decision to undertake the project was rejected by the backers I proposed it to, largely on price.

  3. Off topic alert.

    I just watched: HTML5DevConf: Jeffrey Newman “Recruiters: The Good, The Bad, and the Devious.” https://www.youtube.com/watch?v=0wtM2di5k5g&index=13&list=WL

    First of all It’s interesting to see how people on the other side of the fence thin. Secondly there are so many pathologies in the industry I don’t know where to start. Michael, I wonder what your comments would be for the video / speaker.

  4. Any thoughts on this sad, but rather predictable, development?

    Taxpayer Money to Flow Into Startups Through California Universities (http://valleywag.gawker.com/taxpayer-money-to-flow-into-startups-through-california-1596746596/+aleksanderchan)

    “University of California President Janet Napolitano announced today the repeal of a 25-year-old policy that “barred the university from investing directly in companies that commercialize technology that has emerged through UC research.” Having universities invest taxpayer money into startups may sound risky, but Napolitano has a plan. The UC president also announced the creation of the”UC Innovation Council” to help ensure that the university doesn’t pump funds into any bad investment. The outside council will be made up of innovators ranging from “investment and business executives, venture capitalists and technology experts.”

    • Committees make poor investment decisions. The collusion in the VC-funded world is *why* it does such a shitty job for its passive investors (i.e. the ones paying, rather than receiving, 2-and-20).

      As you have more people involved in a funding or hiring decision, you’re more likely to select a low-variance people pleaser instead of a high-variance innovator (who takes stands and risks offending people because he almost can’t help himself). It’s a silent drain, but devastating in the long run.

  5. I’d never made the connection between “low priority project” and “strictness of arbitrary deadlines” as the project having to continually race against its’ own insignificance. That’s a fascinating insight and raises a number of questions.

    For example, ‘crunch’ is something that’s endemic in the games industry and a contributing factor is that big publishers will impose arbitrary deadlines on their studios to ‘ship a game’. So, what does this say about the significance of these projects? Also, are the deadlines really useless, or do they have some real basis? (EA is not going to go broke because a Need For Speed game is a few months late, for example).

    On the other subject of programmers being bad at estimation. I’ve done a hell of a lot of introspection with regard to this and why I find estimates so unpleasant. A few reasons that come to mind:

    – When you’re asked for an estimate, what you’re often really being asked for is a deadline.

    – In a sprint planning session, I’d often get asked to produce an estimate for “how long will it take to design this?” followed by an estimate of “so how long will it take to implement?”. Which makes no sense – I can’t answer the second question without answering the first. This often came up with bug fixes or ‘small’ jobs – things that could reasonably be assumed to fall within a single sprint for at least some of the time.

    – I feel like there’s an unspoken rule that your estimates relative to other team members signal your competency to the team and to your manager. So if you’re constantly providing estimates that are longer than those made by other team members, it’s a bit like you’re saying “I’m less capable than them”. You feel encouraged to provide shorter estimates (to make ‘everyone happy’) and then hide the difference. I see this come up most starkly when sprint planning wants consensus estimates, despite having team members of varying skill levels in the particular task being estimated.

    – I find that ‘watching the clock’ breaks my flow. If I’m having to track my own time, it’s like a bit of cognitive load that I don’t want. I don’t want to have to think about it, because losing your sense of time is so crucial to really being in-flow. Often tools will ask you to enter how many hours you worked on something as part of tracking. So you end up even guesstimating at that (“well it felt like I spent half a day working on it, so I’ll put in 4 hours”).

    – As a result of the previous point, I don’t really collect historical data that I can reflect on. It’d be great to be able to go back and see how long it actually took me to do stuff. Not to mention that when you leave a company, you generally lose access to any historical performance information like that which you may have built up over the years. For example, I have about two years of Jira task estimate history locked away at the old company I used to work for, that I’ll never get to reflect upon.

    There is a lot about Scrum that I like, but as a whole I agree it still suffers from the arbitrary deadlines problem. I wonder if there’s a business acceptable way that you could do sprint-like activity, eschewing the idea of time estimation and replacing it with effort estimation only. Rather than aggressively fixing the time through schedule, aggressively fix the scope of work and then just work towards completion “until it’s done”.

  6. In a magical world, all work will be done instantaneously. Managers wish they live in this magical world, because if no work is produced by their underlings, then these middle managers will be seen to not be “producing” any work at all.

    The problem here is that managers themselves need to get their “hands dirty”. They need to be involved in writing code and developing software. They need to lead by example, and perhaps this will give them some level of “empathy” for their underlings when they demand deadlines from them.

    And the next problem are the Product Managers. There is a need for a “Product Manager”, but this should be a role, not a position, in a team. Today, in the tech world, as you mentioned, we see that “Product Managers” are the new Investment Banking Analyst or Sales & Trading Analysts, stuffed by fresh-faced Ivy/WASP-y grads.

    Good eng managers will step up and protect their underlings from the demands of the “Product Managers”. It is the Eng Manager who should do all the “deadline negotiations” with the PMs, and shield his/her underlings from being “managed” by the PMs.

    The only reason I would see for “deadlines” on micro-pieces of the project is the need for putting out some version of the product as quickly as possible, as is feasible, in order to evaluate its validity, and get the feedback needed to iterate for the next version. There is value to the success of a product by how rapidly it can be iterated based upon feedback from the marketplace. This will need, however painful, “deadlines” so that a version of the product can make it out of the door.

  7. I have to admit, although I am a big fan of Agile (and Scrum OK :)), you well-structure your arguments on this one. I never thought of categorizing deadlines or projects until this hit me. However, I don’t really agree with “If you’re on a SCRUM team, you’re probably not doing anything important. Important work is given to trusted individuals, not to SCRUM teams.”
    Maybe that’s why developers working on a SCRUM team are *for* and not against Agile and SCRUM. Because they want to defend their work, their teammates, their individual contribution to the team’s work…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s