Statistics, cooperation, politics, and programming.

Open: a simple dice “game”

Let’s say that you’re playing a one-player “game”, where your payout (score) is determined according to the rolls of 101 dice. One of them is black and 100 are white, and your payoff is \$100 times the value on the black die, plus the sum of the values on the 100 white dice. (In RPG terms, that’s d6x100 + 100d6.) The question is: how much more “important” (I’ll define this, more rigorously, below) is the black die, relative to a single one of the white dice?

Most people would say that the black die is 100 times as important; its influence on the payoff is a \$500 swing (from \$100 to \$600) while each of the white dice has a \$5 swing (\$1 to \$6). That would lead us to conclude that the black die is equally important as the hundred white dice, all taken together– or that the black die has 50% of the total importance. That’s not true at all. Why not? Let’s do some simulations. Here’s the code (in Clojure).

``` (defn white-die [] .  (inc (rand-int 6)))```

```(defn black-die [] .   (* 100 (inc (rand-int 6))))```

``` (defn play [] .   (let [bd-value (black-die) .         wd-value (reduce + 0 (repeatedly 100 white-die))] .      (printf "Black die: %d, White dice: %d, Payoff: %d\n" .               bd-value wd-value (+ bd-value wd-value)))) ```

Here are some results:

``` user=> (play) Black die: 100, White dice: 345, Payoff: 445 nil ;; other returns omitted. user=> (play) Black die: 600, White dice: 343, Payoff: 943 user=> (play) Black die: 400, White dice: 352, Payoff: 752 user=> (play) Black die: 100, White dice: 338, Payoff: 438 user=> (play) Black die: 300, White dice: 322, Payoff: 622 user=> (play) Black die: 200, White dice: 345, Payoff: 545 user=> (play) Black die: 500, White dice: 326, Payoff: 826 user=> (play) Black die: 300, White dice: 362, Payoff: 662 user=> (play) Black die: 100, White dice: 353, Payoff: 453 user=> (play) Black die: 500, White dice: 359, Payoff: 859 ```

The quality of the payoff has a lot more to do with the black die than the white ones. A good payoff (above \$700, the mean) seems to occdur if and if only if the black die roll is good (a 4, 5, or 6) because the sum of the white dice is never far from the mean value of 350. We can formalize this intuition by noting that when independent random variables are added, the variance of their sum is the sum of their variances. The variance of a 6-sided die is 35/12 (2.9166…) and so the variance of the 100 white dice, taken together, is 3500/12 = 291.666…, resulting in a standard deviation just slightly over 17. With a hundred dice being summed together, we can assume the sum of white dice to be approximately Gaussian: 99 percent of the time, the white dice will come in between \$306 and \$394. Even if the white dice perform terribly (say, \$300) a ’6′ on the black die is going to ensure a great payoff.

While standard deviation is a more commonly used measure of dispersion, random variables cumulate according to their variance (the square of the standard deviation). The variance of the black die is not 100, but 10,000 times, that of the white die. This means that it’s 100 times more influential over the payoff than all of the white dice taken together. It contributes just over 99 percent of the variance in the payoff.

Politics

What does this have to do with human behavior and cooperation? Well, consider voting. Some people complain about the supposed “disenfranchisement” of voters in large, non-swing states such as California (reliably blue) and Texas (reliably red) under the electoral college system. When the blocs are predictable, that can be true. However, being part of a voting bloc will, in general, magnify ones’ voting power, just as the black die in the example above dominates the payoff to the point where the white dice hardly matter.

If fifteen people agree to vote the same way, they’ve increased their voting power (variance) by 225 times that of an individual, meaning that each one becomes 15 times more powerful. Let’s go a step further. Say there are 29 people in a voting body, and that simple majority is all that’s required to pass a measure. If fifteen of those agree to hold a private vote, and then vote as a bloc based on the result of that vote, the other fourteen peoples’ votes don’t matter at all, so each bloc member becomes approximately twice as individually powerful. This can further be corrupted by creating nested blocs. Eight of those people could break off and hold their own private vote and become a bloc-within-the-bloc. Of course, secrecy is required; otherwise the out-crowd of that bloc might defect. At least in theory, nothing stops a group of five people within that eight from forming a third-level bloc, and so on. This could devolve into an almost dictatorial situation where two people determine the entire vote. It isn’t always long-term stable, of course; disenfranchised people within blocs will (over time) leave, possibly joining other blocs.

One should be able to see, by now, why something like a two-party political system is so common in government. Coalitions build, because it magnifies the individual’s statistical power (percentage of the variance) to form blocs. It seems to continue until there are two coalitions in the 45 to 50 percent range, and what limits this process is that, as a coalitions grow, they become more predictable and less nimble; once they are predictable, unaffiliated (“swing”) voters have substantially more power than they should according to the principles above; while variance potential of a bloc grows as the square of its size, highly predictable blocs have very little actual variance. In other words, the equilibrium happens when the (quadratically growing) bulk power of blocs is offset by the declining true variance inherent to their predictability, leaving the few swing players as individually powerful (being unpredictable) as they would be as members of a bloc.

Economics and work

Bloc power is a major reason why collective bargaining (unionization) is such a big deal. The Brownian motion of individually unimportant workers joining and leaving a company has a minimal effect on the business. There will be good days and bad for the company due to these small fluctuations but, on the whole, an individual’s vote (whether to work or quit) is meaningless amid the noise. The low-level worker has no real vote. Collective bargaining, on the other hand, can be powerful: a large group voting against its management (a strike) at the same time can have a real impact.

The past two hundred years have proven that, without some variety of collective action, workers (even highly skilled ones) are unlikely to get a fair deal. It doesn’t matter how smart, how capable, or even how necessary they are if their votes don’t matter. There are three approaches that have been used to solve this problem (aside from a fourth, beloved by some wealthy, which is not to solve it). The first is to form a union. As much as there is a problem of corruption within unions, I don’t think any reasonable person can review history and conclude them to have been unnecessary. The second is to form a profession, which is essentially a reputation management organization that (a) keeps the individual member’s credibility high enough to keep that person employable, so he or she can challenge management, since professions require ethical obligations that supersede managerial authority (i.e. there’s no Nuremberg Defense); while (b) occasionally leveraging its role as a reputation bank to push, as a bloc, for specific causes. The third approach is a welfare state, which does not confer bloc-like power for low-level producers (i.e. workers) but (a) gives them power as consumers and, more importantly, it (b) gives individual producers the ability to refuse adverse terms of work.

These form a spectrum of solutions, with unions being the most political (an explicit bloc forms, subverting to some extent the Brownian tug-of-war that occurs in free markets and elections) while the welfare state is apolitical (it does not tell capitalists how to run their companies) while pushing a universal sea change in the market– improved leverage for workers in all industries, liberated from a month-by-month need for work income. Professions, as it were, exist between these two extremes; they are not as explicitly political or bloc-like as unions, but their ability to prevent the professional’s credibility from falling to zero– even if fired by one comp[any’s management, he’s still a member of that profession unless disbarred for ethical reasons, and will usually find new work easily– has them functioning like a private, conditional welfare state.

I’m not going to argue, among the solutions above, that any is superior to the others, or that one of those three should be favored uniformly. In fact, they don’t even conflict; societies tend to have all three of the above in some form. They seem to serve different purposes, also spanning a spectrum from local to global, like so:

• The union exists to guarantee, as much as it can, employment at a specific company (local) on favorable terms for good-faith workers. It often wrests from management the authority to terminate people. Its downside is that, because it is an explicitly political organization, it often invents by-laws (seniority systems being the most abhorred) that reduce performance. The extreme guarantees against adverse change that unions often provide may result in a low quality of work, eroding the union’s clout in the long run. Unions are, however, the best solution when there is a small number of potential employers (oligopsony).
• The profession exists to provide credibility (reputation) sufficient to guarantee appropriate employment, but not at a specific employer. The profession doesn’t interfere with individual terminations or promotions, nor does it often tell employers how to behave; its goal is to provide appropriate results for all good-faith members without managing a specific employer. This is more global than the union, because professionals may have to move to different companies or geographic locations to take advantage of the profession’s auspices, but more local than a welfare state because it focuses on a specific class of workers. Professions work well when there is a large and changing set of potential employers, but over a fairly fixed scope of work.
• The welfare state (a global solution, as it involves a definition of social justice that a central government attempts to enforce uniformly) doesn’t guarantee market employment at all. It does, however, attempt to create an economic floor below which people cannot fall. Even if they lose power as producers (because the market may not want anything they can make) they retain some power as consumers. The moral purpose of this is two-fold. First, unneeded workers can retrain and become viable producers. Second, the welfare state’s existence gives workers enough leverage that they stand a chance at getting a fair deal– without necessarily having to form collectives in order to do it. Welfare states do the best job at the large-scale, society-wide problems; for example, they can provide education and training for those who have not yet entered a union or profession.

What’s most relevant to all this, however, is that collective action is as relevant today as it was 100 years ago. There are a lot of people who claim, for example, that labor unions “were good in their time, but have served their purpose”. I don’t think that’s true. There are, of course, many problems with existing labor unions and with the professions, but the statistical politics underlying their formation is still quite relevant.

Technology

Software engineers in particular are a group of people who’ve never fully decided whether they want to be blue-collar (making unionization a relevant strategy) or white-collar (necessitating a profession). It’s not clear to me that either of these approaches, as commonly imagined, will do what we need in order to get programmers fairly paid and their work reasonably evaluated. I would argue, however, that the existing culture of free agency seems to be leading nowhere. Software and hardware engineers, in addition to designers and operational people, need to develop a common tribal identity as makers. Otherwise, management will continue to run divide-and-conquer strategies against them that leave them with the worst of both the blue-collar and white-collar worlds: the low autonomy and job security of an hourly wage worker, but the unreasonable expectations and long hours associated with salarymen.

The needs of the most creative and effective technology workers should be given consideration; maker culture is becoming a real thing, and the societies and organizations that prosper in the next fifty years will be those that find a way to contend with it. Thanks to personal computing, the internet, and quite likely 3D printing, we’re coming into an era in which the zero-sum approach to resources that has existed for thousands of years no longer makes sense. Copying a book used to be a painstaking, miserable process. (The reason for the beautiful calligraphy and illustrations in hand-copied medieval books is that the work would be intolerable without some room for creative flourish.) Now it’s a Unix command that takes less than second. Information scarcity is rapidly ending and of more interest is the culture (maker culture) that has sprung up around that, starting in the open source world that is making its way into software, which is structurally cooperative.

Maker culture is centered on the positive-sum worldview that makes sense in such a world. Makers tend to no longer see each other as competitors amid existing scarcity; rather, the greater war is against scarcity itself.

Good programmers no longer buy in to traditional industrial competition. They’d rather work on open source projects that improve the world (and their own individual reputations) than line the corporate war chest, because the benefits of tapping into the larger society (open source economy) are much greater, not only for them but often also for their employers, than those of restricting themselves to one corporate silo.  They’ll work on closed-source “secret sauce” projects in a somewhat privileged (“ninja”) position, but not in the commoditized role associated with the “code monkey” appellation. Those jobs, as portrayed less than affectionately in the movie Office Space, are going to die out.

In twenty years, top maker talent will no longer be employed so much as it is sponsored. This will be good for the world, as it will generate a much more cooperative economy than what existed before it, but a large number of organizations will find themselves unable to adapt and will fail.

16 thoughts on “Statistics, cooperation, politics, and programming.”

1. Uncephalized |

I think your last paragraph is interesting in light of the rise of Kickstarter and other crowd funding projects in the past few years.

2. I don’t really see the difference between what you’re suggesting and a union. Seems like a propaganda vision a union might present to its members.

“They’d rather work on open source projects that improve the world.”

Most people would rather not do that now, what makes you think people would change simply because the working conditions change? There are a lot of brilliant people no doubt working on the next Google Wave.

• Personally, I’d rather further the world through open source exclusively (particularly in the tooling, as it potentially can have multiplicative effects on productivity), but, for now, I hold a conventional job, and make some time for open source on the side. I’ve been fortunate to meet a lot of interesting people doing this. I see open source as a way to explore what is *really* interesting to me as I try to navigate my career toward it.

You can’t trust Industry to challenge you intellectually; they’re too busy running in place, solving the same problems over and over, while decrying everything else as “too academic.” Ignorance becomes a badge of pragmatism or honor.

• For sure. Occasionally you can get a job that both furthers your skills and accomplishes what the company needs but this usually arises out of pure coincidence, especially at higher levels. Still, while open source software is great, I think “furthering the world” is much too grandiose a statement for that or _any_ technology. Same thing with Michael’s “improve the world.” You might and probably will be able to improve technology and science, but the world, that’s startup propaganda talk and just plain impossible.

If you’re very good and lucky you might make other developers’ lives easier. That’s hardly improving the world. I can think of a few other examples where software has helped a limited number of people (medical field), but it still did not improve the world.

• Well, I disagree with you entirely. Technology is improving the world a great deal. The victories are often silent– things become cheaper, a few years or decades are added to life expectancy for this condition or that, markets become more efficient. It’s incremental and often silent, and there’s plenty of noise.

That said, the gains are being very poorly distributed. See my reply to MG about maker vs. taker culture. We’re makers, but the people running society are governed by taker culture and make sure that the common people just get the scraps of the bounty that’s produced. There’s a lot of wealth that exists now that didn’t in 1975, but almost all of it has gone to the elite (who had very little to do with producing it).

• Ok, I concede that there are some technological improvements that better the quality, not just the quantity (length) of human life, yet how many of these are open source projects or even software at all? What examples do you have of “open source projects that improve the world?”

• This is quite accurate. OS culture is a pure maker culture; people are evaluated based on how much value they add to the world. Most of what people learn in business school is how to excel in a bureaucratic, private-sector taker culture focused on value-capturing.

3. Portlander |

If the topic is labor politics, you can’t not mention immigration policy.

Labor, like any other commodity, is ultimately dictated by supply and demand. Unions and professions, the latter with a credentialing authority that is used to limit supply, fail when there are too many scabs. Scabs result from a low-trust environment and a huge labor surplus that is being forced to choose between near-term survival and long-term social good. Large numbers of immigrants not only increase supply, they erode trust, and change the near-term/long-term trade-offs in that the immigrants won’t even be here for the long-term. Or even if they intend to be here long-term, they still have an escape hatch if things really get miserable.

• True. Too bad there is a huge shortage of skilled labor in the technology and information sector in this country and our immigration laws are so messed up we can’t bring people here who not only want to come (wether they stay is completely irrelevant), but have skills and abilities that are scarce or non-existent in the US.

• Portlander |

There is not a huge shortage of skilled labor. There is a huge shortage of 20-somethings willing to work for relative peanuts. And there is a huge shortage of experienced engineers that already know the latest web programming dialect. But that is not the same as a huge shortage of skilled labor. The H1B scheme is a fraud by our government against its own people so that back-office dinosaurs like IBM and EDS can have their executive level management making big money like all the FIRE guys.

• asdf |

Indeed. I have not met a single rock star H1B in my life even though there is a large contingent at every company. They are generally desperate people with bad English that aren’t particularly good at their jobs but will work for under market wages and do lots of fourth quadrant work on Saturdays. And once they get in they only hire their own.

As someone that grew up in a union household and belong to a credentialed profession I’ve seen what the trust destroying effects of immigration have done to both. And in the really long run it will effect the ability to have a welfare state as well.

An IQ cutoff of 130+/wealth requirement would net all of the rock star immigrants with the potential to start big companies while eliminating the 99% that are mostly just about driving down wage rates.

• Anecdotal experience is not of value here. Sure if we had smart people here we wouldn’t need others but with no education and a hiring process designed to weed out the best engineers, no wonder why people have resorted to immigration. Rightfully so since most programmers coming out of U.S. schools can’t basic computer tasks.

• Portlander |

Industry bromides are of no value here.

• Actually, that’s exactly what this blog is about.

• Verun |

Governments working in concert with the biggest corporations. It’s actually extremely common in the big investment banks in NYC to have entire floors filled with H1-B workers doing the grunt tech work. Until recently I was working in one such bank that literally had (on some floors) the pantries converted into 2, perhaps 4 mini cubicles to house such workers. I don’t know if the big tech companies are trying to imitate the banks, or if the banks took their cue from the tech firms – in either case with current talk of expanding the visa program, this trend is only likely to increase.

I find a lot of talk in the blogosphere of how the primary driver behind big corp’s wholehearted support for expanding the visa program is their innate drive to maximize profits by cutting (primarily labor) costs. However, from what I’ve learned working in the field over the last 5 yrs it seems to me that most H1-B workers are paid about as well, or perhaps only marginally less than, their non-visa holding counterparts. I think the real motivation has always been to have just a captive labor force: employees that basically STFU and do as they’re told. For Indian IT workers, many of whom come from the lower classes in India, the field has become a sort of escape hatch from the utterly broken and extremely unequal system in India. They’re doin what any rational person would do in the situation – don’t forget that for them, this is an opportunity to alter not just their own economic destinies, but that of *generations* of their families. They’re being literally catapulted from the lower classes of a third world country to the middle, and perhaps even upper-middle, class of a 1st world western nation. Expecting them to therefore pass on an such an opportunity in favor of more equitable working conditions and wages would be naive in the extreme.

As per asdf’s comment of never having come across a truly competent H1-B worker, I’d have to say that my experience has been mixed. I think that just as in other groups, there’s a bell curve sort of distibution.

Lastly (and perhaps unrelated to anyone’s prior points) I think the phenomenon of a third world country with an essentially crappy higher education system (barring the few IIT schools and others of it’s caliber there) becoming an IT back office to the entire world is interesting. I think the relentless drive towards abstraction, continuous lowering of barriers to entry with the rise of languages like Java and C#, and the internet have been pivotal.

• asdf |

I’ve met some good foreign workers, but they rarely seem to come through H1B. Either they are citizens, have their own company/work through some consulting firm, some other workaround, etc. It’s very rare to see a standard corp job filled by an H1B where it isn’t just some race to the bottom via pay or conditions.

I have found that pay tends to be an issue, with most H1Bs I can coax the information out of making under market. Though part of the issue could simply be that the sheer number of H1Bs are themselves setting the market. What would the market be without them? I’ve seen employers not fill a job so they could hire an H1B.

There is also a compliance issue. They really do “STFU and do as they’re told.” This isn’t good though. What it mostly means is putting in lots of facetime and doing lots of useless fourth quadrant work. It’s not really doing anything for the business, but it costs the company nothing (unpaid overtime). If your trying to have any kind of united front against these bad management practices H1Bs will form a kind of race to the bottom fifth column. I think the divisions that multicultural workplaces are a big part of why managers like H1Bs. It prevents employee unity. You end up with politically separated ethnic enclaves more concerned with the advancement of their own then any kind of class solidarity.

When it comes to getting actual important work from these people its a problem though. Language barriers. Communication problems. In terms of the kind of necessary first, second, and third quadrant work I’ve never been too impressed. Some of them are OK, but we’ve got plenty of OK people right here in the USA.