Back in 1977, Ken Arrow, having won the Nobel Prize five years earlier, wondered about the internal functioning of firms. “To what extent is it necessary for the efficiency of a corporation,” he wrote, “that its decisions be made at a high level where a wide degree of information is, or can be made, available? How much, on the other hand, is gained by leaving a great deal of latitude to individual departments which are closer to the situations with which they deal, even though there may be some loss due to imperfect coordination?” The answer depends somewhat on whether the firm has one goal or several, on the correlation among multiple goals, and the degree to which different departments contribute to different goals.

In general, though, the answer is sobering for advocates of decentralization. The severally optimal choices of departments rarely combine to yield the jointly optimal choice for the overall enterprise. That’s not to say that centralization is wrong, of course. It merely means that one must balance the healthy and interesting diversity that results from decentralization against the overall inefficiency it can cause.

If we shift focus from the firm to enterprises within an economic sector, the same observations hold. To the extent enterprises pursue diverse goals primarily for their own benefit rather than for the efficiency of the entire sector, that sector will be both diverse and inefficient — perhaps to the extremes of idiosyncrasy and counterproductivity. Put differently, if the actors within a sector value individuality, they will sacrifice sector-wide efficiency; if they value sector-wide efficiency, they must sacrifice individuality.

Higher education traditionally has placed a high value on institutional individuality. Some years back a Harvard faculty colleague of mine, Harold “Doc” Howe II (who had been US Commissioner of Education under Lyndon Johnson), observed how peculiar it was that mergers and acquisitions were so rarely contemplated, let alone achieved, in higher education, even though by any rational analysis there were myriad opportunities for interesting, effective mergers. (Does the United States really need almost 4,000 nonprofit, degree-granting postsecondary institutions, not to mention 14,000 public school districts?) Among research universities, for example, Case Western Reserve University and Carnegie-Mellon University were two of the few successful mergers, there were some instances of acquisitions and subordinations (I’m not counting Brown/Pembroke, Columbia/Barnard, Tufts/Jackson, or their kin), and several prominent failures — for example, the failed attempts to merge the Cambridge anchors Harvard and MIT. (Wikipedia’s page on college mergers lists fewer than 100 mergers of any kind.)

If higher education isn’t going to gain efficiency through institutional aggregation, then its only option is to do so through institutional collaboration. There are lots of good examples where this has happened: I’d include athletic leagues, part of whose purpose is to negotiate effectively with networks; library collaborations, such as OCLC, that seek to reduce redundant effort; research collaborations, such as Fermilab, through which institutions share expensive facilities; and IT collaborations, such as Internet2.

That last is a bit different from the others, in that involves a group of institutions joining forces to buy services together. Why is joint procurement like that so rare in US higher education? I think there are two tightly connected reasons:

• US higher education has valued institutional individuality far more highly than collective efficiency — that is, it assigns less importance to collective utility (that’s a microeconomics term for the value an actor expects) than to individual utility.
• At the same time, it has failed to make the critical distinction between what Ryan Oakes, of Accenture‘s higher-education practice, recently called “differentiating” activities (those on which institutions reasonably compete) and generic “non-differentiating” activities (those where differences among peers are irrelevant to success). As a result, institutions have behaved competitively in all but a few contexts, even in those non-differentiating areas where collaboration is the right answer.

Although it’s a bit of a caricature, the situation somewhat resembles the scenario for the Rand Corporation‘s 1950s-era game-theory test, The Prisoner’s Dilemma. Here’s a version from Wikipedia:

Two suspects are arrested by the police. The police have insufficient evidence for a conviction, and, having separated the prisoners, visit each of them to offer the same deal. If one testifies for the prosecution against the other (defects) and the other remains silent (cooperates), the defector goes free and the silent accomplice receives the full one-year sentence. If both remain silent, both prisoners are sentenced to only one month in jail for a minor charge. If each betrays the other, each receives a three-month sentence. Each prisoner must choose to betray the other or to remain silent. Each one is assured that the other would not know about the betrayal before the end of the investigation. How should the prisoners act?

The dilemma is this:

• The optimal individual choice for each prisoner is to rat out the other — that is, to “defect” — since this guarantees him or her a sentence of no more than three months, with a shot at freedom if the other prisoner remains silent. Individuals seeking to maximize their own success (to make a “utility-maximizing rational choice”, in microeconomic terms) thus choose to defect. In decision-analytic terms, since prisoner A has no idea what prisoner B will do, A assigns a probability of .5 to each possible choice B might make. A multiplies those probabilities by the consequences to obtain the expected values of his or her two options: (3)(.5)+(0)(.5) = 1.5 months for defecting, and (12)(.5)+(1)(.5) = 6.5 months for cooperating. A chooses to defect. B does the same calculation, and also chooses to defect. Since both choose to defect, each gets a three-month sentence, and they serve a total of six months in jail.
• The optimal choice for the two prisoners together, as measured by the total of their two sentences, is for both to remain silent, that is, to cooperate. This yields a total sentence of one month for each prisoner, or a total of two months total. In contrast, defect/cooperate and cooperate/defect each yield twelve months (one year for one prisoner, freedom for the other) and defect/defect yields six months (three months for each). So the best joint choice is for A and B both to remain silent.

So each prisoner acting in his or her own self interest yields more individual and total prison time than each acting for their joint good — each would serve three months rather than one. But since A cannot know that B will cooperate and vice versa, each of them chooses self interest, and both end up worse off.

The situation isn’t quite the same for several colleges that might negotiate together for a good deal from a vendor, mostly because no one will get anything for free. But a problem like the prisoner’s dilemma arises when one or more members of the group conclude that they can get a better deal from the vendor by themselves than what they think the group would obtain. If those members try to cut side deals, the incentive for the vendor to deal with the other members shrinks, especially if the defecting members’ deals consume a substantial fraction of the vendor’s price flexibility. The vendor prefers doing a couple of side deals to the overall deal so long as the side deals require less total discount than the group deal would. Members have every incentive to cut side deals, vendors prefer a small number of side deals to a blanket deal, and so unless all the colleges behave altruistically a joint deal is unlikely.

And so the $64 question: What would break this cycle? The answer is simple: sharing information, and committing to joint action. If the prisoners could communicate before deciding whether to defect or cooperate, their rational choice would be to cooperate. If colleges shared information about their plans and their deals, the likelihood of effective joint action would increase sharply. That would be good for the colleges and not so good for the vendor. From this perspective, it’s clear why non-disclosure clauses are so common in vendor contracts.

In the end, the only path to effective joint action is a priori collaboration — that is, agreeing to pool resources, including clout and information, and work together for the common good. So long as colleges and universities hold back from collaboration (for example, saying, as about 15% of respondents did in a recent EDUCAUSE survey, that their institutions would wait to see what others achieved before committing to collaboration), successful joint action will remain difficult.