The quote I was going to use to introduce this topic — “You’re entitled to your own opinion, but not to your own facts” — itself illustrates my theme for today: that truths are often less than well founded, and so can turn policy discussions weird.

I’d always heard the quote attributed to Pat Moynihan, an influential sociologist who co-wrote Beyond the Melting Pot with Nathan Glazer, directed the MIT-Harvard Joint Center for Urban Studies shortly before I worked there (and left behind a closet full of Scotch, which stemmed from his perhaps apocryphal rule that no meeting extend beyond 4pm without a bottle on the table), and later served as a widely respected Senator from New York. The collective viziers of Wikipedia have found other attributions for the quote, however. (This has me once again looking for the source of “There go my people, I must go join them, for I am their leader,” supposedly Mahatma Gandhi but apparently some French general — but I digress.). The quote will need to stand on its own.

Here’s the Scott Jaschik item from Inside Higher Education that triggered today’s Rumination:

A new survey from ACT shows the continued gap between those who teach in high school and those who teach in college when it comes to their perceptions of the college preparation of today’s students. Nearly 90 percent of high school teachers told ACT that their students are either “well” or “very well” prepared for college-level work in their subject area after leaving their courses. But only 26 percent of college instructors reported that their incoming students are either “well” or “very well” prepared for first-year credit-bearing courses in their subject area. The percentages are virtually unchanged from a similar survey in 2009.

This is precisely what Moynihan (or whoever) had in mind: two parties to an important discussion each bearing their own data, and therefore unable to agree on the problem or how to address it. The teachers presumably think the professors have unreasonable expectations, or don’t work very hard to bring their students along; the professors presumably think the teachers aren’t doing their job. Each side therefore believes the problem lies on the other, and has data to prove that. Collaboration is unlikely, progress ditto. This is what Moynihan had observed about the federal social policy process.

The ACT survey reminded me of a similar finding that emerged back when I was doing college-choice research. I can’t locate a citation, but I recall hearing about a study that surveyed students who had been admitted to several different colleges.

The clever wrinkle in the study was that the students received several different survey queries, each purporting to be from one of the colleges to which he or she had been admitted, and each asking the student about the reasons for accepting or declining the admission offer. Here’s what they found: students told the institution they’d accepted that the reason was excellent academic quality, but they told the institutions they’d declined that the reason was better financial aid from the one they’d accepted.

More recently, I was talking to a colleague in a another media company who was concerned about the volume of copyright infringement on a local campus. According to the company, the campus was hosting a great deal of copyright infringementl, as measured by the volume of requests for infringing material being sent out by BitTorrent. But according to the campus, a scan of the campus network identified very few hosts running the peer-to-peer applications. The colleague thought the campus was blowing smoke, the campus thought the company’s statistics were wrong.

Although these three examples seem similar — parties disagreeing about facts — in fact they’re a bit different.

• In the teacher/professor example, the different conclusions presumably stem from different (and unshared) definitions of “”prepared for college-level work”.
• In the accepted/decline example, the different explanations possibly stem from students’ not wanting to offend the declined institution by questioning its quality, or wanting think of their actual choice as good rather than cheap.
• In the infringement/application case, the different explanations stem from divergent metrics.

We’ve seen similar issues arise around institutional attributes in higher education. Do ratings like those from US News & World Report gather their own data, for example, or rely on presumably neutral sources such as the National Center for Educational Statistics? This is critical where results have major reputational effects — consider George Washington University’s inflation of class-rank admissions data, and similar earlier issues with Claremont McKenna, Emory, Villanova, and others.

I’d been thinking about this because in my current job it’s quite important to understand patterns of copyright infringement on campuses. It would be good to figure out which campuses seem to have relatively low infringement rates, and to explore and document their policies and practices lest other campuses might benefit. For somewhat different reasons, it would be good to figure out which campuses seem to have relatively high infringement rates, so that they could be encouraged adopt different policies and practices.

But here we run into the accept/decline problem. If the point to data collection is to identify and celebrate effective practice, there are lots of incentives for campuses to participate. But if the point is to identify and pressure less effective campuses, the incentives are otherwise.

Compounding the problem, there are different ways to measure the problem:

• One can rely on externally generated complaints, whose volume can vary for reasons having nothing to do with the volume of infringement,
• one can rely on internal assessments of network traffic, which can be inadvertently selective, and/or
• one can rely on external measures such as the volume of queries to known sources of infringement;

I’m sure there are others — and that’s without getting into the religious wars about copyright, middlemen, and so forth I addressed in an earlier post).

There’s no full solution to this problem. But there are two things that help: collaboration and openness.

• By “collaboration,” I mean that parties to questions of policy or practice should work together to define and ideally collect data; that way, arguments can focus on substance.
• By “openness,” I mean that wherever possible raw data, perhaps anonymized, should accompany analysis and advocacy based on those data.

As an example what this means, here are some thoughts for one of my upcoming challenges — figuring out how to identify campuses that might be models for others to follow, and also campuses that should probably follow them. Achieving this is important, but improperly done it can easily come to resemble the “top 25” lists from RIAA and MPAA that became so controversial and counterproductive a few years ago. The “top 25” lists became controversial partly because their methodology was suspect, partly because the underlying data were never available, and partly because they ignored the other end of the continuum, that is, institutions that had somehow managed to elicit very few Digital Millennium Copyright Act (DMCA) notices.

It’s clear there are various sources of data, even without internal access to campus network data:

• counts of DMCA notices sent by various copyright holders (some of which send notices methodically, following reasonably robust and consistent procedures, and some of which don’t),
• counts of queries involving major infringing sites, and/or
• network volume measures for major infringing protocols.

Those last two yield voluminous data, and so usually require sampling or data reduction of some kind. And not all queries or protocols they follow involve infringement. It’s also clear, from earlier studies, that there’s substantial variation in these counts over time and even across similar campuses.

This means it will be important for my database, if I can create one, to include several different measures, especially counts from different sources for different materials, and to do that over a reasonable period of time. Integrating all this into a single dataset will require lots of collaboration among the providers. Moreover, the raw data necessarily will identify individual institutions, and releasing them that way would probably cause more opposition than support. Clumping them all together would bypass that problem, but also cover up important variation. So it makes much more sense to disguise rather than clump — that is, to identify institutions by a code name and enough attributes to describe them but not to identify them.

It’ll then be important to be transparent: to lay out the detailed methodology used to “rank” campuses (as, for example, US News now does), and to share the disguised data so others can try different methodologies.

At a more general level, what I draw from the various examples is this: If organizations are to set policy and frame practice based on data — to become “data-driven organizations,” in the current parlance — then they must put serious effort into the source, quality, and accessibility of data. That’s especially true for “big data,” even though many current “big data” advocates wrongly believe that volume somehow compensates for quality.

If we’re going to have productive debates about policy and practice in connection with copyright infringment or anything else, we need to listen to Moynihan: To have our own opinions, but to share our data.

Robert Benchley’s core premise, in his funny but oh-so-true article “How To Get Things Done“, is that being productive requires deceiving oneself about priorities. My premise today, closely related, is that what we say isn’t what we should say, and that what we actually do isn’t either of those.

That’s obviously an issue in higher-education information technology, my usual focus, but today I’m writing about Chicago restaurants, and specifically about discrepancies among expertise, preferences, and behavior. Such discrepancies are clearly important in higher-education IT practice. The reader will have to deduce that relevance since I’m going to stick to the matter at hand and hope I finish before I get hungry.

Chicago’s a great restaurant town, no matter whether one is interested in the latest molecular gastronomy, regional cheeses and salume, appropriate garnishes for hot dogs or Italian beefs, or nuanced differences among the seven moles of Oaxaca. So it was wholly appropriate that Michelin would choose Chicago to be its third American Red Guide city. Following a year’s worth of visits, Michelin critics awarded stars to 23 restaurants, and Bib Gourmand recognition to an additional 46. Presumably, since they’re chosen by the experts, Michelin-rated restaurants are where one should prefer to eat.

Years ago I needed a nested list of something to teach myself how to use <li> tags in hand-coded Web pages. I arbitrarily chose to draft a list of  restaurants where I’d eaten and might eat again, and over time that list has grown steadily. When I became a Quicken addict, as a byproduct of record-keeping I began tracking how frequently we ate at various restaurants.

Since we’ve lived in Chicago for almost fourteen years (and still spend most of our time there), we’ve developed preferences that guide both what we recommend to visitors and presumably guide where we actually eat. So I have data on where we should eat (that’s the Michelin ratings) and where we do eat (my Quicken data), and since our recommendations to friends over the years have been pretty consistent, I can also say what our preferences are.

I thought it might be interesting to compare those. I’m curious what one might deduce about why restaurant-choice behavior differs from stated preferences, and why personal preferences differ from those of experts. I munged together a list of restaurants comprising the union of

• the Michelin starred and Bib Gourmand lists and
• Quicken data on where we’d dined (arbitrarily defined as an expense of at least $20) at least once in 2008-2010.

Without looking at frequencies, I then rated the restaurants we’ve ever dined at based on how we characterize them to others. I reduced the ratings to five categories: Not Rated (which usually means I don’t remember the place), Okay, Good, Favorite, and Event.

There are 172 restaurants on the combined list. Of those,

• Michelin awards 69 at least one star or a Bib Gourmand rating,
• we dined at least once in 2008-2010 at 103 restaurants that remain open,
• 9 places we dined have closed (none of those is on the Michelin list), and
• we rate 105 of them Okay or better (we rate more places than we dined at because we have opinions about places we haven’t visited since 2007).

First, let’s look at should versus say.

• There were 26 restaurants that both Michelin rated Bib Gourmand or Starred and we rated Good, Favorite, or Event.
• Our Event restaurants (definition: places definitely worth going to once, or if someone else is paying) all received stars from Michelin.
• However, four of their one-star restaurants we rated merely Good.
• Conversely, of our 13 Favorite restaurants in the jointly-rated set, Michelin awarded one star to 5 and Bib Gourmand to 8.
• If we expand the set to include restaurants Michelin didn’t rate and our Okay ratings — that’s a total of 105 restaurants — we learn that Michelin didn’t rate 9 of our Favorite restaurants, and we thought 3 of their Bib Gourmand restaurants were merely Okay.

It’s interesting to look at the discrepancies.

Our 9 Favorites not rated by Michelin are Avec, Café des Architectes, Coco Pazzo Café, Gioco, La Sardine, mk, Pelago, Rosebud Steakhouse, and Shaw’s Crab House. Three of the Favorites omissions from Michelin’s list are curious: Avec, Café des Architectes, and mk are clearly good as or better than many restaurants Michelin rated. The same is true of Pelago, but it may be too new for the Michelin people to have rated it. Some of the others make our list and not Michelin’s because part of their appeal to us is location (that is, they’re near where we live: Coco Pazzo Café, Pelago, and Rosebud Steakhouse), and Michelin does not include that in its judging. And some we find to be consistently good for the money or to be especially welcoming places, which are judgment calls.

Conversely, we rated 4 of Michelin’s one-star restaurants Good, and 3 of their Bib Gourmand places merely Okay. The 4 in that first group are Naha, NoMI, Spiaggia (one of Barack and Michelle Obama’s favorites), and Takashi, and in each case it’s because what we ate, although fine, didn’t live up to billing or price. The 3 in the second group are Bistro 110, Green Zebra, and The Purple Pig, the first of which just hasn’t impressed us that much, and the latter two of which we found disappointing especially given the hype that attaches to them.

All in all, though, we’re not too bad on should versus say. That is, what we recommend and what experts tell us we should be recommending are reasonably well aligned. Yet it’s clear that exactly what criteria one uses becomes very important as one approaches the edge of Should or Say. The general point is that understanding experts’ or one’s own preferences is important. Small differences in criteria can produce substantial differences in what one seeks.

Now the more Benchley-like issue: How does what we say correspond to what we do? Let’s start with an interesting extreme observation: Of the ten restaurants we visited most in 2008-2010, only 1 — Perennial — is on Michelin’s lists, and only 5 — Rosebud Steakhouse, Coco Pazzo Café, Pelago, Perennial, and Avec — are on our own Favorites list.

So what we say and what we do most frequently clearly aren’t quite the same (although at least our top 10 don’t include any restaurants we rate below Good). That is, if we constructed our recommendations based on our behavior, our list might be quite different.

Fortunately, the overall comparison between our ratings and our visits is less discrepant. Across the 105 restaurants we rated, 6 are in the Event class, and we ate at half of those once in 2008-2010 — not a bad number, given that these can cost upwards of $100 per person before wine, tax, and tip. Another 22 we rated Favorite, and we ate at 17 of those at least three times — but two of them went unvisited.

There are 56 restaurants in our Good category, and we ate more than once at almost half of them. Looking at those 25 places, it’s clear that some of them we should rate a bit higher, since they’re at the border and we clearly like them: for example, Branch27, Mercat, Quartino, Rhapsody, Taxim, and The Gage.That list may be the most interesting of all: It clearly suggests that accumulated behavior should figure explicitly in recommendations. That is something the Michelin ratings don’t do.

But this list also illustrates how location can bias behavior-based ratings. Frankie’s Scaloppine is a perfectly good restaurant, albeit one located on the fifth floor of an indoor shopping mall. We visited it 17 times not because it warrants a Favorite rating, but because’s it’s inexpensive, fast, and a block from our home. Several other frequently-visited restaurants on our Good list have that same attribute: even though they’re only Good, they nevertheless have attractive features.

So what should we make of all this? The point, I think, is that we have to be very careful to understand why experts tell us what they tell us, we have to be very careful to understand why we state our preferences as we do, and most of all we need to pay close attention to our behavior. That’s not to say that data should drive preferences — if we do that, nothing will ever change — but rather that divergence between what we say and what we do tells us something, and that something requires attention lest we confuse mythology with fact.

Okay, I can hear at least one of you asking: Where’s that list of Favorites? Here they are. They’re labeled with * for Michelin stars and + for Bib Gourmand, and are in italics if we dined on them at least thrice in 2008-2010:

Bon appétit! ¡Buen provecho! Time to eat!