- BIOETHICS FORUM ESSAY
Home Plate, the Cheese Lady, and Bad Sonnets: On the Limitations of Regulation
What follows is drawn from an address delivered in January 2007 at a meeting of the American Health Lawyers Association. It is a very slightly revised version of a keynote address delivered in November 2006 for PRIM&R.
Today, I will discuss certain aspects of regulation – a subject that has interested me ever since my student days.
Item: When an undergraduate, I encountered a situation in which the freshmen in a residence, but only the freshmen, were required to wear a tie and jacket for dinner. Their sense of injustice was matched by their intrepid mode of protest. They posted guards at the doors and windows, complied strictly with the regulations, and showed up wearing ties and jackets. Just ties and jackets. Nothing else. Picture that.
Item: In college I was in Army ROTC for a while; it was mandatory. I then discovered I had received some modest funds from the Army in error. I tried to return that money. Lots of regulations covered disbursements; none allowed for the return of funds. I persisted. The Army people begged me, citing the national interest, to go away, deposit the check, and never return. I complied.
Item: When a graduate student in the early 1960s – a few of you have at least heard of those – I worked briefly at Lockheed Aircraft as a technical writer. I wrote a report for the Midas Satellite Division, as instructed, and submitted it early. The next day, aware that a few small changes could improve the report, I asked the division manager to return it. He said I did not have a high enough level of security clearance to see it. Pointing out that I had written it, and knew in detail all that was in it, was futile. He was following regulations; I could not get it back.
Of course, he wasn’t French. A few years ago, Bill Lowrance – an observant, insightful, and bemused American colleague living in Geneva – described differing cultural attitudes toward rules, regulations, and laws by saying that the Swiss stop at stop signs, look both ways before crossing, park only where it is allowed, and understand the train schedule to describe reality, whereas the French consider rules, regulations, and laws to be interesting suggestions.
Even in my graduate school days, I wondered what it is about regulations that seems so often to be a barrier to any semblance of common sense. And so I started to think about barriers as well. As an academic administrator, I’ve often focused as much on removing barriers to constructive action as I have on providing incentives. Of course, some kinds of barriers, such as barrier islands, can just disappear naturally. I’ll return to that phenomenon shortly.
I had the great privilege of presenting a keynote address at a PRIM&R meeting – nearly a decade ago – in Boston, in the shadow of Fenway Park, home of the recently disintegrated World Series champions of 2004. On that occasion, the title of my remarks was “Senator Proxmire, the Baltimore Orioles, and the National Research Agenda.” I cited the Baltimore Orioles to make a point about algorithmic decision-making and the fallibility of predictive judgment.
I’ll quote briefly from that address now:
“I remember one person suggesting that, if the National Science Foundation had its act together, it would support just those research projects that led to constructive results – the ones that worked. After all, what are they being paid for? That leads me, to your relief at last, to the Baltimore Orioles.
“When the aforementioned debate about the selection of scientific research was taking place, I called the Baltimore Orioles. I said, “You have the best farm team system in professional baseball. I want to know how successful you are in signing new hires. I know that for someone to get a contract in the Orioles’ system and become professional, it is not nearly enough just to be the best ball player a high school has ever produced, or to be an all-state star. It is not enough just to be fabulously good. You have to be immensely better than that to become professional.” They said, “That’s right, and some years are much better than others. In a bad year everybody falls by the wayside.” I asked, “How many, out of a hundred signed, on average, ultimately will wear the Orioles uniform and play in the major leagues?” The reply was, “Well, the range is about one to two percent.” I was tempted to ask, “Isn’t that inept? Why don’t you just give contracts to the ones who are going to make it?” Of course, everyone understands why that is impossible and knows that they have to take risks because good predictive ability about such outcomes does not and cannot exist. Even people who are deeply stupid about how science works seem to understand the baseball analogy, and can make the transfer – at least, if they are led to it. The general point is about uncertainty and the taking of risks.”
This afternoon, for the sake of continuity, I will continue the baseball theme, starting as one ought, at home plate.
In describing the new chief justice, John Roberts, the eminent New York Times reporter Linda Greenhouse wrote, “That raises the question of whether the chief justice’s performance conforms to his own stated goal: to be a ‘minimalist’ judge who decides no more than necessary, an umpire simply calling balls and strikes.” My colleague David Potter, who has 35 years of experience as an umpire – experience which I’m sure serves him well in his capacity as our associate dean for student services – took exception to that characterization of umpiring. In response to Linda Greenhouse, he wrote,
While it is true that there is a detailed rule book and several case books, much of what an umpire decides is based on circumstances, interpretation, unwritten ‘common law’ which one learns on the field. In many respects the situation is like that of the beat cop. Were he or she simply to enforce all the laws as written, the officer wouldn’t last a month, and shouldn’t. Both the umpire and the officer on the street have a much more basic and important task than strictly enforcing the law: They must each help assure that the community in one case, and the ball field in the other, are able to function as intended.
The thrust of Potter’s observation is not that rules and regulations are unimportant or to be scorned, but that they are only a part of the story – tools to use in pursuit of larger objectives. And relying on strict compliance as the proper use of those tools can undermine the pursuit of those objectives.
My subtitle today references the limitations of regulations. That is a special case of a larger category: the limitations of algorithmic modes of analysis generally – of rule-based, quantitative assessment of matters that may instead require the kind of approach that Potter advocates.
Now, to the story about the cheese lady. This is no allusion to Monty Python, although it could be. I once visited a friend at the National Institutes of Health who put a piece of paper on a granite counter in his laboratory and asked me to sign my name without touching the paper or dotting the i. I did so. He then picked that paper up with tweezers and weighed it on a laboratory scale. We looked at the digital readout, which had many numbers. He put the paper back on the counter, and said “Now, without touching the paper, dot the i.” So I dotted the i. We weighed the paper again, did the subtraction, and determined the weight of the dot of the i. For his work in molecular genetics, that kind of precision really matters.
A few months later, at the cheese counter in an Amish shop, I said, “I’d like half a pound of that cheddar, please.” A straight-faced Amish woman took a lovely instrument with a curved blade and a wooden handle on each side, approached a large block of cheese and pondered it carefully, and then cut off a segment. She put that segment on a scale, and it read 0.50. Astonished, I said, “You’ve done this before. I want to see it again. So, I’d like half a pound of the Swiss cheese, also.” Dead-pan, she cut the Swiss cheese, put it on the scale, and it read 0.46. And she looked me straight in the eyes and monotonically said, “It’s the holes.”
Suppose instead she had said, “Sorry, I’ll do it again,” come up with 0.53, and then said, “Let me just trim that a little bit,” and got another 0.50 – only to say “It still might be a little over, but below the detection limits of my scale. I’d better get one of those scales from NIH.” That kind of passion for precision would be pure pathology. This would be someone in the wrong line of work, on the brink of not being in that line of work anymore. Such a little difference doesn’t matter. It’s a marginal difference that means nothing.
However, if we go to a neurosurgeon because of that growing pressure on the brain, we do not want to hear, “Well, that’s close enough, anyway.” An extremely small difference in cranial pressure, as that pressure approaches the limits of elasticity, can make all the difference that matters between success and failure, between life and death. So, we need good judgment to understand how much a little difference will matter in deciding what level of precision a context demands.
A sophisticated understanding of precision is not about maximizing accuracy in all contexts. It’s about the relationship between little things and big things. It’s about the impact of small differences at the margins. It demands a kind of experienced judgment for which there are no algorithms. Without that judgment, there’s little chance of making the distinction well between doing a task flawlessly and doing it unacceptably, whether the task is selling cheese, umpiring a baseball game, or doing surgery. There’s no more chance than a computer has of being an inspired poet, writing a sonnet of enduring beauty and power.
Indeed, that difference between merely doing a task in compliance with the rules and doing it well is nowhere better illustrated than by the example of crafting a sonnet. There are clear rules – regulations, if you like – for writing a sonnet. Imagine an Office for Protection from Bad Sonnets. The regulators from that office would check to see if the rules had been followed – what else could they do? And at universities throughout the country, at least those whose faculty write or teach sonnets with support from federal funds, there would be concern to be in compliance with the regulations. Some examples will help, but let’s start with the regulations.
A sonnet is a fourteen-line poem, typically of either the Italian (that is, Petrarchan) or English (that is, Shakespearean) form – defined by distinct rules. The Italian form is divided into the octave and the sestet, each with a specific rhyme scheme. The English sonnet – which we will consider today – has three quatrains and a couplet, with the typical rhyme scheme being:
abab cdcd efef gg.
The last two lines are often an epigrammatic close, reflecting on the contents of the first twelve lines.
There’s no better example of the Shakespearian sonnet, of course, than one by Shakespeare. Here’s # 55, called “Not marble nor the gilded monuments”:
Not marble, nor the gilded monuments
Of princes, shall outlive this powerful rhyme;
But you shall shine more bright in these contents
Than unswept stone, besmear’d with sluttish time.
When wasteful war shall statues overturn,
And broils root out the work of masonry,
Nor Mars his sword, nor war’s quick fire shall burn
The living record of your memory.
‘Gainst death, and all oblivious enmity
Shall you pace forth; your praise shall still find room
Even in the eyes of all posterity
That wear this world out to the ending doom.
So, till the judgment that yourself arise,
You live in this, and dwell in lovers’ eyes.
That, I submit, is as vividly clear an example of a fine sonnet as one can find. In starkest contrast, I offer a sonnet I wrote for a class in which the students were studying David Hume’sDialogues Concerning Natural Religion. I did not exactly write it to illuminate Hume’s discussion of the problem of evil, but to mark the contrast between a sonnet that merely complies with the regulations and one that is any good – which this one is not. Brace yourselves:
I went into the dinning hall today
And saw a bunch of stuff I do not like.
I ordered them to take it all away,
And said if not then I would have to strike.
Security then came and hauled me off
And made me pay a great humongous fine.
All this because I did not like that trough
Of rotten things on which I will not dine.
Oh! Would some patron saint I’ve not yet seen,
Take pity and respond to my dire plight
By taking me to find some haute cuisine
To have for dinner almost every night.
Is this too much to hope for, I now ask?
A perfect god would find this a small task.
This dreadful sonnet complies strictly with the rules, and so would pass muster with our new Office of Protection from Bad Sonnets. Some serious suffering would result from its approval. But that is not the worst problem here. Let’s revisit that authentically fine sonnet, #55. Imagine a letter of complaint from an angry stickler to the member of Congress from his district, thus:
“Dear Congressman Thwartlots:
What is going on with your Office of Protection from Bad Sonnets? I am trying to uphold standards we can all be proud of. I represent the Center for Sonnetarian Integrity, and we vote. And I see that in Shakespeare’s sonnet #55, there is a problem that your overpaid bureaucrats have just missed, or don’t care about, and I don’t even know which is worse. Look at these lines yourself:
Not marble, nor the gilded monuments
Of princes, shall outlive this powerful rhyme;
The first line has ten syllables. Does the second? You’re in Congress, for Heaven’s sake. You ought to know something about power. Can you count? Try saying “power” in one syllable. You can’t. That line has eleven syllables, and this sonnet, I don’t care who wrote it, is no example to hold up for our young people today, who have to learn the difference between what follows the rules and what does not….”
I’ll spare you the rest of this very long letter, but not the rest of the story. Congressman Thwartlots makes a call or two, and OPBS rescinds approval of sonnet 55. One university, teaching that sonnet with the support of a federal grant, hangs tough and insists it is a good sonnet. OPBS initiates an inquiry. The provost’s office gets involved, committees form, hearings ensue, and now various people – some of them well educated, some of them elected officials, and a few of them even both – are spending their time assessing and disputing various pronunciations of “power” – as they are really thinking about the press, the risk-managers, and the flow of cash. Local Sonnet Monitoring Committee members resign, and others refuse to serve, because they have come to see such committee work as futile or even dangerous. All that is surely even more damaging than having to hear my fully compliant, technically perfect, really bad sonnet.
Of course, one good dose of sensible, good critical judgment would have avoided all these problems. But that’s only possible when a system of oversight allows for good judgment. When there is no place for judgment, even a minor deviation from regulations can lead to serious storms of accusations, recriminations, distractions, and unproductive interventions. We all know that serious storms can do serious damage.
One such storm, Hurricane Katrina, was both unexpected in its specificity and yet of a kind that was expected and even predicted. Among its consequences has been a flood of analysis, including a focus on some ostensible noncompliance with medical regulations in its immediate aftermath. Steven Miles, among the nation’s most thoughtful, humane, and eloquent physicians, provided this perspective:
When disaster and medical ethics collide, when the respirators, the labs, the x-rays go dark, the serum does not arrive, the extra sitting nurses disappear, the telecommunications turn off, the on-line medical records become as unobtainable as the paper files in the flooded basement, the medical ethics landscape is changed – not in a way that suspends the ethical rules of practice but in ways that can make them impossible to logically use.
This is not a screed against respect for regulations, but against regulations blindly invoked in contexts of disaster. We also all know, from hard experience, that although human subjects research requires and is governed by regulation, inherent in that research enterprise lurks the surety that once in a while, unpredictably, there will be disaster, just as surely as there will be future Katrinas, tsunamis, and earthquakes. When such disasters occur, there may or may not be villains, and noncompliance with regulations may or may not be part of the story. It makes no sense to ask only whether regulations were followed, without making the crucially important, and sometimes subtle, distinctions among the various ways non-compliance can occur.
Wrestling with such stressful matters can prompt one to want to lock the office, turn off the cell phone, and just head for the beach. But that’s not always possible. One of the harms caused by Katrina was the complete disappearance of some of the small islands in the Gulf of Mexico, and there was extensive damage to many beaches. And that is distressing because we care about beaches, for many reasons. We love them, yet we know that they are dynamic, ever-changing, often fragile interfaces between land and sea.
Let’s consider going to the beach on one of the beautiful barrier islands off the coast of North Carolina. You buy some expensive beachfront property there and build a fine house safely behind the high tide margin. Time passes, there’s a storm once in a while, a lot of sand washes away, and one day your house is no longer on the dry side of the water’s edge. It is largely in the water even at low tide, and it starts breaking apart. That sort of thing does happen. Or, perhaps you said, “Hey! My beach is eroding. Call the Corp of Engineers. Build a seawall. Nourish the beach with trucks of sand. Protect my private property. And do it with taxpayer subsidy.” This, too, happens often. And sometimes, the whole island just goes away, as we saw when Katrina was raging.
The leading expert on beachfront erosion is Orrin Pilkey of Duke. He and Linda Pilkey-Jarvis have a splendid new book, with the captivating title Useless Arithmetic. Pilkey is an accomplished scientist, so this is intriguing. I’ll skip the details, and just summarize the main theme. It is that there is a widespread and dangerous propensity to be attracted to false claims of precision and accuracy, and that mathematical models, in particular, although much revered, are not adequate to the task of predicting Earth surface changes.
The authors offer as a stunning case study the example of Bruun’s Rule for calculating expected shoreline erosion. It is based on provably bad assumptions and leads to embarrassing consequences, yet has been adopted throughout the world as a tool for planning. The reasons for its mystifying and indefensible adoption, they point out, are that it is available, is simple to apply, has no serious competitors, and has at least a thin veneer of ostensible precision. And that is apparently enough.
It’s not that the authors want us to abandon empiricism, or to make guesses uniformed by evidence. Instead, they call for the use of informed qualitative judgment. We can, on the basis of evidence, justify claims about trends, directions, and proportionality. And we can do that best if we have some reasonable judgment about what makes good sense – rather than following some formulaic approach that can be applied algorithmically but which can also be detached from reality – especially when that formula is made sufficiently malleable – by using fudge factors – to accommodate any observable data, thus making it essentially unfalsifiable.
After showing, in detail, why mathematical models for Earth surface change are irreparably misleading, they conclude:
[N]ot recognizing complexity is what has allowed us to escape from reality through quantitative mathematical modeling…. It is a world where mathematical equations characterize events and processes, equations that can describe only a small part of the picture in very simple fashion. The intuition of an experienced scientist is gone. At best, only a small fraction of the processes that lead to the desired endpoint prediction can be considered….. [I]f we wish to stay within the bounds of reality we must look to a more qualitative future, a future where there will be no certain answers to many of the important questions we have.
These authors address the challenges of predicting changes in the Earth’s surface, but they illuminate brightly the general difficulties of seeking more precision than current understanding allows and the dangers of acting as if we had such precision – or had a right to expect it. The complex vagaries of dealing with human patients or human subjects are no less challenging.
If it weren’t so early in the day, I’d propose a toast to what the Pilkeys have said, and to the lucidity with which they have said it. And it would be with good wine, too. There’s no reason to drink bad wine. It wouldn’t have to be great wine, but something at least that would prompt the oenophiles among us to nod with approval. And we wouldn’t want it to be hugely expensive. If price were no object, it would be easy to make a good choice – perhaps the 2004 Charmes-Chambertain at about $100 a bottle. But nearly all of us must keep an eye on the budget. So maybe we would seek the advice of experts, looking to their carefully developed ratings to guide us.
Most of us are by now familiar with the increasingly prominent 100-point rating scales used by leading wine critics, the publications where their work appears, and the merchants and marketers who rely on those ratings. If we see an 82-point wine for $25, we know enough to keep looking. If we see a 94-point wine for $50, most of us still know enough to keep looking. And when we see that 91-point wine for $12, we brighten – knowing that we stand before a great opportunity.
But it’s nonsense. It’s a silly dance, and even the choreographers acknowledge that. As The New York Times reported recently, Joshua Greene, editor of Wine & Spirits, says “On many levels it’s nonsensical…. I don’t think it’s a very valuable piece of information.” William Tisherman, former editor of Wine Enthusiast, says, “The deeper you get into this, the more you realize how misleading and misguided this all is.” And Michael De Loach, of the quite admirable Hook and Ladder winery – they make a splendid and reasonable Chardonnay, for example – says “If Parker or Spectator don’t give you a high enough score, you can make up your own…. [A]pplying a 100 point scale to wine is dishonest. It makes the consumer think it is scientific.” It’s the wine world’s analogue to Bruun’s Rule.
The great English philosopher G.E. Moore, discussing the grounding of moral judgment, gave the name “The Naturalistic Fallacy” to any attempt to infer what ought to be done from propositions about the natural world, no matter how well substantiated those were. This point of view is often summarized by the aphorism, “You can’t derive an ‘ought’ from an ‘is’.” There’s a long lineage to this perspective, from pre-Socratics such as Protagoras, through the Scottish empiricist David Hume – and they all understood that human values require human judgment to be applied even after all the relevant facts are assembled. Indeed, even the determination of which facts are relevant requires judgment about values.
Such judgments about values are inherently reflected in the framing of the regulations we adopt, and that is widely understood. The need for evaluative judgment in the implementation and enforcement of regulations is comparably inherent, but far less visible, and we overlook or override that need at our peril.
We have an affinity for what can be counted, measured, checked against lists, verified, recorded, reduced to PowerPoint slides, and then defended in a way that protects us from bearing substantive decisional responsibility. But ethics is precisely about substantive decisional responsibility, and the protection of human subjects and of patients – and the many other participants who are at risk in medical settings – are matters of ethics.
It may be intriguing to discover that there are substantive connections among such ostensibly diverse subjects as cheese shops, baseball umpires, sonnet writers, and all the rest, but that’s not enough. What is intriguing is not always important – except perhaps to whomever it intrigues. So we have to ask finally whether there is an operational message here that is persuasive. Unsurprisingly, I think there is.
I call on you all now to enlist in a Campaign for Responsible Judgment.
Bad judgment exists, of course. But the remedy for that is not to try to exclude judgment altogether. Nor should flawed judgment necessarily be a hanging offense, if it does not flow from venality or from culpable negligence or ineptitude.
So here’s what I charge you to do:
1. Sharpen your detectors, and ask always whether the matter before you rests on prior judgments that might not bear close scrutiny – or requires new judgments that will.
2. Be on the lookout for the judgment averse – the ubiquitous, tedious algorithmic automata who have no sense of the larger purpose that is undermined by their parochial perfectionism. They can do great damage.
3. Determine for yourself what the scope of your role is to be. Whether you are in-house counsel, working in a firm that represents a research institution, an advocate for patients or research subjects, an IRB member, a risk-manager, or anything else, if you are asked to address a specific question, pause and ask what purpose the question is meant to serve. And if you applaud that purpose, ask whether addressing the question as it was presented to you does serve that purpose.
4. In your workshops and panels and all the other discussions at this meeting, view the content through the lens of a Responsible Judgment Advocate. And that should be a prelude to an enduring habit of mind. Thus, and finally,
5. In all your roles and dealings, be an active advocate for informed, empirically grounded, responsible qualitative judgment, whenever algorithmic and purely quantitative analyses are inadequate – which is most of the time.
I’ll end with one more anecdote. I was asked by a federal agency to join a site-visit team. The three of us, including a program officer from the agency, were lodged at a lovely hotel. Instead of having dinner in the hotel’s expensive dining room – which the program officer assured us would be appropriate – we went to a terrific ethnic restaurant in a neighborhood a few miles away. It cost about one third of what the hotel dinner would have cost. I paid the $4 taxi fare back to the hotel. Later, my request for expense reimbursement was rejected. I had not, in the words of the rejection letter, “documented the unavailability of dinner at the hotel.” I replied that the problem was the lack of lodging at the restaurant. Eventually, I got my expenses paid – minus that $4.
I rest my case.