With permission, I am making available here a transcript of the Analysis programme broadcast by the BBC (British Broadcasting Corporation) on 4 November 1999. The discussion, led by Andrew Dilnot, Director of the Institute for Fiscal Studies, considered the origins of the audit culture -- an emergent socioanthropological phenomenon of formidable power -- and the increasing recognition, today, that while auditing is useful and necessary for some purposes it is not the Answer to Everything, and indeed that the drive to audit everything, and to constrain everything by rigid rules, measures, and targets, `can entirely shut down the drive for innovation'. And auditing everything has infinite cost (as demonstrated below), hence zero cost-effectiveness.
Included in the Analysis programme was the most succinct statement I have yet seen of a fundamental property of audit dynamics, Goodhart's law: What's counted counts. (The next most succinct is When a measure becomes a target, it ceases to be a good measure.)
The present notes (and some of the discussion in lucidity3) are part of my own personal attempt to understand the whole phenomenon and its implications, including the cost. Even a quick web search suggests a vast burgeoning of the audit culture. Institutions are springing up everywhere, with names like Centre for Quality Assurance, that seem dedicated to creating ever more elaborate self-auditing and external auditing systems. An unconscious assumption seems to be involved, or wishful thinking, that one can have an absolute `assurance' of quality, value, and other such desirables. The language itself, the use of words like `assurance', propagates the assumption. Goodhart's law never seems to be mentioned despite its warning about the real effects, which cut far deeper than mere games with targets and measures. The affair of Summerhill School (see below), and the multi-billion-dollar crisis in patent law (see below), are two among many reasons to believe that we have a serious problem on our hands.
This by the way is not a personal attack on the increasing number of upright, honest and hardworking people who are charged with becoming auditors or audit researchers. The problem does not seem to be something that can be blamed on any individual. Rather, there seems to have emerged (in the sense of complex-systems theory) an audit culture in the deepest sense of the word culture, involving sets of largely unconscious assumptions that have somehow become embedded in the minds of many people, through complex collective processes of which we have little clear understanding. After all, we have little clear understanding of the complex collective processes going on in the tens of billions of neurons of a single human brain. And part of the problem is our proneness to assumptions that are not only unconscious, but also mutually contradictory.
In what follows, I am going to simplify and caricature things quite deliberately, in the hope of producing a shock of recognition.
What are the most basic principles behind the drive to audit everything? Arguably, they are fairness, objectivity, and prudence. What could be more reasonable, or more worthy of our aspirations? What could be more essential to a modernized democratic society? These principles seem to be deeply embedded in the minds of many journalists, politicians, and members of the public. Let me try to be more explicit:
Actually, on the suspicion that there might be something wrong with Principle 3, it tends nowadays to be replaced by a more sophisticated version:
Corollary 1: Despite sounding reasonable the principles are incoherent, wasteful, and socially crippling. Their application will tend to reduce a nation's global competitiveness. They are part of the trend toward dumbing down -- toward soundbite madness, toward instant remedies, toward unilateral intellectual disarmament, toward apparatchik-ridden bureaucracy. (Something like this might be worth saying in public. But beware! I argue in lucidity3 that the unquestioning belief in absolutist principles -- ahead of conscious thought, and flying in the face of the simplest logic -- is a manifestation of the same human hypercredulity instinct that leads not only to totalitarian bureaucracy but also to ordinary religious fundamentalism, witch-hunting, and other socioanthropological phenomena of comparable power. `Look how that wicked person has brought the demons and succubi among us. Everyone knows that the demons are real. Has not the Pope himself said so? Something must be done. What could be more reasonable?')
Corollary 2: Because advanced human societies need a highly skilled, intelligent, reliable, enterprising workforce, they will come to recognize, notwithstanding the hypercredulity instinct, that auditing and measuring everything is both a mirage and a huge waste of resources, and that there's actually no alternative to reliance on trust -- and to rebuilding trust where necessary -- for instance trust in professional ideals and ethics. An advanced society will recognize this explicitly and live with the risks. It will use auditing resources in new and cost-effective ways by concentrating them not on trying to monitor and measure everything but, rather, on checks and balances against gross human failings. It will find new ways of valuing and encouraging professional ideals and ethics, instead of devaluing and discouraging them through Principles 2 and 3. It will find new ways of promoting flexibility rather than rigidity. It will recognize the validity of a deeper and more ancient principle, one that has applied for hundreds of millions of years throughout the living world, and will continue to apply in future: survival needs diversity.
Knowing the importance of flexibility and diversity, an advanced society facing the uncertain future will also find new ways of encouraging, among schoolchildren and their teachers for instance, other unmeasurables such as the curiosity, interest and enthusiasm, and the personal room for creativity, that are crucial to the development of high-level skills, versatility, and world-beating performance in an uncertain future. An advanced, knowledge-based society will have rediscovered and acknowledged the fact that the most highly developed skills and learning come from the ignition of interest, not the imposition of auditing.
An advanced society will have cleared up the confusion between three very different things: (a) the need for quality control of mechanical processes like manufacturing nuts and bolts, or other routine processes like medical testing, (b) the need for checks and balances against human irresponsibility, and (c) the need to promote enterprise, creativity and problem solving at a high level. Present ideas of educational `standards', in the UK at least, thoroughly conflate all three, as the Summerhill affair made starkly clear, to say nothing of the situation in the UK universities.
Perhaps we can hope for an increasing public acknowledgement of these points by more of our journalists and politicians, perhaps even some soundbites from the heights of government, daring to say -- oh sacrilege! -- that auditing and rigid rules are subject to Goodhart's law and not the Answer to Everything, or even -- oh horror! -- that some things of supreme importance are not numerically measurable. I for one am grateful to Andrew Dilnot and the BBC's Analysis, and the experts interviewed on the programme, for bringing out many of the points so clearly and honestly. (One of the experts, an eminent economist and the Director of the Said Business School, Professor John Kay FBA, went so far as to admit that `we are intellectually in a mess.') If these points were seen to be more widely appreciated, and officially appreciated -- with perhaps a touch of humility in high places -- then there might begin to be some abatement of the skills shortages and shortfalls that hold back the knowledge-based economy.
There is a still deeper point in all this. Why does survival require diversity? Why do we need such things as professional ideals and ethics instead of rigid rules, if we want our society to realize its potential and sustain a skilful workforce? There is a fundamental reason: combinatorial largeness. This is the largeness of the number of grains on the proverbial chessboard (over 10,000,000,000,000,000,000), or of a compound interest factor over a million years (ca. 1010,000), or of the theoretical number of states of the human genome (again ca. 1010,000, a number of the order of 1 with 10,000 zeros after it). The number of pathways to the future is still larger, and not only full of danger but also full of hope, as illustrated by the marvellous confounding of pundits by the Grameen Bank of Bangladesh. If the way ahead forks once, there are 2 ways to go. If each branch forks again, there are 2 x 2 = 4 ways. If the pattern repeats itself eight more times then there are 210 = 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 1024 ways, and if eighty more times there are 282 = 4,835,703,278,458,516,698,824,704 ways, nearly 5 million million million million. The ways ahead keep on forking for ever. Ten more forks multiply the possibilities another thousandfold, twenty another millionfold, on and on and on. As human evolution emerges from the Palaeolithic and systems and societies become ever more complex, the number of possibilities to be coped with increases inexorably and unimaginably steeply, like compound interest as time increases. Such numbers soon exceed the number of rules in all the Earth's audit books, and even the number of lines of code in the Internet. It's no wonder that so many things are unforeseeable.
Today's audit culture, knowing nothing of combinatorial largeness and its implications, takes it for granted that there is one best pathway -- one `best practice' -- and that everything is best constrained in advance by setting rigid targets, measures, and frameworks of rules, just as in sausage manufacture. (For what could possibly be fairer or more modern? We don't like dead rats in our sausages, and we don't like bad eggs in our institutions.) The audit culture, collectively speaking, thus fails to remember that human beings, along with other life forms, normally and naturally deal with combinatorially large -- unimaginably large -- numbers of unforeseen, and unforeseeable, possibilities. Many of these are, of course, ways for things to go wrong; and humans are quite good at coping with things going wrong, provided their hands aren't tied too tightly. By the same token, rigidity -- if taken seriously -- would be a recipe for vastly increasing the number of things that go wrong, especially in complex systems engineering like computer software and genetic engineering, and in medicine, in education, and in human societies themselves.
The same mistake -- the failure to grasp the implications of combinatorial largeness -- shows itself in dogmas of behaviourist psychology and in the tabula rasa theory of culture and education. The child's mind is a blank slate, we are assured by people who know nothing about brain function, a blank slate on which culture can write its imprint -- its finite, measurable, auditable, testable imprint. This could be part of why the audit culture is so confident that it can improve educational establishments -- that it can `raise standards' -- by insisting on rigid targets, measures, and rules. And we now seem to be in danger of taking rigidity far too seriously, with devastating effects on tomorrow's workforce and tomorrow's competitiveness. Confidence is becoming confidence trickery, deeply inadvertent though it may be.
The point was sharply underlined by events that hit the news in 1999, leading to an exchange in the House of Lords on 30 June. One of the most advanced and successful school-age educational establishments in the world, Summerhill School -- successful even by conventional measures such as examination results, and widely respected for its extraordinary contribution to democracy and for fostering of all kinds of intelligence, creativity, enterprise, and adaptability -- was put under threat of closure for no other reason than daring to take a stand against the rigidity of audit-culture principles, and for daring to show again and again, by clear example, that there is more than one good pathway to the future. Here is a telling illustration of how rigidity takes us toward disaster, like a bus driver so sure that the road is straight that he shuts his eyes and locks the steering wheel.
As far as I know, the Summerhill case still hangs in the balance. (UPDATE: their Appeal Tribunal was heard in the week of 20 March 2000. On the first day, it was put to the tribunal that `In any intelligent education system Summerhill would be viewed as a precious resource.' On the third day, the auditors, under the glare of cross-examination, withdrew the immediate threat of closure. They also offered to contribute to legal costs; it seems likely that they were afraid of what might be revealed under further cross-examination. However, the auditors' public statements after the settlement make it doubtful whether the affair is really over.) If you have never heard of Summerhill, take a look at some of its founder's ideas (for instance here and here). These ideas show profound insight into human nature, our genetic-memetic inheritance in all its future-busting diversity. They are in close accordance with the findings of leading researchers on behaviour and development, such as Professor Patrick Bateson, presently Biological Secretary of the Royal Society of London. See for instance the beautiful new book Design for a Life: How Behaviour Develops, by Bateson and Martin (Jonathan Cape, Random House, 1999), and another beautiful and insightful book The Runaway Brain, by Christopher Wills (BasicBooks 1993, HarperCollins 1994).
The audit culture is, collectively, ignorant of all this. In promoting and enforcing single pathways and single `best practices' to cover all circumstances it unconsciously assumes that people are, or ought to be, simple machines, like clockwork, predictable and controllable in simple ways. It is an old and oft-repeated mistake, one of the worst mistakes of Enlightenment thinking and, incidentally, a prime reason for the misuse of science and the distrust of science. `Look what Newton did with planetary dynamics; let's do the same with social dynamics!' It's exactly the mistake that led first to behaviourist psychology and then to the horrifying experimentation on humans and human societies by 20th-century totalitarian states. Such is the real price of taking rigidity seriously. As Summerhill School and the Grameen Bank remind us through their obvious success -- along with countless other big and small examples -- humans are not simple machines. They -- we -- have astonishing potential and astonishing adaptability.
Think of it! Europeans don't burn witches any more. We no longer panic at the sight of a comet. Demons in the air have shrunk to a small minority of alien abductors. The Grameen Bank has released millions of women from child-bearing slavery, in Bangladesh and elsewhere. We have an Internet that works. We are reaching new levels of self-understanding, scientifically as well as intuitively. We are beginning to respect in full measure the complex reality not only of humans, not only of computer networks, but also of `simple' life forms as they used to be called. Even an Escherichia coli bacterium, less than a thousandth the mass of a single human neuron, is not a simple machine. A single E.coli, we now know, is dauntingly complex, with a combinatorially large number of internal states, fluctuating, reconforming and evolving under thermal agitation on picosecond timescales -- vastly too complex for us yet to simulate in detail, even with today's most powerful supercomputers, let alone to understand in detail. An E.coli contains millions of protein molecules of up to thousands of types, many of them acting as computational elements in some kind of massively parallel information-processing operation. Contemplating an E.coli in our present state of knowledge is a bit like contemplating the Internet when you have just found out how wires and transistors work and programs stored but know very little about programming languages, and nothing at all about multi-scale computational architectures and interacting layers of software. (An example to underline these points is PTGS, post-transcriptional gene silencing. We know it happens and presume that it is important for survival; but we have no idea how it works: e.g. Science 290, 1159 and 1108, 10 Nov 2000.)
Today's audit culture knows nothing of complexity and complex systems. It assumes, unconsciously, with the best of intentions, that the road ahead is straight and that the steering wheel not only can but must be locked, for long stretches of time. It assumes, unconsciously, that `accountability' means nothing but strict conformity to yesterday's objectives -- yesterday's measurable objectives -- whether by self-auditing or external auditing, and however imperfect the forward view may have been when yesterday's objectives were set. It equates success with measurable `output', the measures being chosen in advance. (For what else could possibly be fair and objective?) A clear example of the damage this is doing to the very heart of commerce -- never mind education, enterprise, and innovation in general -- is given in James Gleick's discussion of the crisis in patent law. In that case the damage, or rather the measurable fraction of it, the mere commercial loss, is now exploding toward hundreds of billions of dollars per year. Patent examiners `measure their own performance in terms of their output', which of course is the number of patents approved, regardless of whether the patents make sense. (More is Always Better: what could be more reasonable? And too bad that judgments of `making sense' are not measurable. That's unrewardable professional ethics.) No wonder the incredible and unthinkable is now happening: the patenting of naturally-occurring patterns, the patenting of nature herself, the patenting of the very genes that you and I and our children have inherited from hundreds of millions of years of evolution. Goodhart's law cuts deep.
The culture needs to reform itself, not by the auditing of the auditors of the auditors but by rediscovering for itself, collectively speaking, the limitations of measurement, and the supreme importance of diversity and flexibility. It needs to rediscover the supreme importance of trust and responsibility, including professional ethics, despite their occasional abuse. We already trust our bus drivers and airline pilots. An advanced human society will have accountability mechanisms of a new and more coherent kind, mechanisms that provide checks and balances yet respect Goodhart's law and don't gratuitously destroy trust. Such a society will prepare for the future by opening its eyes and unlocking its steering wheels, by rediscovering and acknowledging the complexity, and unpredictability, of the living world of which we are part, and by nurturing its astonishing human potential, which itself we must once again learn to trust.
I am grateful to Professor Marilyn Strathern FBA for some socioanthropological perspectives. See also, for instance, the little book The Audit Explosion by Michael Power (London, Demos, 58pp., 1994). This is a sober, thoughtful, and carefully-argued analysis by a professional from the financial auditing world (Michael Power was Coopers Lybrand Fellow and is now Professor at the London School of Economics). The book looks toward possible ways to regain balance, moderation, and realism. It seems that by 1994, in finance, the expansion to two levels (auditing of the auditors) was already in place, but apparently not to three levels by then. The Summerhill affair must surely be taking us closer to two levels in educational auditing. (`For look at the damage those government inspectors did, and the lies they told. Now we can't trust any government inspector.') I haven't seen the infinite regression, the expansion to limitless numbers of levels and limitless cost implied by Principles 1 and 3, explicitly mentioned except here and in lucidity3 (string-search `neoaccountability'), or on p. 46b of the published version, Interdisciplinary Science Reviews 23 (1998), though Professor Power tells me that it is mentioned in his longer book, The Audit Society: Rituals of Verification, Oxford, OUP 1997. Also relevant is the forthcoming compendium edited by Strathern, M., Audit Cultures: Anthropological Studies in Accountability, Ethics and the Academy. London: Routledge, 2000 (in press).
Note added 15 February 2001: The publication of the draft human genome in today's Nature and tomorrow's Science compels one minor revision above. The theoretical number of states of the genome, which an educated guess would have put at very roughly 1030,000 until recently, now seems likely to be closer to 1010,000. This is still unimaginably large: 1 with 10,000 zeros after it. The theoretical number of states (with some genes turned on and some turned off) is 2 raised to the power of the number of genes in the genome. Educated guesses at the number of genes used to be of the order of 100,000, but this is being revised downward to more like a third of that number.
Note added 14 May 2002: This year's BBC Reith Lectures by Baroness Onora O'Neill added powerfully to the voices trying to blow the whistle on audit madness. In the third of five lectures, O'Neill attacked the problem head-on and pleaded for `intelligent accountability'.
Last substantial revision 3 December 2000.
Last updated 14 May 2002.