Radical
Statistics
POLICY RESPONSE
Trust versus scope: A critique of the White Paper Building Trust in Statistics
Ray Thomas
The main proposal of the White Paper - the establishment of a Statistical Commission - is a development of great potential significance. A Statistical Commission can be expected to increase the involvement of members of the public and user organizations in the production of official statistics. This involvement can be expected to make a positive contribution to the quality of the established range of official statistics. The Commission could also become a channel of communication that could bring about improvements in the coverage of official statistics. Those who believe that official statistics should portray social and economic conditions could try to enlist the support of the Commission.
In other ways, the White Paper is disappointing. It is difficult to deny the importance for government and for the Government Statistical Service (GSS) of the development of trust in statistics. But it is a mistake to make trust the centrepiece of statistical policy as the title of the White Paper proclaims. As every social scientist should know, trust in statistics should always be limited. Building unjustified trust would be an enemy, not a friend, of statistics.
There is little need to build trust in statistics for births and deaths because these statistics are the product of carefully controlled registration procedures. But trust should only grudgingly be given to many other kinds of statistics because they are the products of government policies. Such statistics typically measure government activity rather that the social conditions they purport to describe. Statistics of Claimant unemployment are a well-known example.
Some statistics are inherently unreliable. Statistics concerning homelessness, tax fraud, benefits fraud, immigration and the informal economy are obvious examples. Should statistical estimation in such areas be discouraged or ignored because of the difficulties of producing statistics that are worthy of trust?
The emphasis on trust in the White Paper is associated with failure to acknowledge an inescapable variation in the quality of statistics and a failure to acknowledge the wide variety of functions fulfilled by statistics. These failures have stunted the development of the otherwise promising concept of National Statistics. The Green Paper boldly speculated that National Statistics should aim to 'describe the state of the nation' (Treasury, 1998, 4.5). But the White Paper cuts this ambition down to the bone by limiting the scope of National Statistics to the outputs of the Office for National Statistics (ONS).
The cutting down has come from the government not from the GSS or from other statisticians. Tony Blair's introduction to the White Paper does not acknowledge any role for statistics beyond those of helping government decision-making and assessing governmental performance.
The Statistics Commission
The proposal to create a Statistics Commission is a celebration of the overturning of the Rayner Doctrine that statistics should be produced primarily for government's own business. It is also a personal triumph for Ian MacLean, kingpin of the Statistics Users Association, who put the idea for a Statistics Commission forward more than twenty years ago (MacLean, 1978), and which received powerful endorsement by Sir Claus Moser in his Presidential Address to the Royal Statistical Society (Moser, 1980).
The length of the gestation period, the authority of its originators, and the range of functions that could be undertaken suggest that the Statistical Commission will swiftly become an esteemed part of the British statistical system. It seems likely that in the not-to-distant future we shall wonder how we ever managed without a Statistical Commission.
Different paragraphs of the White Paper give different descriptions of the functions of the Commission. But four principal functions can be identified (Treasury, 1999):
- Priority setting and quality assurance (3.1-3.4)
- Providing users with a clear channel to express their views (2.7, 2.10, 2.12)
- Ensuring freedom from political interference (3.5-3.7)
- Advising on the scope of National Statistics (4.1-4.5)
The White Paper confesses to uncertainty about the practicality of the Commission performing the first three of these functions over the full range of official statistics. This gives rationality for limiting the scope of National Statistics. The White Paper proposes that the scope of National Statistics should initially be limited to publications of the ONS and 'with the agreement of Ministers, other statistics published by departments' (4.4).
Unemployment and other administrative statistics
The Statistics Commission could usefully get into a number of areas on which the White Paper is silent. One of these is the role of administrative statistics - that is statistics that are the product of government activity - as distinct from survey statistics that aim to establish facts independently of governmental activity.
The Report on the Consultation on the Green Paper found that there was 'broad support for the need for the Head of National Statistics to have access to all administrative data sources' (ONS 1999a, 4.9). Access to administrative data in the information age is crucial to the development of government statistics. But this access issue is not carried through from the consultation to the White Paper.
The White Paper is also silent on the problem of relating trust to administrative statistics. This is a serious omission because the White Paper, the preceding Green Paper and the pledge for an Independent National Statistical Service all originated with allegations of fiddling the Count of Claimant statistics under the Thatcher governments (see Thomas, 1998). The Count of Claimants, like most other administrative series, is exact and trustworthy. The Count trustworthily measures what it is supposed to measure - the numbers entitled to receive Job Seekers Allowance. And, like other administrative series, the Count of Claimants reflects changes in government policy and procedures.
The White Paper does not deal with the problems of relating trust to the Count of Claimants. It does not suggest that the ONS should take direct control of the Count of Claimants. The White Paper does not suggest that governments should abrogate their responsibility to decide who is entitled to receive unemployment pay. But the kind of issues raised by the allegations of fiddling Count of Claimants are likely to occur again and again with other administrative statistics such as those relating to crime, hospital waiting lists and educational performance.
The Statistical Commission could well say things that both the government and the GSS find difficult to say about administrative statistics. The Commission, with its degree of independence, could point out that there is no lack of integrity in administrative statistics such as the Count of Claimants, except that they are the products of governmental policy and administrative procedures. The Commission could advise on the importance of maintaining comparability where there are changes in the definitions and procedures that determine such administrative statistics.
The Statistical Commission could also act as a pressure group for open government and help to make administrative records available for the production of high quality statistics. The Commission could, further, aim to help the GSS to achieve compatibility between administrative statistics and the survey statistics that they often overlap with.
The earnings index fiasco
Since the publication of the Green Paper, the attitude of the government and that of government statisticians has been shaped by the experience of revision of the Average Earnings Index. Quality control in this instance was not up to standards wanted by users. Revision of the current estimates provoked criticism. But the strongest complaints were provoked by retrospective revision of the series. 'High power' users were upset by these revisions because they were obliged to recalibrate the statistical models that they used in conjunction with the current estimates.
It is unlikely that a Statistics Commission would by itself have anticipated or avoided these problems. But the Commission could perhaps have avoided the fuss made about what is an exceptional occurrence. The abject apology given by Tim Holt, the Head of the GSS, was hardly necessary.
A Statistics Commission could give a more balanced response. The Commission could well point out that statistical series, like the Average Earnings Index, have to be revised occasionally to take account of changing conditions, and that there is always a problem in balancing the need for changes with the desirability of maintaining continuity in the series. A Commission could point out that there is no need for government statisticians' responsibilities to extend as far as preserving the stability of the models users make on the basis of government statistics.
The fact that the GSS finds it difficult to say such things helps to explain why the White Paper is modest in defining the range of National Statistics. There are many statistical series that could be made to seem weak in the light of the intensity of the scrutiny given to the revision of the Average Earning Index. Hence the White Paper's confession of uncertainty about the practicality of the Commission performing its functions over the full range of official statistics.
Measuring the performance of government
The bulk of the White Paper seems to reflect the concerns of the ONS and GSS. Government statisticians want to be seen as the producers of trustworthy statistics, and they received strong backing on this matter from the Royal Statistical Society. But Tony Blair's introduction captures the essence of the government's attitude to statistics. The function of statistics is to help government decision-making. The integrity of statistics is important for democracy because the electorate can monitor the performance of government.
The Rayner doctrine that statistics should be collected primarily as they are needed for the business of government may have been overthrown. But Blair extends the Rayner doctrine only as far as to include statistics that measure the performance of government. There is no acknowledgement that statistics might be needed on social conditions independently of current government activities.
Blair has been seduced by the image of trustworthiness wanted by government statisticians into taking a narrow view of the scope of statistics. But Blair seems to be speaking for all sides of the House of Commons. The perception of statistics as measures of government performance is reinforced by the all party Treasury Committee Report on the ONS (Treasury, 1999).
Blair's perception of the value of statistics as measures of its performance must to some degree stem from the favourable picture given by economic statistics since the 1997 election. Like the Tsarina's mirror, the statistics have been telling the government that it is the fairest of all. But Blair should know that the statistics presented to government and to the public are the product of policy-making, as well as being a measure of the effectiveness of policy-making.
It would be hopeless for any government to deny, for example, that the use of performance indicators in areas of health and education do not influence the activities of teachers, doctors, nurses, and the activities of the administrative staff supporting these activities. The use of statistics as performance indicators is intended to influence performance, and of course performance indicators do influence performance. Performance indicators also influence the way performance is reported. Performance indicators do not cover all aspects that are matters of public concern. Giving attention to what is measured will mean neglect of aspects of performance that are not measured.
The images in the government's statistical mirror will usually give a relatively favourable reflection. But the real world is a different matter. The Tsarina's mirror eventually gave a negative response and got thrown away not because the Tsarina had become less beautiful but because of changes in the world outside; the Tsarina's stepdaughter had become more beautiful.
It is difficult to imagine William Hague fulfilling the kind of role fulfilled by the Tsarina's stepdaughter! But the simile is appropriate in other ways. The public will ultimately judge the performance of the government not by the government's own statistics but by the realities, not covered by the statistics, that people face in their everyday lives.
Getting facts about society
Blair's trust in statistics as measures of performance is excessive, but the emphasis he gives to the value of statistics for decision-making is fully justified. The typical function of statistics is to support decision-making - at all levels of government. But the corollary is that it is exceptional for statistics to be used to help formulate policy (see Thomas, 1999).
The distinction between statistics for decision-making and statistics for policy-making is important for the proper functioning of democracy. The exercise of democratic government involves the development of new policies, as well as decision-making in accordance with existing policies. The development of new policies should be informed by systematically collected information in the form of statistics.
The quest for trust conflicts with the need for statistics for policy-making. Statistics that really add to our knowledge and so have the potential to influence public policies often begin with small-scale surveys. Giving emphasis to components of trustworthiness such as reliability and generality derogate from the value of such statistics.
The Green Paper referred to the value of statistics for the formulation of policy (1.1). But there is no mention of the contribution of statistics to policy-making in the White Paper. Blair's characterisation of the function of statistics as supporting government decision-making implies a retreat from the position of the Green Paper. Statistics are not and should not be limited to this role. It would be a retrograde step if the creation of National Statistics led to neglect of statistics that aimed to describe the state of the nation - as was suggested in the Green Paper.
Responses of the Royal Statistical Society
The White Paper singles out the Royal Statistical Society (RSS) for its help in arranging open meetings in different parts of the country for discussion of issues raised by the Green Paper. The response by the RSS itself with 21 pages is the longest of all 200 responses to the Green Paper (ONS, 1999b, 030).
The RSS response proposed that National Statistics should include all statistics of public interest produced by government departments or any other public sector body at the national, regional, and local level. The RSS argued that in order to achieve integrity of National Statistics there needs to be 'a cultural shift' among producers and users of statistics. For example, government should no longer have access to National Statistics prior to publications.
The RSS response proposed a National Statistician with the status of a public servant, a Statistical Commission of twenty members representing the interests of users. The National Statistician and the National Statistical (sic) Commission would set up a system of self-certification for a system of national kitemarking to give evidence of quality control. The powers and responsibilities of the National Statistician and the Commission should be reinforced by legislation.
The RSS asserted that one of the responsibilities of the National Statistician and the Commission would be 'to comment quickly on major abuses by users'. The RSS response speaks of 'perceived abuse and misuse by Minister and officials'. But the main target appears to be the press and mass media that continue to report the monthly Claimant unemployment statistics in spite of the RSS recommendation of 1995 that the LFS unemployment figure should be the headline figure!
The RSS response, in summary, advocated that the National Statistician and the Commission should take control of all official statistics and rename them National Statistics. The government's reaction, as expressed in the White Paper, suggests that the RSS overplayed its hand. The White Paper shows acceptance of many of the RSS ideas, but has drastically curtailed their scope.
A more serious weakness in the RSS response is its blinkered outlook. There was no attempt to articulate the purposes associated with the production of statistics. It was left to the Radical Statistics Group (ONS, 1999b, 027) to remind the government that these purposes, though not expressed in the RSS Response to the White Paper, are expressed in the RSS Charter.
The Radstats response advocated that National Statistics should be approached in the spirit of the early aims of the RSS, namely: 'to collect, arrange, digest, and publish facts illustrating the conditions and prospects of society in its material, social and moral relations' (RSS, 1985). The Radstats response goes on to discuss the practical problems of achieving the production of such facts about society.
The RSS response deserves the same the-best-is-the-enemy-of-the-good epithet as the White Paper itself. So much emphasis is given to quality, integrity, and control by statisticians that the purpose of producing the statistics was forgotten about.
REFERENCES
MacLean, I. (1978), 'The Role of a National Statistics Council', Annual Statistics Users' Conference on Dissemination of Statistics, November, London: Central Statistical Office.
Moser, C. (1980), 'Statistics and Public Policy: The Address of the President', Journal of the Royal Statistical Society, Series A (General), Volume 143, Part 1: 1-31.
ONS (1999a), Statistics: A Matter of Trust. Report on the Consultation Exercise 24 Feb to 31 May 1998, London: Office for National Statistics .
ONS (1999b), Statistics: A Matter of Trust. Compendium of Replies to the Consultation Exercise, London: Office for National Statistics .
RSS (1985), Regulations and List of Fellows, London: Royal Statistical Society.
Thomas, R. (1998), 'Integrity for statistics or for statisticians?' Radical Statistics, 69, Autumn: 60-71.
Thomas, R. (1999), 'Statistics and policy making', Radical Statistics, 71, Summer: 66-73.
Treasury (1998), Statistics - A Matter of Trust - A Consultation Document, Cm 3882, London: The Stationery Office (The Green Paper).
Treasury (1999), Building Trust in Statistics, Cm 4412, London: The Stationery Office (The White Paper).
Treasury Committee (1999), Office for National Statistics, Vol 1 Report and Proceedings of the Committee, London: The Stationery Office.
Ray Thomas
Social Sciences
The Open University
35 Passmore
Milton Keynes MK6 3DY
Tel: 01908 679081
Fax: 01908 550401
r.thomas@open.ac.uk
