Radical
Statistics
Targets or Tracers? The Role of Numbers in Public Policy
Tim Blackman
Introduction
Few public services managers today are without responsibility for a set of quantitative targets that they are held accountable for achieving. The targets are typically defined higher up the hierarchy and are rarely the outcome of any significant participation in their definition by the units responsible for delivering them. There is usually not only pressure to meet targets but also responsibility for meeting requirements about the processes by which targets should be met, audited through inspections. And as Onora O’Neill has pointed out in her 2002 Reith Lectures for the BBC, there is no necessary connection between conforming to these process requirements and meeting the organisation’s targets (O’Neill, 2002).
The possible dysfunctional effects of targets are well-documented (Smith, 1995). They include gaming, for example when teachers focus on students at the borderline of an exam performance measure like the school’s number of GCSE passes at grade C or above, and biasing, such as when hospitals give a lower priority to clinical need in order to meet targets for the length of time patients are on their waiting lists. Other possible effects are sub-optimization, such as when a deliberately low target is selected to demonstrate subsequent success, and measure fixation, when time is diverted away from delivering the service to compiling reports on measuring performance. Important consequences of these problems are that managers are frequently caught between centrally-dictated targets and local priorities, and that results defined as the degree of success in meeting targets lose their credibility as a method for judging performance.
The response to these problems is often to seek better targets, and a great deal of effort has been invested in this type of work (Boyne, 2002). For example, if exam results do not measure the effectiveness of a school, the solution may be to use a measure of value added; if waiting list times do not measure the effectiveness of a hospital, the solution may be to use mortality rates. However, there are more fundamental problems with management by targets than the search for valid and reliable measures of performance, and these follow from the inadequate theory on which it is based.
Theoretical Problems with Management by Targets
The first problem is that in order to hold an identifiable management accountable for performance, the system which delivers the outcome that a policy objective aims for is often mis-specified. Thus, a school is specified as the system for delivering exam results when the school actually comprises many sub-systems, some achieving better results than others, and has inputs from many other systems, such as students’ achievements in feeder schools and their material and home circumstances (Fitz-Gibbon, 1996; Byrne and Rogers, 1996). These all substantially influence outcomes at school level: exam performance is likely to show more variation across subjects within schools than for all subjects across schools, and the characteristics of a school’s intake are much more important than the effect of the school alone in determining its exam results. Similarly, a housing department is specified as the system for delivering low void rates when the housing stock is part of a housing system governed by a number of key social and economic parameters that influence demand for social housing (Nevin et al., 2001).
There have been some attempts to address this issue, such as ‘whole system’ planning initiatives in health and social care. This approach received strong endorsement from the National Health Service R&D Strategic Review’s Working Group on Ageing:
It is essential that the unit of evaluation for older people with complex needs is a meaningful system rather than specific interventions, departments or services. The ultimate systems model for evaluation of health services is the population laboratory in which an epidemiological study identifies all people in a defined population who have a particular problem and then traces them through the health care system For example, a system for returning old people with proximal femoral fracture to their pre-injury levels of function and pre-injury domicile will include trauma and geriatric units, physiotherapy and occupational therapy, social services, primary care and housing. The relative contributions of these may vary from place to place as well as from patient to patient (Evans, 1999).
But while there is increasing recognition of whole systems in health and social care, it is still the separate sub-systems that are held accountable for performance, with their own sets of targets. This was recently exemplified by the Government’s April 2002 budget which threatens local authorities with financial penalties if they fail to provide the community care needed to ‘unblock’ hospital beds (Hanlon and Shaw, 2002).
A second theoretical problem with management by targets is that the approach assumes that the delivery system is in equilibrium so that a given input achieves a proportionate output: the conditions for ‘command and control’ management. There seem to be few examples of this input-output equilibrium and many that suggest there is no such direct relationship. Pawson and Tilley (1997) set out a whole new evaluation methodology based on extensive evidence that there is no direct relationship between money spent on crime reduction initiatives and successful outcomes because local contexts vary in ways that produce quite different interactions between the extra resources and local behaviour. Another example is the expectation following establishment of the National Health Service in 1948 that its cost would begin to decline as the population grew more healthy as a result of free access to health care, while in fact the cost of the NHS grew considerably (Blackman, 1995). And in the field of regeneration, Campbell (2002) points out that there is no relationship between job creation and reducing unemployment in a locality, nor any relationship between economic growth and job creation.
While there will be circumstances when additional resources produce proportionate outcomes, it seems more common for additional resources to produce disproportionately large or small outcomes or, probably more frequently, for the outcomes to vary unpredictably. The present Government largely ignores this complexity in its commitment to make better outcomes – defined, measurable and accountable - a condition of extra public spending, together with the associated apparatus of performance management and audit. The problem is that the systems into which it is injecting these extra resources are too complex for the relationship to be so direct. The Government then turns to bad theory: it must be poor management that is to blame and this can be exposed using performance indicators and league tables. Evidence is beginning to accumulate that this cannot be the sole or even main reason. For example, White (2002) shows that the local level of deprivation almost perfectly explains differences in performance between many local councils in London; where it does not, this seems to be a consequence of imperfections in the deprivation measure. Despite ‘beacon’ councils being held up as examples for others to emulate, their performance appears in general to be reflecting local conditions.
Interaction between inputs and system conditions mean that there is no ‘normal’ state for the systems that are expected to produce public policy outcomes. These systems do not exist as autonomous structures but emerge through interactions that produce structures ‘at the edge or order and chaos’. This term captures the idea of a system state that is a dynamic creation of interactions, neither very tight nor very loose in structure. The system started by having resources available, it imports these resources, actively organises by interacting with them, through this accumulates information and structural complexity, and exports transformed resources to its environment. These are the basic characteristics of dissipative systems: they are far from equilibrium and inherently evolutionary because of the influence of either external or internal change on their interactions and therefore their dynamic structures (Byrne, 1998). This produces variety rather than uniformity, and it is no more possible to command the behaviour of dissipative systems than it is to describe their behaviour using law-like statements (Kauffman, 1995). Targets assume the future is predictable for all systems, dissipative or not. While this could be true for the general properties of a dissipative system under given conditions, it cannot in general be true with any degree of accuracy over more than the very short-term.
A key feature of a dissipative system is that its system state depends on relationships between the system and other systems in which it is immersed. These other systems can be summarised as the system’s environment. Environmental parameters define the system states that are possible in a particular ‘phase space’, states that in complexity science are known as attractors. A change in a key parameter can flip the system, moving it from one attractor to another in a phase transition. Byrne’s work has illustrated this very well for neighbourhood systems in two regions of England as they flipped from sharing broadly similar social characteristics in a ‘full employment/universal welfare’ phase space during the 1950s-70s, to systems that had bifurcated between affluent and deprived attractors in the ‘deindustrialised/residual welfare’ phase space of the 1980s (Byrne, 1997). Urban policy interventions were largely irrelevant in influencing these neighbourhood trajectories and indeed had to catch up with the emergent phenomena which followed, such as neighbourhood crime and drug cultures.
A third theoretical problem with targets is that the local system state is regarded as amenable to command and control. The local system state comprises the initial conditions with which any intervention, parameter change or input of resources interact, producing an output that will depend on both these initial conditions and the subsequent dynamical interaction between the new resources and the system’s self-organisation. Internal conditions are more than the system’s internal resources and include its values. Stacey (2000) uses the idea of ‘shadow organisations’ to describe the informal relationships and private values that people live and work by in an organisation, contrasting these with the official relationships and values with which they interact. Progressive management practice seeks to achieve alignment between the values of the company and those of its employees and customers so that it can work on the basis of high levels of trust. While trust was once widely regarded as a feature of the public service ethos, the sector’s audit culture has undermined this at a time when the private corporate sector appears to be discovering the importance of trust and alignment. These are regarded as key to creating an internal system state is constantly emergent as people behave adaptively in responding to new situations informed by the values of their organisation. Targets are not regarded as reliable or valid as ends in themselves but are re-framed as tracers picking out the key features of change as it happens. Employees become agents of change alert to the feedback messages that these tracers send, modifying their own behaviour with the understanding that outcomes are co-produced between themselves, colleagues and customers, with the resources each brings to the interaction.
Thus, Fitz-Gibbon (1996) concludes from her study of school effectiveness that the key to effectiveness is to feed back information on valued outcomes to those responsible, so they can make adjustments that ‘close in’ on the desired outcomes. Organisations in which professionals have substantial autonomy are typical of the knowledge organisations that are regarded as the future of modern economies as well as many public services. They are self-organising systems and are not just created by policy-driven (command and control) or market-driven (survival of the fittest) ‘natural selection’. The people involved in delivering services are therefore best placed to constantly improve them. They are often best placed to locate problems and they possess knowledge about the problem over and above monitoring information.
However, the fact that outcomes are co-produced means that the delivery organisation is not the only part of the equation. Its customers, clients and users also produce the outcome by bringing their own resources and behaviour into an interaction with the formal service provider. This is increasingly understood in fields such as health and social care. It is also the means by which structural factors such as class, gender and ethnic inequalities enter the relationship in the sense that users are vectors of these inequalities, reflected in the resources and attitudes they bring to their encounter with the delivery organisation.
A pedagogic turn?
If performance management of target achievement is based on bad theory it might be expected not to work. A leading figure in the UK’s New Labour governments, David Blunkett, was recently quoted as saying, ‘There is mounting frustration in government that, after nearly five years in power, the promised transformation in public services has yet to be seen. Centrally set targets have proved elusive, and even the modest pledges of the 1997 election were unexpectedly difficult to achieve’ (The Guardian, 22 February 2002). Ambitious targets are not the problem here but an approach based on the introduction of targets that have little relationship to internal system states and little regard for the influence on performance of wider system parameters.
Organisations which attend to their internal system state, especially the knowledge of their personnel, trust and alignment, and scan and respond to their environments, especially with scenario modelling and anticipatory planning, are inevitably learning organisations. This is a term now frequently encountered in the management literature but there has been a surprising lack of engagement in this literature with pedagogic research, which might be expected to provide some useful clues for learning organisations. Key pedagogic concepts would seem to have direct relevance to issues of organisational performance, such as alignment between methods and outcomes, the importance of performing understanding, and learning as transformation. Issues of trust and blame, which now feature prominently in debates about targets and performance, are also prominent in debates about good pedagogy.
Biggs (1999) contrasts theory X and theory Y approaches to pedagogy, with the former assuming that students cannot be trusted and the latter supporting student autonomy and, in particular, student learning. Theory X is reflected in the following aspects, which are remarkably similar to the problems encountered with management by targets:
- Negative reinforcement, using anxiety to ‘motivate’.
- Blame-the-student explanations
- Time stress: failure to consider student workload and no time available for student reflection.
- Students given little input to decisions that affect them.
- Anxiety caused by harsh sanctions, bullying, sarcasm, lack of consideration of students’ perspective and work and time pressures.
- Cynicism, with students feeling that lecturers and not playing straight and don’t actually believe in what they are telling students.
The fundamental problem with management by targets is that it can engender a culture of ‘who is to blame and who is to be rewarded?’ Management by learning, on the other hand, aims for a culture of ‘What do we know and how do we find out what works?’ Table 1 elaborates this contrast.
Table 1: Comparing organisational cultures
Management by targets |
Management by learning |
Central leadership
Rule-based decision-making
Evaluation based on audit and performance indicators
Users are units of resource/statistics |
Distributed leadership
Flexible decision-making
Evaluation based on monitoring feedback and learning
Users are co-producers |
One approach to the learning organisation is the increasingly influential paradigm of evidence-based policy and practice (Trinder and Reynolds, 2000). This is the approach taken by Fitz-Gibbon and Tymms (2002) who place emphasis on the need for good indicator systems as well as controlled experimentation. For these systems, ‘An indicator can be defined as an item of information collected at regular intervals to track the performance of a system. The indicator systems that have formed the basis of our learning are all designed to feed back valuable information of interest to teachers and administrators in schools and colleges. We see our indicator or information systems as significantly empowering schools as they participate with a university in "distributed research"’ (Fitz-Gibbon and Tymms, 2002, p. 2).
This extends the whole system to include a university research team and the approach is informed by research and statistical methods. For example, Fitz-Gibbon (1996) advocates the use of confidence intervals in reporting performance indicators that compare ‘value added’ measures (residuals) with predictions derived from regression analysis. Fitz-Gibbon and Tymms (2002) identify the following requirements for good indicator systems:
- The indicators have good reliability and validity.
- The outcomes that are monitored can be altered.
- There is positive reactivity: the indicators promote change in the system’s behaviour.
- There is an effective technical infrastructure for generating the indicators.
- Data can be turned around quickly.
- The indicators can be readily interpreted.
- Residuals are inspected over time.
- There is good feedback and dialogue about the indicators and what they indicate.
- Experimentation is a feature of the indicator system, preferably using randomised controlled trials.
Although Fitz-Gibbon and Tymms (2002) argue for collecting a good range of indicators, there is an issue with their methodology due to the nature of public services as complex and interlocking systems. Their approach implies that the relationship between inputs and outputs is explainable in terms of linear regression. In fact, typically more than half of such variation remains unexplained and the relationship between inputs and outputs is non-linear and discontinuous. Interactions complicate any direct, uniform relationships between independent and dependent variables, with outcomes emerging from these interactions in ways that are very difficult to predict. This difficulty arises from the lack of precision that is possible in measuring the system’s initial conditions and a lack of knowledge about the system’s dynamical behaviour. In other words, the systems that may produce the outcomes desired by policy aims are complex systems as understood in terms of complexity science (Eve, Horsfall and Lee, 1997).
Complexity
Both management theory and evaluation methodology have converged on a new type of systems thinking as offering the conceptual tools needed for understanding complexity (Stacey, 2000; Pawson and Tilley, 1997). Complexity reveals itself over time not in terms of uniform relationships but of transitions in system behaviour due to interactions with either internal or external perturbations. Such a trajectory might start with a steady state, or what is known as convergence on a single attractor, when all iterations say the annual cycles of resource allocation return the same outcomes. Then, as a key parameter changes such as a cut in budgets the system begins transforming and outcomes flip between different values, with local systems starting to diverge into different states. Finally, the parameter value may change further and shift the system into chaotic behaviour so that local systems start returning different values in wholly unpredictable ways. Although a conceptual sketch, this scenario might well capture the effects on delivery organisations of increasingly cutting back their budgets. It could, for example, explain the observation made by Blackman, Brodhurst and Convery (2001) that standards of social care for older people among European states shows increasing internal variability as the national social care system becomes increasingly less well-resourced.
Blackman (1991) has suggested involving local residents and community groups as well as planners and politicians in monitoring and acting upon change across urban areas by using geographical information systems. Spatial monitoring of trends over time drawing on a range of flow indicators has the potential to detect emergent patterns. Interventions can then be geographically targeted, especially as many public services are already geographically organised. Nevin et al. (2001) use this approach in their analysis of spatial concentrations of neighbourhoods at risk of abandonment along the M62 corridor. They treat neighbourhoods not as ‘cases’ but as systems of relationships, with the neighbourhood embodying sets of wider relationships as a site of local interaction. Although not informed by complexity theory, their analysis reveals complexity. They demonstrate the importance of initial social and physical conditions at neighbourhood level, identify the influence on neighbourhood systems of wider system parameters such as the employment level and interest rates, locate attractors that are either stable or unstable, and highlight the significance of positive and negative feedbacks. They call for interventions that will achieve phase transitions by creating a new landscape of stable attractors for neighbourhoods. This is a long way from the urban policy measures that have more typically been used to address these problems, and which have delivered relatively little new net funding and been managed through a plethora of local targets.
Conclusion
To argue for an alternative to management by targets in the public services is not to argue against change and transformation. It is to argue for recognising self-organisation in information-rich environments and the need to align these conditions with the aim of improving the system’s performance. It is the people involved in running the system who are best placed to improve it on a constant basis, given conditions within the organisation that are learning-orientated and discursive. This implies a substantial degree of democratic organisation and flat structure to avoid the distorted communication that emerges from major imbalances of power (Habermas, 1984). It also implies that diversity and redundancy are not squeezed out of the system, as they are essential elements of innovation and adaptability (Kauffman, 1995; Stacey, 2000). Finally, targets are re-formulated as tracers, the monitoring information needed to track change. If this information is fed back continuously, it encourages constant dialogue about what it means, where the organisation is going, and where it should be going. This learning is then enacted, which will produce new feedback, new learning and new action.
Table 2 is an attempt to summarise how managing complexity looks quite different to managing by targets according to the conventional performance management paradigm of today’s audit culture. The systems are defined differently and the key variables and parameters are different.
Table 2: Comparing performance management and complexity
Performance management paradigm |
Complexity paradigm |
|
System boundaries |
Neighbourhood Hospital |
Urban economy Care pathway |
|
State variables |
Housing voids and units Patient throughputs |
Quality of life Quality of care |
|
System parameters |
Targets Targets |
Inequality Resourcing formulas |
Column 1 is an operationalisation of performance management theory based on command and control. Column 2 is an operationalisation of complexity theory based on whole systems, alignment and interaction. This article has sought to argue that the latter should work better than the former because it is based on theory of how things actually work, freed from the distortions and dysfunctioning of performance management as we mostly know it today. However, insofar as the conventional performance management paradigm serves dominant interests in a highly unequal world, its emergence is by no means only a question of winning rational argument! The article’s arguments, though, are provisional and more research is needed to test them, not least to investigate whether the huge number of targets that now characterise public policy in the UK bear any significant resemblance to outputs and outcomes.
References
Biggs, J. (1999) Teaching for Quality Learning at University, Buckingham: Open University Press.
Blackman (1991) Planning Belfast, Aldershort: Avebury.
Blackman, T. (1995) Urban Policy in Practice, London: Routledge.
Blackman, T., Brodhurst, S. and Convery, J. (2001) Social Care and Social Exclusion, London: Palgrave.
Boyne, G. A. (2002) ‘Concepts and Indicators of Local Authority Performance: An Evaluation of the Statutory Frameworks in England and Wales’, Public Money & Management, 22 (2), pp. 17-24.
Byrne, D. (1997) ‘Chaotic places or complex places: cities in a post-industrial era’, in S. Westwood and J. Williams (eds) Imagining Cities, London: Routledge,
Byrne, D. (1998) Complexity Theory and the Social Sciences, London: Routledge.
Byrne, D. and Rogers, T. (1996) ‘Divided spaces:divided schools’, Sociological Research Online, 1 (2) <www.socresonline.org.uk/socresonline/1/2/3.html>.
Campbell, M. (2002) ‘Beyond the fragments? Growth, jobs and inclusion’, in Sheffield Hallam University (eds), Joined-up regeneration: Objective 1 and urban regeneration grasping a unique opportunity, The Sheffield Hallam University Chancellor’s Conference, 22 June 2001, Sheffield: Sheffield Hallam University Press, pp. 3-5.
Evans, J. G. (1999) Ageing and Age-Associated Disease and Disability, Report of Topic Working Group, NHS R&D Strategic Review <www.doh.gov.uk/research/documents/rd3/ageing-final-report.pdf>.
Eve, R. A., Horsfall, S. and Lee, M. E. (1997) Chaos, Complexity, and Sociology, London:Sage.
Fitz-Gibbon, C. T. (1996) Monitoring Education: Indicators, Quality and Effectiveness, London: Cassell.
Fitz-Gibbon, C. T. and Tymms, P. (2002) ‘Technical and Ethical Issues in Indicator Systems: Doing things right and doing things wrong’, Education Policy Analysis Archives, 10 (6) <http://epaa.asu.eda/epaa/v10n6/>.
Habermas, J. (1984) The Theory of Communicative Action I: Reason and the ationalization of society, Boston:Beacon Press.
Hanlon, J. and Shaw, V. (2002) ‘Brown’s cash cow firmly tied to hospital bed post’, Local Government Chronicle, 19 April, p. 1.
Kauffman, S. (1995) At Home in the Universe: The Search for Laws of Self-Organization and Complexity, London: Viking.
Nevin, B., Lee, P., Goodson, L., Murie, A. and Phillimore, J. (2001) Changing housing markets and urban regeneration in the M62 corridor, Birmingham: Centre for Urban and Regional Studies, University of Birmingham.
O’Neill, O. (2002) ‘Called to Account’, Reith Lectures 2002, Lecture 3, <www.bbc.co.uk/radio4/reitH3002/3.shtml>.
Pawson, R. and Tilley, N. (1997) Realistic Evaluation, London: Sage.
Smith, P. (1995) ‘On the unintended consequences of publishing performance data in the public sector’, International Journal of Public Administration, 18 (2 & 3), pp. 277-310
Stacey, R. D. (2000) Strategic Management & Organisational Dynamics, Harlow: Prentice Hall.
Trinder, L. with Reynolds, S. (eds) (2000) Evidence-Based Practice A Critical Appraisal, Oxford: Blackwell Science.
White, S. (2002) ‘League leaders’, Local Government Chronicle, 19 April, pp. 12-13.
Wiggins, A. and Tymms, P. (2002) ‘Dysfunctional effects of League Tables’, Public Money & Management, 22 (1), pp. 43-48.
Tim Blackman
School of Social Sciences and Law
University of Teesside
Middlesbrough
TS1 3BA
Email t.j.blackman@tees.ac.uk
