Radical
Statistics
Response to A National Framework for Assessing Performance - a consultation document
Radical Statistics Health Group
Radical Statistics Health Group is a group of statisticians and others concerned with the way statistics is used in political debate about health and the NHS. It is not affiliated to any other organisation. We welcome the demise of the 'purchaser efficiency index', which we have frequently criticised, and see the development of the new framework as a very positive development. Nevertheless we have some concerns about the way it is being taken forward.
Availability and quality of data
The consultation document is realistic about the variable quality of the data available and the gaps in it. It is disappointing, therefore, that there is no systematic attempt to review the scope and quality of NHS data and the approaches that could be taken to tackle the problems identified. There is also no discussion of which aspects of the framework can potentially be monitored through routine data collection and which need a more in-depth approach, such as surveys or qualitative methods, to look at very specific aspects of care. Instead, many references are made to individual areas where new indicators are being developed, but no mention of any overall strategic approach to what is being done and how this relates to the development of information systems.
Monitoring inequalities in health
As the publication of the framework comes at a time when inequalities in health are being deserved prominence, a major omission from the framework is the use of socio-economic indicators to help in the interpretation of differences between and within populations. We strongly welcome the emphasis on monitoring social and geographic inequalities in both health and access to care, but are disappointed at the lack of discussion about how this should be done. In particular, there is a dearth of data about the socio-economic background of service users, and where data such as occupation are collected, this is not usually done in a structured or consistent way.
Activity data
The 'purchaser efficiency index' has left an unfortunate legacy in NHS statistics. Its overwhelming focus on activity has distorted the scope and range of NHS data collection. This should be explicitly acknowledged and the strategy being developed to overcome this bias should have been outlined in the document. Although we have frequently criticised the use of activity statistics as proxy measures of efficiency, performance or the 'success' of the NHS, this does not mean that we think they have no value and should be discarded. It is important to continue to collect data to monitor NHS activities and to enhance them through record linkage and in other ways, in order to create person-based records. Once activity data is no longer seen as measuring anything apart from activity, its quality and usefulness could well increase!
Interpreting differences between populations and service providers
Apart from referring to 'perverse incentives', the document does not explicitly acknowledge the extent to which observed differences between NHS trusts and between general practices may reflect differences in the way data are recorded. These can arise from differences in hardware and software, from differences in the way staff use and from differences in the quality and completeness of data recording.
Although allusions are made to the fact that indicators, particularly in areas I and VI, are likely to reflect factors outside the NHS, there is no explicit discussion of the extent they can be used to assess NHS performance. In particular, some discussion is needed of any possible temporal association between NHS input and possible output. There are also major problems in using costs as a measure of productivity, particularly in relation to capital charges and the impact on them of the private finance initiative.
Although it is stated that the framework can be used for monitoring by health authority, primary care group or NHS trust, there is no acknowledgement that some data collected about users of particular services cannot readily be collated for geographical populations and vice versa. In particular, it is unclear how data would be compiled for primary care groups. It does not seem that they will be population-based and there are many inconsistencies in the ways in which general practices record information, even given the same hardware and software. Clearly much work needs to be done on general practice data in the context of a primary care led NHS.
Use of the framework
The differences between health authorities are highlighted in the consultative document and also in the white paper The New NHS, but no serious attempt appears to be made to assess what differences might be expected in the light of random variations and socio-economic differences in the population. Most of the graphs the document have vague arrows pointing at one suburban and one industrial health authority, but there is no sign of any attempt to look at variations within similar districts in the context of overall variability, because of the way data are presented. This might have been possible in 'Area VI' (: 10), where the districts are subdivided according to an undefined classification, had the data been presented as histograms within each type of area.
Use of indicators
In recent years, work on indicators has tended to focus on differences between areas in a given year. Monitoring time trends within areas might be of more relevance locally, as it would enable changes in outcome or proxy measures of health to be monitored in relation to initiatives aimed at changing but frequent boundary changes in recent years have made this almost impossible.
Ranking indicators for local areas into league tables has become a well established practice, but we question its utility. For individual localities, it would be more useful to make comparisons with national statistics and with the general distribution of other authorities might be more illuminating, particularly if health service indicators could be interpreted in the light of data about the circumstances of the population.
The proposal for a set of high level indicators
Given the problems which have been identified with the scope, coverage and quality of data, the focus on creating a small set of high level indicators is a cause for concern. We do not think that this is the best use of scarce resources at this time. In particular, it compares unfavourably with the very much more constructive approach taken in Table 3 on page 12 of the document. It is a pity that so little emphasis was given to this and so much to the 'high level indicators' and we suggest that it is developed further, rather then going ahead with the high level indicators.
We have many criticisms of the proposed indicators, but do not have time for detailed analysis. Many are based on small numbers of events and in other ways fail to meet the criteria set out in Attachment C. Most measure some defined entity, but are interpreted as measuring a much broader range of factors. In the absence of evidence that these are highly correlated, they do not measure what they are stated to measure. Many are standardised for age and sex, but it might be more relevant to monitor differences between groups defined by age and sex.
It is not made clear how indicators are to be put together to form composite indicators, nor why there is any need to group data in this way, particularly if they are unrelated. The use of the retail prices index, which contains a large number of items, is cited as a precedent, but the analogy is inappropriate for a composite indicator containing only two or three possibly unrelated indicators. The resulting composite is likely to be sensitive to the method used to combine and weight the components. In addition, in the absence of evidence, it will be difficult for anyone, particularly the general public, to understand what they are meant to be measuring.
Given the scepticism about the old Patient's Charter indicators and the league tables for schools, it does not seem to be a useful exercise to proceed with this small set of 'high level indicators'.
Alison Macfarlane
for Radical Statistics Health Group
