Skip to Content


Home » Research Projects » Metrics, Targets and Performance

Missing the target

Can current measures of performance in the NHS be improved?

The aim of performance measurement and target setting by government is to improve the quality, performance and accountability of public services such as health and education. But do targets achieve this aim? To evaluate the extent to which targets are met, and any unforeseen adverse consequences of target setting, requires robust, comprehensive performance measures which cover the full range of services being delivered. Otherwise, providers may concentrate only on areas that are measured to the exclusion of others. Are current measures of performance sufficiently robust to achieve the government’s goal of improved public service provision? Researchers will consider a range of general issues relating to performance measurement and targets in public service provision. They will then explore these issues using the specific example of NHS Trusts. The study will identify all the elements required to make a performance measure effective and investigate whether these are contained in existing performance measures. It will examine the effectiveness of targets currently used in the NHS and then develop and analyse a number of alternative performance measures. What the research means for policy makers and the wider community

Research methods In the first, desk-based phase of this project, researchers will explore the theoretical and statistical aspects of alternative methods of performance measurement. The second phase of the project will be based on analysis of NHS data. These data include the number of finished consultants’ episodes, inpatient waiting times and mortality rates, and information on various types of staff, capital and intermediate inputs, as well as the NHS Star Ratings themselves.

Further Information: Project Posters

Updated Project Poster 2009

Below is a summary of this project’s provisional findings. It was originally presented as a dissemination poster, which is available here as a pdf document. All figures can be found at the bottom of this poster summary as thumbnails, which one should click to view full-size images. Alternatively, where figures are reffered to in the text, click the linked text for a full-size version.

 

ARE STAR PERFRORMERS MORE PRODUCTIVE?

Background

To evaluate any public sector organisation we need robust performance measures, but unlike the private sector – where the quality and quantity of outputs are more identifiable – public sector productivity is notoriously difficult to measure (Figure 1). From 2001 to 2005, the UK Government introduced star ratings to evaluate the performance of NHS Trusts in England. Numerous criticisms were levelled at star ratings and the associated target system by health professionals (including claims they led to financial imprudence and distorted clinical care for political reasons), but the relationship between performance on star rating- type performance measures and hospital productivity remains unclear, and the need for a measure of output remains.

Aims

Given that an ideal performance measure accurately quantifies performance, encourages improvements in the quality of the public service being delivered and avoids unintended consequences, our aim was to: » consider what the desirable properties of such measures are, theoretically; » evaluate how far existing measures in the NHS (such as the star ratings) satisfied these desirable properties; » examine other potentially useful measures, specifically a productivity index that measures the outputs of NHS Trusts relative to the inputs required to produce these outputs.

What We Did

To measure outputs, and thus productivity (in the absence of the market prices used in the private sector to indicate the value of the services provided) of NHS Trusts, we placed data from the Hospital Episode Statistics database (which provides extensive information on all NHS hospital activities in England) within a theoretical framework that uses information on outcomes to estimate the nature of the services provided, for almost all NHS acute trusts in England between 1997 and 2004. This output measure was divided by labour input (drawn from DoH data on earnings and labour volume) to create a productivity index. The productivity index was then compared to existing measures and used to highlight the influence of background variables, such as relative deprivation, which could affect productivity of different trusts.

Provisional Findings

» There was no statistical relationship between productivity and star ratings; we found huge variation between Trusts’ productivity across and within star rating categories (Figures 3 and 4). » A possible explanation is that key targets (Figure 2) put little weight on the financial aspects of performance for some rating levels, thus removing incentives to balance benefits and costs of providing services that would exist without targets. »The context in which trusts operated was important; trust productivity levels in 2003/04 correlated positively with social services expenditure on over 65s and negatively with the extent of patient deprivation. More qualitative evidence of hospital productivity against operating context would therefore add significantly to our ability to evaluate trust performance.

Figures

Click on the figures to enlarge omahonyfig1.jpg omahonyfig2.jpg omahonyfig3.jpg omahonyfig4.jpg

Other Project Outputs and Related Webpages

July 2006: Metrics, Targets and Performance, National Institute of Economic Review, Number 197.

Research Team

Mary O'Mahony

Mary O'Mahony

Mary O’Mahony is a Senior Research Fellow at the National Institute of Economic and Social Research (NIESR) and Professor of International Industrial Economics at the University of Birmingham Business School. She is also a Public Services Fellow of the ESRC funded Advanced Institute for Management Research (AIM). Her current research interests include: gauging the impact of information technology on productivity and measuring productivity in public services such as education and health.

Tel: 0121 414 3940
Email: m.omahony@bham.ac.uk

Philip Stevens

Philip Stevens

Philip Stevens is a Senior Research Fellow at NIESR. Since arriving at the Institute in 2000, his work has included: a study of the effect of tuition fees on universities; investigations of different methods of cost efficiency analysis in local government; and a feasibility study of the analysis of public sector productivity funded as part of the government’s Evidence Based Policy Fund; a study of recruitment and retention difficulties in UK higher education; and an investigation into the measurement of output in the NHS.

Tel: 020 7654 1927
Email: p.stevens@niesr.ac.uk

Lucy Stokes

Lucy Stokes

Lucy is Research Officer at the National Institute of Economic and Social Research (NIESR).

Tel: +44 (0)20 7654 1933
Email. l.stokes@niesr.ac.uk