Skip to Content


Home » Research Projects » Correlates of Success in Performance Assessment

A fair measure of success?

How performance assessment of public bodies may be falling short

Over the last 15 years British governments have searched for new ways of assessing how well bodies such as local authorities and health authorities are working. As a result, local authorities in England are now judged by way of comprehensive performance assessment (CPA) and health authorities receive a Star Rating score between zero (poorly performing) and three stars (high performing). Proponents of CPA and Star Ratings believe they lead to improvements in services. Critics consider them unreliable and cite the famous dictum of Charles Goodhart (the former chief economist of the Bank of England) that “when a measure becomes a target, it ceases to be a good measure”. This project will examine whether performance assessment succeeds in making local authorities more efficient and effective. It will identify the factors that lead a local authority to gain a ‘good’ or ‘bad’ score and investigate whether the CPA in particular is a fair, reliable and valid measure of success.

What the research means for policy makers and the wider community

Research methods

The project will assess the performance of British local authorities according to population size, indices of deprivation and type of local authority. Researchers will interview a selected sample of performance auditors and auditees from local authorities in England, Scotland and Wales. Researchers will also interview policy makers who have been responsible for assessment policies. Throughout the project, researchers will involve interested parties through this project’s partner organisation, the New Local Government Network (NLGN).

Further Information: Project Posters

Updated Project Poster 2009

Below is a summary of this project’s provisional findings. It was originally presented as a dissemination poster, which is available here as a pdf document. All figures can be found at the bottom of this poster summary as thumbnails, which one should click to view full-size images. Alternatively, where figures are reffered to in the text, click the linked text for a full-size version.

 

DOES POVERTY EQUAL POOR PERFORMANCE?

Background

Comparing the performance of local authorities with differing demographics is problematic, and different governments approach the problem in different ways. In England, central government assesses the quality of local authorities through Comprehensive Performance Assessments (CPAs), as summarised in Figure 1. Since it first appeared in 2002, CPA has remained controversial for its uniform and prescriptive approach and its disregard of local constraints such as deprivation that are not easily influenced by authorities. Is the CPA a valid measure of local performance, or does it merely reflect local deprivation and other external constraints? Scotland and Wales have gone down different routes and grant authorities the right to assess themselves. Is there anything to be learnt from these divergent approaches?

Aims

We aimed to:

» Identify the differences between the assessment approaches in England, Scotland and Wales and draw out lessons learned.

» Identify the factors that explain the differences in CPA scores between English authorities.

» Determine the extent to which various types of deprivation (in terms of income, health, education etc.) affect CPA scores in different types of authorities (London Boroughs, Counties, Metros etc.).

» Convert findings into practical policy advice suitable for the Audit Commission’s task to develop a new assessment framework for 2008 onwards

What We Did

» To analyse the relationship between deprivation and a local authority’s CPA score we developed a statistical model, using CPA scores from 2002, 2003, and 2004 as the dependent variable. These scores were tested against several explanatory variables, including the 2004 composite index of multiple deprivation (IMD) and the seven constituent domains that make up this composite, as well as resource spending, political control, ethnic mix and age diversity (see Figures 2 and 3).

» To draw out the similarities and differences between the differing performance management regimes in England Scotland and Wales we conducted 20 semi-structured elite interviews with auditors, auditees and other stakeholders in these countries.

Provisional Findings

» In most cases, the greater the deprivation, the lower the CPA score. However, using the 7 individual IMD domains revealed that different types of deprivation had different effects, depending on the type of authority, so a single measure of deprivation (as used in previous studies) could not fully capture the impact of deprivation.

» Using the same indicators to determine multiple deprivation and CPA score created a risk of circularity, and to a certain degree, authorities were able to ‘buy’ better CPA scores by spending beyond the level specified by central government.

» Performance management in Scotland and Wales, marked by partnership, flexibility, self-assessment and the absence of league tables did not allow performance to be compared between authorities.

Figures

Click on the figures to enlarge

mcleanfig1.jpg              mcleanfig2.jpg              mcleanfig3.jpg

Other Project Outputs and Related Webpages

Project page on the ESRC Society Today website.

March 2010: Dans quelle mesure la privation affecte-t-elle le travail des autorités locales anglaises ?,  Revue Internationale des Sciences Administratives, Vol 76 (1)

June 2008: The limits of performance assessments of public bodies: external constraints in English local government Environment and Planning C, Government and Policy, 26(4): 767 – 787

April 2007: The Perils and Pitfalls of Performance Measurement: The CPA Regime for Local Authorities in England, Public Money and Management, Volume 27 Issue 2.

December 2006: Evaluating the Performance of Local Government : A Comparison of the Assessment Regimes in England, Scotland and Wales, Policy Studies, Volume 27 Issue 4.

November 2006: The Limits of Performance Assessments of Public Bodies: The Case of Deprivation as an Environmental Constraint on English Local Government, Nuffield College Working Paper 2006-w16

July 2006: Assessing Public Service Performance in Local Authorities through CPA : A Research Note on Deprivation, National Institute of Economic Review, Number 197.

Research Team

Iain McLean

Iain McLean

Iain McLean is Professor of Politics and Director of the Public Policy Unit, Department of Politics & International Relations, University of Oxford. He frequently gives evidence to parliamentary select committees and is currently a specialist adviser to the Commons Treasury Committee. His recent applied research has concentrated on the spatial distribution of public expenditure around the UK, including the controversial Barnett Formula and the formulas for distributing grant to local authorities and health authorities.

Email: iain.mclean@politics.ox.ac.uk

Dirk Haubrich

Dirk Haubrich

Dirk is Research Officer at the Department of Politics and International Relations, University of Oxford. He specialises on public policy (particularly its delivery, assessment and regulation), but also publishes on issues such as terrorism and international ethics. Prior to his current post he was a Research Fellow at the Department of Philosophy, University College London (2003-05) and an executive with management consultants Accenture (1995-2003).

Email: dirk_haubrich@yahoo.com

Warren Hatter

Warren Hatter

Warren Hatter joined the New Local Government Network (NLGN) in 2002 to develop a research programme and establish the Research Unit. Working on a diverse range of topics and projects linked to the Modernisation and New Localism agendas, the Unit’s work has played a part in the success of NLGN, which was recently named Think Tank of the Year by Prospect magazine.

Email: warren.hatter@nlgn.org.uk

James McGregor

James McGregor

James has worked at New Local Government network (NLGN) since July 2005 and specialises in case-study research on local government-related policy issues. James has previously worked as a researcher and project manager for the public and private sectors, both in London and Washington DC. His publications include ‘Turning Neighbourhoods Around: making a difference with Neighbourhood Renewal Funding’ and ‘International Brownfields Redevelopment’

Email: JMacgregor@nlgn.org.uk

Roxana Gutierrez-Romero

Roxana Gutierrez-Romero

Roxana Gutiérrez-Romero is a Research Officer at the Social Disadvantage Research Centre (SDRC) and at the Centre for the Analysis of South African Social Policy (CASASP). She is also a Teaching Fellow at the Faculty of Economics at the University of Cambridge. Her main research interests are in the fields of poverty, inequality, development economics, and applied econometrics.

Email: roxana.gutierrezromero@socres.ox.ac.uk