Home » Research Projects » Correlates of Success in Performance Assessment
A fair measure of success?
How performance assessment of public bodies may be falling short
Over the last 15 years British governments have searched for new ways of assessing how well bodies such as local authorities and health authorities are working. As a result, local authorities in England are now judged by way of comprehensive performance assessment (CPA) and health authorities receive a Star Rating score between zero (poorly performing) and three stars (high performing). Proponents of CPA and Star Ratings believe they lead to improvements in services. Critics consider them unreliable and cite the famous dictum of Charles Goodhart (the former chief economist of the Bank of England) that “when a measure becomes a target, it ceases to be a good measure”. This project will examine whether performance assessment succeeds in making local authorities more efficient and effective. It will identify the factors that lead a local authority to gain a ‘good’ or ‘bad’ score and investigate whether the CPA in particular is a fair, reliable and valid measure of success.
What the research means for policy makers and the wider community
- Researchers will investigate whether public bodies in desirable areas of the country perform better than those in poorer areas. If so, rewarding good performance will tend to grant autonomy disproportionately to rich places which will also be rewarded by ‘lighter touch’ (and less onerous) inspection. Is this fair? Or does it create a tension between any government’s aim to make all public bodies as efficient as possible and its aim to be equitable? (i.e. to treat all comparable citizens equally, wherever they live).
- The study will explore whether performance scores are reliable, valid (i.e. measure what they purport to measure) and lead to more efficient services, or are a poor use of taxpayers’ money. Is Goodhart’s view that ‘when a measure becomes a target, it ceases to be a good measure’ a very basic but insurmountable problem?
- This study will identify possible areas for improvement in the way public bodies are assessed.
Research methods
The project will assess the performance of British local authorities according to population size, indices of deprivation and type of local authority. Researchers will interview a selected sample of performance auditors and auditees from local authorities in England, Scotland and Wales. Researchers will also interview policy makers who have been responsible for assessment policies. Throughout the project, researchers will involve interested parties through this project’s partner organisation, the New Local Government Network (NLGN).
Further Information: Project Posters
Updated Project Poster 2009
Below is a summary of this project’s provisional findings. It was originally presented as a dissemination poster, which is available here as a pdf document. All figures can be found at the bottom of this poster summary as thumbnails, which one should click to view full-size images. Alternatively, where figures are reffered to in the text, click the linked text for a full-size version.
DOES POVERTY EQUAL POOR PERFORMANCE?
Background
Comparing the performance of local authorities with differing demographics is problematic, and different governments approach the problem in different ways. In England, central government assesses the quality of local authorities through Comprehensive Performance Assessments (CPAs), as summarised in Figure 1. Since it first appeared in 2002, CPA has remained controversial for its uniform and prescriptive approach and its disregard of local constraints such as deprivation that are not easily influenced by authorities. Is the CPA a valid measure of local performance, or does it merely reflect local deprivation and other external constraints? Scotland and Wales have gone down different routes and grant authorities the right to assess themselves. Is there anything to be learnt from these divergent approaches?
Aims
We aimed to:
» Identify the differences between the assessment approaches in England, Scotland and Wales and draw out lessons learned.
» Identify the factors that explain the differences in CPA scores between English authorities.
» Determine the extent to which various types of deprivation (in terms of income, health, education etc.) affect CPA scores in different types of authorities (London Boroughs, Counties, Metros etc.).
» Convert findings into practical policy advice suitable for the Audit Commission’s task to develop a new assessment framework for 2008 onwards
What We Did
» To analyse the relationship between deprivation and a local authority’s CPA score we developed a statistical model, using CPA scores from 2002, 2003, and 2004 as the dependent variable. These scores were tested against several explanatory variables, including the 2004 composite index of multiple deprivation (IMD) and the seven constituent domains that make up this composite, as well as resource spending, political control, ethnic mix and age diversity (see Figures 2 and 3).
» To draw out the similarities and differences between the differing performance management regimes in England Scotland and Wales we conducted 20 semi-structured elite interviews with auditors, auditees and other stakeholders in these countries.
Provisional Findings
» In most cases, the greater the deprivation, the lower the CPA score. However, using the 7 individual IMD domains revealed that different types of deprivation had different effects, depending on the type of authority, so a single measure of deprivation (as used in previous studies) could not fully capture the impact of deprivation.
» Using the same indicators to determine multiple deprivation and CPA score created a risk of circularity, and to a certain degree, authorities were able to ‘buy’ better CPA scores by spending beyond the level specified by central government.
» Performance management in Scotland and Wales, marked by partnership, flexibility, self-assessment and the absence of league tables did not allow performance to be compared between authorities.
Figures
Click on the figures to enlarge
Other Project Outputs and Related Webpages
Project page on the ESRC Society Today website.
March 2010: Dans quelle mesure la privation affecte-t-elle le travail des autorités locales anglaises ?, Revue Internationale des Sciences Administratives, Vol 76 (1)
June 2008: The limits of performance assessments of public bodies: external constraints in English local government Environment and Planning C, Government and Policy, 26(4): 767 – 787
April 2007: The Perils and Pitfalls of Performance Measurement: The CPA Regime for Local Authorities in England, Public Money and Management, Volume 27 Issue 2.
December 2006: Evaluating the Performance of Local Government : A Comparison of the Assessment Regimes in England, Scotland and Wales, Policy Studies, Volume 27 Issue 4.
November 2006: The Limits of Performance Assessments of Public Bodies: The Case of Deprivation as an Environmental Constraint on English Local Government, Nuffield College Working Paper 2006-w16
July 2006: Assessing Public Service Performance in Local Authorities through CPA : A Research Note on Deprivation, National Institute of Economic Review, Number 197.
