Скачать 434.56 Kb.
The independent review of the Best Value Audit process
Final report to the Accounts Commission and Audit Scotland
Clive Grace, Sandra Nutley, James Downe and Steve Martin
Over the last two years, some two thirds of local authorities in Scotland have been reviewed through the process of Best Value audits. Some of these audits have reported good or very good performance against the statutory Best Value criteria; others have been less favourable. Some councils have welcomed the auditors’ findings, but others have contested them vigorously. For all concerned - in Audit Scotland, which undertakes the audits, as well as in the councils themselves - the Best Value audit process has been a demanding and challenging one, and has represented a significant investment of resource in terms of staff and councillors’ time, as well as money.
In the tradition of ‘physician heal thyself’, it is now appropriate for the Best Value audit approach to be subject to independent scrutiny and review in order to evaluate its strengths and weaknesses to date, and make recommendations about its future course and development. This is the purpose of this report. The study was commissioned as an independent evaluation from the Universities of Cardiff and Edinburgh. It was led by Dr Clive Grace and Professor Sandra Nutley, with Dr James Downe and Professor Steve Martin completing the research team.
It has been a fascinating and rewarding study to undertake, and as a research team we have learned a lot ourselves, as well as hopefully contributing to knowledge and understanding about the Best Value audit approach. We would like to thank all those who gave so generously of their time and views in interviews and by responding to the survey we carried out. Special thanks go to Carol Calder, our key contact at Audit Scotland, who was immensely helpful throughout the work.
The overwhelming impression from all the material we collected was of a local government ‘family’ who were all strongly committed to improving local authority services and community leadership - most importantly the councils themselves, but also COSLA, the Improvement Service for Scotland and SOLACE, as well as Audit Scotland, the Scottish Executive, and other stakeholders. This is undoubtedly a decisive moment for local public services in Scotland with the changes ushered in by the new electoral arrangements, with the arrival of a new administration in the Scottish Executive, with local authority performance indicators being revised, and with the Crerar Review of scrutiny due to report in the near future. We hope that our report will help to contribute to this momentum for change and strengthen the shared endeavours of all stakeholders to secure improved services and governance.
Acknowledgements and Authors 4
Summary and Recommendations 5
1.0 Introduction 10
Annex 1 The Research Team 60
Annex 2 Interview and case study topic guide 63
Annex 3 Survey questions 69
Annex 4 Bibliography 72
We acknowledge with gratitude the assistance of the many national stakeholders and officers and councillors in local authorities who agreed to be interviewed in the course of the study. We are also grateful to the officers who completed our survey, and we want to place on record our appreciation of the invaluable help and guidance that we have received from Audit Scotland.
The views expressed are however our own and do not necessarily reflect those of Audit Scotland, the Accounts Commission or the case study councils.
Clive Grace is Honorary Research Fellow in the Centre for Local & Regional Government Research, Cardiff University
Sandra Nutley is Professor of Public Management at the University of Edinburgh and Director of the Research Unit for Research Utilisation
James Downe is Senior Research Associate in the Centre for Local & Regional Government Research, Cardiff Business School
Steve Martin is Professor of Public Policy and Management and Director of the Centre for Local & Regional Government Research at Cardiff University
Summary and Recommendations
S1. This independent review of the Best Value audit (BVA) process was commissioned by the Accounts Commission as part of its wider review of the progress of the audits of Best Value and Community Planning. The study was undertaken by a team from the Centre for Local & Regional Government Research at Cardiff Business School and the School of Management at Edinburgh University. The overall objectives of the independent review were to:
S2. The approach we took was to consider six main areas of investigation:
We undertook our investigations between mid December 2006 and the end of March 2007.
S3. Best Value and Community Planning were introduced as statutory duties for local councils by the Local Government in Scotland Act 2003 which requires them to:
S4. In developing its BVA approach, the Accounts Commission (and Audit Scotland) drew on its prior experience of the Local Government Management Arrangements audits. The key principles which underpinned the resulting BVA process were:
S5. Key features of the way in which BVAs are currently undertaken include audit by a central specialist team and a cyclical audit. The first Best Value audit report was published in 2004. Reports on 18 of the 32 councils had been published by 31 March 2007.
S6. Audits have taken place against the backdrop of a number of very important wider policy developments including the Public Service Reform agenda. The Scottish Executive's Public Services Reform document sets out an approach which seeks to drive up quality and encourage innovation and to ensure that services are increasingly user focused and personalised.
S7. Our overall assessment is that the BVA approach is now established, has gained credibility, and in broad terms has been effective. The best test of this is that even those who have significant criticisms of the BVA approach almost invariably argue that it needs to be fine tuned rather than abandoned. It is also warranted by comparison with the broadly equivalent methodologies in Wales and in England as reflected in the WPI and CPA.
S8. There are some strong similarities between the BVA approach and both of these regimes. BVA, the WPI and CPA draw on the same underlying rationale i.e. the assumption that continuous improvement in a council’s functions and services is only possible over the medium term if the corporate processes are working well. This model of improvement has particular relevance as councils are increasingly required to address complex ‘wicked’ issues which cross traditional service and organisational boundaries.
S9. Quality of BVA reports - There has been an overall improvement in the quality of the reports as the BVA process has developed and matured. We compared published BVA reports with what we took to be ‘best practice’ examples of CPA reports (WPI reports are not public documents) and found no fundamental differences in quality.
S10. Professionalism of audit teams - One of the most important aspects of the BVA approach has been the degree of expertise and professionalism of the audit teams which have carried out the work. There is convincing evidence from a number of sources that the perceived credibility of teams has varied. Many of the weaknesses appear to us to have been in the early phases of the development of BVA. It is plain from Audit Scotland's own evidence to us that they have recognised the issue and have invested, and continue to invest, considerable efforts to strengthen the Best Value audit teams across all of these dimensions. The changes to the Best Value audit teams are fundamental to the success and development of BVA and the contribution it is capable of making.
S11. Clarity of method - Because the audit teams’ theory of improvement has not been made explicit, there has been a lack of clarity about the criteria that they are using to judge councils. This is an important area for attention because it fuels concern about the transparency of the process and the issues on which judgments are based.
S12. In our view a future round of the BVAs will therefore need to be based on a more explicit understanding of how improvement happens and will need to ensure that the:
S13. Consistency - The issue of perceived inconsistency arises in a number of forms. The first is that councils that have gone through the process later have experienced it as more demanding and challenging than those which went through it earlier. The second ‘inconsistency’ relates to judgements made between councils. The third perceived ‘inconsistency’ that we have found is between the BVA report and other audit reports which a council has received previously. The fourth ‘inconsistency’ is that some councils report that there has been a significant difference between the verbal feedback given to them by audit teams at the end of their work on-site and the written report which has followed (the latter is seen as having been more critical) and then in some cases a further gap between the judgements made by Audit Scotland, and those reached by the Accounts Commission.
S14. The issue of ‘impact’ - It is clear that BVAs have had significant positive impacts on councils, but that their effects have been uneven. Nearly all councils have benefited from the process of self-assessment and the conscious and systematic review of their own processes and performance that it has entailed.
S15. There is no evidence that BVAs have had a direct impact on service users. Given the nature of the process and the way in which it has been conducted, this is to be expected, but this needs to be made clearer to councils and others. The impact on councils’ partners has also been negligible because to date BVAs have focused on councils’ corporate capacity rather than on community planning. But this is clearly an area that demands more attention in the future.
S16. Despite the improvements which have been made as the BVA process has developed the evidence that we have gathered points to a number of issues which still need to be tackled in addition to those that we have highlighted above. These include the strong managerial focus of BVA reports; relatively little attention to community planning; insufficient linkage between the Best Value audit work and the work of other inspectorates, and the annual audit process; the phrase Best Value Audit has caused confusion; limitations in the use and interpretation of performance data and of the statutory performance indicators and in the extent to which the BVA reports deal with questions of probity and propriety and with the ‘joining up’ agenda; and, BVA reports have limited value in terms of public reporting.
S17. The way forward - Our recommendations are informed by ‘two faces of improvement’:
S18. Our recommendations address three key areas:
S19. Developing the purpose and method of BVAs - In our view, the future development of the purpose and method of BVAs needs to:
S20. The BVA approach is now sufficiently well established and well regarded that the first round of audits can be seen as providing credible baselines against which to measure future progress by councils. There are a number of ways in which we believe that the Accounts Commission and Audit Scotland need to clarify the purpose, scope and methods of BVAs. In our view the credibility and capacity of the BVA team needs to be developed as a matter of priority. This should involve putting in place a Development Plan for the Team as a whole and supplementing the existing team with senior ‘associate’ peers drawn from right across the UK. There is ample evidence from councils and other evidence that BVAs and other forms of audit and inspection are not currently sufficiently co-ordinated. There are a number of ways in which this issue could be addressed including improved co-ordination of the timetabling of inspections and better sharing of information between inspectorates; the introduction of relationship managers or a lead agency to be the first point of contact with councils on behalf of all inspections; and the merger of some or all of the inspectorates that oversee councils. There is a need in future to Improve the links between BVAs and the annual audit process, and improve the links between BVAs and other inspections. In our view there are a number of areas in which reporting of BVAs might be improved including removing the anomaly around publication, increasing public involvement in BVAs, improving reporting to the public, and a more active engagement of the media.
S21. Developing and adapting BVAs in the light of changes taking place in councils - In our view the future development of BVAs should involve:
S22. Developing BVAs in the light of developments in the overall strategy for reform and regulation of public services - There are at least four important developments that we believe could help to shape the context within which future BVAs are implemented:
Section 1: Introduction
1. This independent review of the Best Value audit (BVA) process was commissioned by the Accounts Commission as part of its wider review of the progress of the audits of Best Value and Community Planning. The study was undertaken by a team from the Centre for Local & Regional Government Research at Cardiff Business School and the School of Management at Edinburgh University. It is intended to complement the public consultation which has been undertaken by the Accounts Commission and which involved submissions from a wide range of interested parties. The consultation and the independent review between them will help to inform the Commission’s consideration of how Best Value audits should develop in the future.
2. The Commission considered that the time was right to review the BVA approach and any key changes that might be needed in the second round of audits once all 32 councils have completed the first round. A key objective for the Commission was to move on from current systems and processes to achieve a more rounded assessment of service delivery and outcomes which takes more account of the views and experience of citizens and service users.
3. The overall objectives of the independent review were to:
4. In reaching our conclusions we were asked by the Commission to take account of the:
5. Experience from BVAs to date had suggested that the methodology might need to become more flexible, risk-based and proportionate, in order to recognise the differing levels of progress being made by councils, and their capacity to sustain continuous improvement. Other issues that we were particularly asked to consider included: the perception that there were differences between the Commission’s findings and the conclusions contained in the Controller of Audit’s reports; whether or not to introduce some form of scoring or rating of councils; the potential use of peers in the audit process; and ways of enabling more opportunity for the views of service users to be reflected in the audit process.
6. We were asked to address a number of key areas for inquiry but were not required to confine our analysis only to these questions if other important issues presented themselves during the course of our work or if our own experience and analysis pointed to other questions. In the event, during the course of the review we introduced a number of different ways of thinking about the BVA process, and these are reflected in our recommendations.
7. The review was based on in-depth interviews with key national stakeholders and senior officers and councillors in a range of different councils which had contrasting experiences of BVAs. As academic researchers we were able to bring independence to the study combined with substantial experience of policy relevant research and an in-depth knowledge of inspection of local government across the UK. The Commission and Audit Scotland made it very clear that there was a genuine willingness on their part to hear contrary, challenging and uncomfortable views if the evidence called for that.
8. The approach we took was to consider six main areas of investigation:
9. We addressed these questions by gathering evidence from eight main sources:
We undertook our investigations between mid December 2006 and the end of March 2007.
Section 2: Background and Policy Context
10. Best Value and Community Planning were introduced as statutory duties for local councils by the Local Government in Scotland Act 2003 which requires them to:
11. Prior to the 2003 Act, the duty of Best Value existed on a voluntary basis, having been introduced in 1999/2000 as a quid pro quo for a moratorium on compulsory competitive tendering. The 2003 Act and the supporting statutory guidance and advisory notes1 built on the practices that had been developed since 2000, and significantly extended the powers of the Accounts Commission and the Controller of Audit's reporting duties to include the duty of Best Value and Community Planning.
12. In developing its BVA approach, the Accounts Commission (and Audit Scotland) drew on its prior experience of the Local Government Management Arrangements audits. It also sought to learn from the experience of the equivalent systems in England and Wales (Comprehensive Performance Assessment (CPA) and the Wales Programme for Improvement (WPI) respectively). It was keen to ensure that the Best Value audits provided an overall assessment of an authority’s corporate capacity but, at the same time, it was keen to avoid a ‘rules-based assessment’ or one that ended up with an overall label or score for performance. This stance was primarily one of principle but there was also a pragmatic recognition that the CPA approach could not have been transferred to Scotland at the time, even if it had been deemed desirable, because of the lack of requisite performance data.
13. The key principles which underpinned the resulting BVA process were:
14. Key features of the way in which BVAs are currently undertaken include:
15. Box 1 provides an overview of the key features of current BVAs.