Evidence

From the outset What Works Scotland sought to develop knowledge around the systems, processes and partnerships that support the use of evidence in transforming public services for all of Scotland’s communities.

There are currently some major areas of activity that are central to the Scottish Approach to public service reform that are under-researched and poorly evidenced. For example, there is little systematic, independent evaluation of partnership and prevention activity across Scotland. The same is also true of outcome-focussed work and little is known about the long-term effects of taking part, or not taking, part in community engagement.

Research insights

Good public service reform requires a collaborative approach to both learning and research. This collaboration must cut across all levels of the system, from those who use the services through to those who design and commission them. Critically, it must also involve those charged with evaluating their performance.

Evaluation is most useful when it measures outcomes that are relevant to communities and other stakeholders, and it is only by working with them that their views can be incorporated into the design and planning of evaluations. Working with partners, including NHS Health Scotland, we have developed and applied a systematic and collaborative approach to evaluation planning to a range of national and local interventions, including the Community Empowerment Act and Glasgow’s Thriving Places initiative. Evaluability Assessment is increasingly being used by the Scottish Government to plan the evaluations of national policies, as well as being taken by other stakeholders. We are now developing guidance materials to make the approach accessible to the widest possible range of potential users.

There is no ‘one size fits all’ approach to either generating or using evidence. Approaches to using evidence differ depending on the context and purpose for which it is needed, and on the previous experience of staff in using evidence in their work. To meet this, services have to adopt a variety of approaches to the way they collect and display evidence (Developing a evidence service for the children and families workforce – PDF).

Co-producing research with the people whose lives an initiative seeks to change is fundamental to the successful outcomes.This requires space for open discussion, an analysis of the community’s assets and the alternatives on offer.

Whilst focusing on outcomes can be a challenge in evaluation it can also help to cement a partnership. In order for people to be able to actualise change, public services need to actively facilitate meaningful alternatives to peoples’ current circumstances. Focusing on improving outcomes for some members of a community can benefit all members of a community.

Our work developing an evidence service for those who work with children and families demonstrates the requirement for clear communications and the encouragement of open discussion. Getting groups to work together and talk together helped to foster a shared understanding of what evidence type was needed and why, and how best that evidence could be presented (Developing a evidence service for the children and families workforce – PDF). Giving all those involved the opportunity to work collaboratively to identify gaps in knowledge helped partners think through and articulate what they wanted to know and why.

Collaborative action research is an excellent method to support the evaluation of community relevant outcomes and to meet the needs identified above. It provides a space in which policymakers, service providers, and those who use the services can collaborate to develop critical reflective practice. It is a flexible, inclusive approach that enables: participatory and collaborative activity; research, inquiry and reflection; and strategies for action, prevention, culture change and perhaps social change. It can be used to frame improvement tools, desk research, and shared analysis.

We used a collaborative approach to develop a toolkit to evaluate Participatory Budgeting collaboration. This made sure that all involved fully owned, understood, and were able to advocate for the use of the evaluation toolkit they designed. It enables evaluation to be developed that takes account of the culture, context and needs of the service.

Implications for policy and practice

Evaluation takes time, is resource heavy and time consuming. It has to be planned into the development process. This is particularly pertinent at a time of institutional flux and high staff turnover as local authorities restructure departments, services and job roles. This instability impedes the ability of staff and partners to develop and sustain collaborative work.
Adopting an outcome-based approach can help forge collaboration and outline the pathway to policy implementation, bringing together a range of diverse stakeholders.
A good approach must embrace complexity, value the perspective and contributions of multiple stakeholders and capture evidence to support improvement and transformation. For this to be effective the evidence-gathering approach has to be contextualised and bring together data from different sources to develop a broad picture of what is happening.
The use of community impact assessments, equality impact assessments, strategic community assessments or auditing of the processes, and longitudinal studies makes it easier to recognise who benefits, and who does not, from any given initiative.

Successful partnership working, innovation and prevention requires all agencies to focus on being learning organisations. This involves learning from examples of activities which have been tried elsewhere and could be adapted to the local context; embedding a culture of evaluation so there is a robust evidence base providing insights into successful and unsuccessful local initiatives; and embracing innovation as a way of developing and improving approaches to prevention.