How What Works Scotland and Glasgow community planning partners worked together to develop an evaluation approach to assess the impact of participatory budgeting activities.
Using a collaborative action research (CAR) model, the group devised an evaluation toolkit for participatory budgeting (PB) activities in Glasgow.
As part of our collaborative work with Glasgow Community Planning Partnership (CPP), in 2015 What Works Scotland supported a group of professionals in Glasgow with establishing a Participatory Budgeting Evaluation Group.
The group included members from various community planning partners and was led by Evelyn O’Donnell from Glasgow City Council and Alex Byers from Glasgow Life. Other group members included representatives from the NHS, the Third Sector Forum and Foundation Scotland.
What Works Scotland researchers Richard Brunner (University of Glasgow) and Oliver Escobar (University of Edinburgh) supported and facilitated the group.
The collaborative action research process
The CAR group met 15 times from mid-2015 to December 2016, working out together – through dialogue and evidence-gathering, – what was important for them to include in a pilot PB evaluation toolkit for Glasgow.
During those 18 months, the group undertook these activities:
- Heard evidence from Glasgow Libraries about their PB evaluation process for children’s books.
- Reviewed and discussed national and international PB evaluation literature, including:
- 15 Key Metrics for Evaluating Participatory Budgeting: A Toolkit for Evaluators and Implementers published by Public Agenda, the Participatory Budgeting Project and the North American Participatory Budgeting Research Board
- Participatory Budgeting in Scotland: An overview of strategic design choices and principles for effective delivery by Chris Harkins and Oliver Escobar.
- Analysed a variety of PB pilots in Glasgow.
- Considered PB evaluation frameworks emerging in the rest of Scotland.
- In collaboration with Fife CPP, visited the Paris PB team to learn about the technical and political dimensions of how they evaluate their Europe-leading PB processes. Evelyn O’Donnell, Fife CPP and What Works Scotland reflected on the learning in a set of public blog posts.
- Devised, drafted, and consulted on the content of the pilot PB evaluation toolkit.
- Learned how the toolkit needed to take a ‘pick and mix’ approach, able to be used by small PB projects in Glasgow for evaluating a small number of dimensions, and also be applied as a ‘full fat’ version by large PB initiatives. It needed to be useful for the community and the third sector as well as for the statutory sector.
- In collaboration with What Works Scotland, finalised the design of the PB toolkit for use as a working pilot.
- Took ownership for updating the pilot toolkit in future, based on ongoing learning.
The group drew together professionals in Glasgow from different service areas and the third sector to develop the pilot toolkit. The collaborative action research process offered those involved the space to operate as a new team, and to learn together and from each other.
As members were involved in real-life PB projects through the lifecourse of the group, they were able to draw on this ‘live’ PB knowledge and feed that into discussions.
The group drew on international, national and local evidence in the process. Some members were able to travel internationally to Paris and bring that learning into the group, as well as their learning from collaborating on the visit with Fife CPP.
Advantages and challenges of the CAR process
Through this process the group generated their own bespoke PB evaluation toolkit, rather than buying something ‘off the peg’. The process of doing this sought to enable group participants to be able to fully own, understand, and advocate for the use of the Glasgow PB evaluation toolkit, based on evidence, and for that toolkit to specifically apply to the culture, context and needs of Glasgow.
Challenges for the CAR process were maintaining involvement by the whole group throughout. Group members all had different priorities, and although the group met at different times and venues, core group involvement was by three or four professionals.
This reflects the challenge that What Works Scotland has found in other CAR groups in the four areas of Scotland in which we have focused our CAR work. Collaborative action research and co-producing outputs and outcomes can be done, but sustaining group member involvement throughout is hard to do.
“The baton is being passed to Glasgow CPP to take forward this work, now that the pioneering groundwork has been done by members of the PB collaborative action research group. As the evaluation toolkit is piloted in PB projects large and small, those co-ordinating PB processes across the city will learn from how the toolkit is applied in practice. The pilot toolkit will likely change over time in response: it is a live and dynamic document.”
Richard Brunner, Research Associate, What Works Scotland
