At the Old Fire Station (one of the organisations that founded the Marmalade core group), we spent many years struggling with evaluation. We found that conventional form filling, box-ticking and number counting didn’t always help us to learn and understand the impact of our work.
In 2017 we started using ‘Storytelling’ and it offered something different. it didn’t presuppose outcomes, but let people tell us what change and impact meant for them in their own voices and words. It was a meaningful, collaborative and enjoyable process, and it helped us to actually listen and learn.
In the wake of the pandemic, as much of our day-to-day practice as an art centre was put on hold, we decided to run a free webinar on the Storytelling methodology. We’d never run a webinar before, let alone one on evaluation, and we expected only a handful of people to come along. You can imagine our surprise when over 200 people from across the world appeared on our screens!
In April 2022, it happened again. When we programmed a session at Marmalade on ‘Meaningful Measurement’ we were joined by over 80 people from local authorities, cultural partners, health and wellbeing organisations and more.
Over the past 2 years, we’ve had conversations with people working across housing, children and young people’s services, health and wellbeing, arts and culture and beyond. And we’ve been astounded by the breadth of people all grappling with the same question: How do we change the way we evaluate?
Common evaluation challenges we discovered
Whilst the people we are speaking to are varied and broad in their focus, they often describe common frustrations with conventional approaches to evaluation:
• What we measure isn’t always useful and doesn’t necessarily help us learn.
• Evaluation can be time-intensive and resource-heavy
• It can get in the way of building relationships.
• It can feel extractive and meaningless for those asked to take part.
• Data and form filling alone can paint an incomplete picture which doesn’t allow for the complexity of people, their lives, and the wider systems we exist within.
• It can feel like an add-on that gets in the way of work.
Given these experiences of evaluation are relatively commonplace and widespread, it begs the question - why don’t we just do things differently? But as conversations at the Marmalade Meaningful Measurement event in April 2022 suggested, changing the way we evaluate can feel both complex and multi-faceted. It’s often shaped by governing and funder reporting requirements, expected to deliver on both accountability and learning and is often under-resourced.
However, changing the way we evaluate has the potential for a wider impact. What we decide to evaluate – the KPIs and indicators set – ultimately determine what we prioritise and where we place our focus. And the questions we ask, who asks them and how, dictate whose voice shapes the narrative around impact, change and significance.
So, if we can change the way we evaluate, we can edge towards shifting power and finding more human-friendly ways of working.
The Meaningful Measurement action inquiry
Building on the growing momentum around evaluation and learning across the city, In April 2022 the Marmalade Core group launched an Action Inquiry on Meaningful Measurement.
The inquiry focuses on bringing together partners from across the city to learn and experiment together with more meaningful approaches to measurement, with the intention of using this learning to influence the system locally.
The Inquiry will focus on experimenting with different approaches towards evaluation, and shared learning around what works (and doesn’t), and what’s needed in order for these ways of working to be embedded and sustained.
Exploring Storytelling and Signal
At the moment we are doing two pieces of work to explore different ways of measuring impact; Storytelling and Signal. Storytelling is a rigorous method of capturing qualitative 'stories' which describe the most significant changes experienced by people involved in different ways in a project or service. The rich data collected is user-led and owned and can help us to understand both what works and why. Signal is a tool developed in Paraguay to measure household need, using parameters identified by communities themselves. It is a tool that helps families to own their own needs, organise and prioritise these and allows aggregate data to be collected for communities to mobilise themselves around those needs.
What's Coming Up and How to Join In!
Over the coming year (until Marmalade 2023), we will be continuing to test and experiment with these tools with small groups of people and organisations interested in meaningful measurement.
From September we will begin to bring together the data and learning from both projects, culminating in two Learning Days in September. These will help us to present, unpick and contextualise the data we have, and start to consider, together with communities and services, how we can use this information to drive progress and meet the arising needs in our communities.
In November, we also plan to bring together funders and governing bodies to share the learning from these sessions and discuss how we can support more meaningful approaches to measurement going forward. We hope to start doing this and to share progress at our next Marmalade!
If you’d like to find out more about this and how to take part, drop us a line - we’d love to hear from you!
