By Robin Ashleigh 

David Ben Gurion famously said, “It’s not enough to be up to date; you have to be up to tomorrow.” To be “up to tomorrow” in the not-for-profit sector, a thoughtful evaluation of your work and its impact is absolutely essential.

Prove your impact

In the ever-challenging fundraising market, robust evidence of your impact is crucial. More and more of our supporters are asking us to demonstrate our impact. Whether it’s an individual donor, or a trust or foundation, the trend is the same: philanthropists are becoming more discerning and require convincing that their money will be used effectively, and ultimately make a difference.

Improve your impact

At UJIA, we conduct impact evaluation to help us understand where our programmes have achieved positive outcomes. This is not only about celebrating success. What really matters is the ability to understand what works and replicate that success.

In the real world, not every project succeeds every time. A thorough evaluation should improve your understanding of the programme, and the resulting insight should lead to recommendations to increase the chance of positive outcomes next time around.

Build an impact-focused culture

Before you begin evaluating your work, it’s important to articulate what you do and why. What are you trying to change for your beneficiaries and the wider society? How will your work make those outcomes happen? Working through these questions will help you develop a Theory of Change.

This is a crucial first step in understanding your impact and to do it properly, you need to consult your staff, volunteers and partners. They are the experts who know your organisation and your community inside out.

Developing your Theory of Change is a great way to drive an impact-focused culture within your organisation, and gets people thinking and talking about your intended outcomes. At UJIA, we found that “Impact Model” was a clearer name, which was easier to understand. Our Impact Model provides an up-to-date narrative about how we achieve our mission and vision. Therefore, it is the starting point for planning our programmes, and the basis for our impact evaluation, which should test the assumptions or beliefs that we have now documented.

Conduct a ‘Pro’ Evaluation

The chances are your organisation already collects some data. Maybe you report on the number of activities or resources you provide, or the number of people who attend your programmes. These are known as your outputs and are an important part of the story. However, reporting on outputs alone does not give the whole picture. To really understand your work, you need to investigate your outcomes: “the changes, benefits, learning or other effects that happen as a result of a project or organisation’s work.” See: https://knowhow.ncvo.org.uk

If you want to collect meaningful evidence to evaluate your programmes, you’ll be in a pretty healthy position if you focus on ‘the three Pros’: Profile, Process and Progress.

Understand the profile of your participants

Start by asking some simple questions about who attends your programme or uses your services. How old are they? What is their gender? Where do they come from? Do they identify with a particular denomination or stream of Judaism? Summarising the demographic profiles of your participants will help you understand if you are engaging your target population.

As well as the basics, you can use your registration forms and pre-programme surveys to find out about your participants’ prior experiences. In our case, it is often helpful to know whether participants went to Jewish schools, whether they have been part of a Youth Movement, and how many times they have visited Israel.

All of this ‘profile’ data is useful in its own right, but it also allows you to divide your sample into groups and compare their experiences. This becomes possible if your participants also provide feedback about ‘process’ and ‘progress.’ For instance, at UJIA, we found that one of our seminars was given consistently lower satisfaction ratings by those from more orthodox backgrounds. In response, we made changes to the programme for the following year and were able to improve the experience for participants of this profile.

Understand the process

Once the programme has concluded (or throughout the programme if yours is a long-term initiative), ask your participants about the ‘process.’ Are your participants satisfied with the overall programme? Were the activities enjoyable, interesting or relevant? Were the staff or leaders knowledgeable or inspiring?

When developing your Theory of Change, you should have identified the key components of your programme, so use this opportunity to ask the participants if they are happy with all that you provided, focusing on those activities that you deemed important in driving your desired outcomes. If there are other important stakeholders besides the direct users, you could also seek their perspectives, and maybe there are more objective measures of quality that you can use to further enhance your understanding.

Evidence about the ‘process’ helps you understand if you met your delivery objectives, for which you are accountable. This information is important as it can show you which areas of your project require improvement. When this is coupled with evidence on your participants’ ‘progress’, you will be able to look for any relationships between your activities and your outcomes. Which parts of your delivery appear to lead to positive outcomes, and how is this different depending on the ‘profile’ of your participants?

Understand your participants’ progress

You won’t really have any meaningful evidence until you attempt to understand your participants’ ‘progress.’ How have your participants changed as a result of the programme? What is different about their knowledge, attitudes and behaviours since they took part in your activity? Self-assessments provided by participants can be helpful in understanding a project’s outcomes, especially if they are completed before and after the intervention. It is also worth following up a few months or years down the line to see if any benefits have been maintained in the longer term.

Informal Jewish education has a great tradition of encouraging self-reflection, so why not get added value from participants’ self-assessments by using them in coaching or mentoring conversations? This is something that has proved very effective at UJIA, especially in our leadership programmes where we have asked participants to consider their own strengths and areas for development.

It can also be worthwhile to seek the perception of others who know the participants well. For example, you might want to supplement the self-assessments of young people with the views of their parents, mentors or teachers. You could ask similar questions to those which the participants answered, or you could tease out more nuanced themes using semi-structured interviews.

What are you waiting for? Start collecting meaningful evidence today

Impact evaluation takes time and effort, and it can be tricky to uphold meticulous research practices due to the logistical constraints within your organisation. It’s all about being realistic, applying rigour in the right places and remaining considerate of your colleagues’ and participants’ time. Using ‘the three Pros’ is a simple yet effective way to get started and should help you collect meaningful, high quality evidence. So, what are you waiting for? Prove and improve your impact with ‘the three Pros.’

Robin Ashleigh is Head of Impact Evaluation at UJIA, the UK’s leading Israel charity, where he is leading a comprehensive programme to define and measure the organisation’s outcomes and impact.