Evaluating research brokers and intermediaries: What is success? And how do we measure it?

Identifying outcomes and impact, monitoring and information of research brokering and intermediation

Led by: Anna Downie , Strategic Learning Initiative, IDS

Anna Downie began the session by summarising what funders want – “to know how we’re reducing world poverty through our websites”. When it’s put like that, it’s clear what a task that actually is.

In a round of introductions, participants shared why they were particularly interested in M & E and the challenges they face. Challenges ranged from wanting to develop indicators and frameworks to wanting to justify your existence and evaluate without doing yourself out of a job! One thing everyone had in common was wanting to know what influence they had.

Evaluation is tough. Some of the particular challenges in M& E that intermediaries face are that:

  • They are trying to evaluate a wide range of research
  • They are trying to change use of information and affect a policy environment full of complex processes
  • There is no ‘one size fits all’ framework of indicators for intermediaries
  • Intermediaries have different concepts of outcomes – is it access, use, behaviour change, attitudes or impact?

Having tried to evaluate the impact of information in a previous role myself, I was interested in a technique used in outcome mapping methodology where you’re asked what you expect to see, hope to see and would love to see. This helps to hone in on minimum and hope for the maximum. I think I used to focus on what I’d love to see which meant my expectations were perhaps a little too high. I guess the key is to be realistic.

Web statistics tend to give more questions then answers – who are the people using the information and why are they using it? Going beyond figures of people hitting your website to assessing impact is difficult. As Andrew Chetley (Healthlink) pointed out long term impact often happens downstream by which time you’ve moved on to other things. Once the moment has passed, how do you still claim these changes that have occurred? Andrew shared some of the lessons learnt from Healthlink

  • It’s important to have a baseline survey and check what the situation is before – this can help you to track changes later on
  • Make sure you document every change you see, it may in future create a rich picture of understanding change. The ‘Most Significant Change’ technique is a useful complimentary process which gets people to share stories and debate

The IDS Knowledge Services are now trying to make more explicit their pathways to change and be clearer about the steps in between a website and development outcomes. See Anna’s powerpoint presentation for more details of IDS’ Theory of Change.

Participants were given a chance to discuss some key questions: What challenges do we all face? What sorts of outcomes, indicators, frameworks and methods are relevant?  We didn’t quite stick to these questions and some more challenges emerged which tends to happen in just about any M&E discussion! Here are some key comments from particpants:

  • The ‘why’ question is fundamental – what is your reason for building a repository?
  • It is difficult to measure influence and impact if you haven’t defined the term ‘pro poor’. There’s an underlying assumption that people living in poverty have a quieter voice and that we’re at a great distance. It’s scary that we are still having this debate. Maybe communities should be participants in evaluation process
  • Donors often pull us in different directions which is another silence not addressed. Maybe we should build capacity of donors on difficulties to measure impact?
  • We need a greater awareness of the environment in order to evaluate
  • Challenges lie in cultural differences; people are often reluctant to express things that are acceptable in other cultures. People also often hide drawbacks and concentrate on success
  • A multiplicity of methods is useful as we are dealing with complexity. No single framework will do it, so we need to triangulate or even octangulate!
  • If research has had any impact should we have seen some changes on poverty indicators by now? Have we been measuring the wrong things, should we have been measuring at a more local level? Maybe looking for change is too difficult, maybe intermediaries are just about providing access to information for people if they want to use it.

Thoughts, comments please!

Yaso Kunaratnam, IDS

Related links: Evaluating research brokers and intermediaries presentation
M & E resources for intermediaries

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: