Essential guide to evaluating peer work

Why do we need to focus on evaluating peer work?

Current evaluation methods are falling short for peer responses.

Most peer workers can talk at length about the positive and proactive influence that their work has. They know their work has positive effects. But it is often hard to back up that knowledge with robust and accessible evidence.

This comes down to the strength of a peer response’s evaluation processes. Unfortunately, most evaluations of peer responses do not:

• Effectively capture everything they do
• Show the full value and impact of their work

This is because traditional evaluation methods do not work well for evaluating peer work.

Why do current evaluation methods fail peer responses?

Inappropriate evaluation frameworks

The evaluation frameworks used are better suited to non-peer work than peer work.

Both peer and non-peer responses contribute to the goal of improving community health. They also provide some similar kinds of supports and services (such as health education, clinical testing, and providing access to resources for harm reduction).

Because of these similarities, the frameworks used to guide evaluations of peer work often:

• Were originally developed to evaluate non-peer work
• Measure direct and short-term to medium-term impacts of a program on its individual participants
• Result in the use of generic service-delivery indicators

However, there are important differences between peer and non-peer work that these generic service-delivery indicators do not (and cannot) account for.

Peer responses have important impacts on their community’s health that are not the direct result of service delivery

Peer responses have an important role in enhancing both individual and community empowerment and positive sense of self. However, these impacts are neither immediate, nor can they be attributed to single programs or interventions. Rather, these impacts are the indirect, collective result of sustained, high-quality work over time within communities.

Peer responses also play an important role in providing policy and other advice to the health sector and policy environment. When peer responses have strong relationships within the health sector and policy environment, this advice can lead to changes within the sector that improve community health outcomes.

To effectively evaluate peer work, evaluation frameworks need to somehow also account for these broader impacts.

Interactions between peer workers and their communities in their personal lives are relevant and vital inputs

Peer responses often know what they need to do because of the knowledge and insights their peer staff pick up through interactions in their personal lives (rather than through direct service delivery). However, these interactions are not strictly part of the work being evaluated. Hence, traditional evaluation processes ignore them as ‘confounding’ factors.

This results in evaluations that do not accurately reflect how peer responses:

• Create positive impact
• Adapt the way they work in response to changes in their community or the health system and policy environment
• Maintain effectiveness and relevance in a changing environment

Engagement with communities is not just a process, but also an impact

Engagement with communities gives peer response the knowledge needed to act properly (making engagement an input). However, the actions of high-quality peer work often also result in strengthened community engagement (making strong engagement an impact). Traditional linear evaluation frameworks do not capture these kinds of cyclic relationships.

Overemphasis on funder priorities

Evaluations often focus too strongly on funder priorities, and not strongly enough on those of the peer response itself.

First and foremost, evaluation should help the peer response with its own goals. When done well, evaluation should tell rich, exciting, and persuasive stories about all the innovative and positive impacts that peer responses have. It should also guide peer responses to ways they can continually improve to make their work even more responsive and targeted. This is the true value of evaluation.

Evaluations are also useful for providing feedback to external funders and donors. Of course, this is important, but it should be secondary.

Despite this, reporting to funders is often the main (if not the only) focus of a peer response’s evaluation processes.

Community organisations often receive multiple streams of funding from diverse funders. These different funders are usually only interested in supporting particular aspects (or outcomes) of a peer response. They tend not to value or understand the peer response’s overall impact. Rather, they information about the direct, short-term to medium-term impacts of the program (or programs) they are funding.

By focusing on the direct and short-term to medium-term impacts of a program on its individual participants, evaluations of peer responses miss a lot of information about:

• Non-direct impacts on personal agency and self-worth
• Community mobilisation and empowerment
• Policy participation
• Providing advice to the health and social services sectors

This makes it difficult (or impossible) for peer responses to show their full impact and value.

It also creates two further barriers to a peer response’s capacity to conduct robust evaluation:

• Inadequate resources
• Inappropriate evaluation indicators

Inappropriate evaluation indicators

Evaluations of peer responses often use inappropriate or incomplete indicators. This is related to the previous two points we discussed (inappropriate evaluation frameworks and overemphasis on funder priorities).

The evaluation indicators that interest donors often fail to measure the full extent and impact of peer work. (This is not least of all because most of these indicators originally came from evaluating non-peer work).

These indicators typically focus on individual-level impacts, leaving system-level impacts and synergies unseen and unmeasured. However, these system-level outcomes often add significant extra value for both communities and funders.

The upshot of this is that both the peer response and the funders miss out on the opportunity to learn so much more.

Insufficient resources

Peer responses often lack the resources to carry out robust evaluation.

Cost is a very real barrier for many organisations. Evaluation is a resource-intensive process that requires specific skills and expertise. Peer responses often lack adequate funds, staff, and other resources to conduct robust evaluations of their work.

Funders can often exacerbate this barrier by actively opposing evaluation activities that look beyond their own priority impacts.

How can we improve evaluations of peer responses?

Peer responses need to have evaluation processes that capture everything they do, so they can both:

• Share stories about and improve their work
• Collect the information that funders want

One solution to this is to use evaluation frameworks that are tailored to peer work. At the same time, evaluation processes need to be as streamlined and as relevant as possible.

This is precisely what the W3 Framework can help peer responses achieve.

Using the W3 Framework to improve evaluations of peer work

The W3 Framework can be used to guide evaluations of peer work. Peer responses that have used the W3 Framework in this way report that it has helped improve their evaluation processes, including through:

• Developing appropriate evaluation frameworks
• Streamlining data collection
• Showing impact beyond individual-level service access or knowledge and behaviour change
• Converting peer insight into compelling evidence
• Generating evidence to enhance organisational credibility within the health system and policy environment

Developing appropriate evaluation frameworks

The W3 Framework can be a starting point for developing theories of change that cover all four W3 Functions. Such theories of change can:

• Help peer responses better understand and describe their work
• Cover all areas of a peer response’s work, including those that are often missed, such as adaptation, positive impacts on the health system, and policy participation
• Broaden focus from individual-level impacts to include also community-level and system-level impacts

Peer responses can use this to help them design evaluation frameworks that result in more robust and more useful evaluations.

Case example

AFAO and AIDS Councils’ Theory of Change

The Australian Federation of AIDS Organisations (AFAO) is Australia’s national peak organisation for the community-led HIV response.

It adapted the W3 Framework to create a ‘Theory of Change’ for its peer-led work and that of the AIDS Councils within the overall HIV sector.

The resulting theory of change shows the:

• Unique role of AFAO and the AIDS Councils within Australia’s overall HIV response
• Link between their on-the-ground work and community-level and system-level impacts

Streamlining data collection

Applying the W3 Framework to peer work can help streamline data collection by helping peer responses:

• Tailor evaluation indicators to their specific needs
• Find the smallest number of data points that give the largest amount of information
• Gain more value out of the information that they already collect
• Convert the wealth of peer knowledge that peer workers have into usable evidence

Moving beyond individual-level impacts

As mentioned above, evaluations of peer work tend to focus on direct, individual-level impacts that cannot measure the full range of a peer response’s work. This ultimately:

• Results in substantial misrepresentation and underestimation of its effectiveness and overall impact
• Undermines its credibility within the sector
• Results in lost funding
• Reduces its overall effectiveness

The W3 Framework provides a structure to help peer responses develop indicators across all four W3 Functions. This helps them show their impact, not only on individuals but also at community and policy levels. It also helps them describe how these broader, higher-level impacts flow back through the system to improve individual and community health outcomes.

In other words, the W3 Framework can help peer responses to show the full breadth and depth of their work.

Case example

Living Positive Victoria

Living Positive Victoria is a peer-led organisation for people living with HIV (PLHIV).

In 2016, Living Positive Victoria merged with Straight Arrows, Victoria’s lead organisation for heterosexual PLHIV and their families.

The organisation experienced a period of rapid growth and began working with new communities.

This created a need for consistent and streamlined organisational evaluation processes to help:

• Understand its impact across all its programs and activities
• Adapt to its changing environment

Living Positive Victoria developed an organisation-wide W3 Framework–led evaluation process that enabled it to:

• Ensure that its evaluation processes support its work across all four W3 Functions
• Demonstrate the impact of individual programs and of the whole organisation
• Collate insights gathered through programs and feed them into annual business planning
• Enhance the link between program outcomes and strategic goals
• Better articulate and communicate its impact to stakeholders and funders

The value in what we are finding is across all of [W3’s Functions] because we aren’t just looking at program delivery — we are looking at systematic approaches to improve supports for people living with HIV.

Staff member

Living Positive Victoria

This process resulted in a shift in organisational culture where evaluation is now:

• A central part of project planning
• A strong feature of budgeting
• An element of staff recruitment

Converting peer insight into compelling evidence

When it comes to providing evidence for evidence-based interventions, peer insights are usually not valued as highly as social or epidemiological research or health service data.

However, the sectors where peer-based responses work can change so rapidly that these formal data become outdated by the time of release. This means that peer insights are often the only source of real-time information the sector has. Additionally, formal research and epidemiological findings can be misconstrued in the absence of nuanced, contextual interpretation that skilled peers can provide.

The W3 Framework provides a structure to help peer responses draw data from peer insights and community anecdotes, and present them in more meaningful, useful, and persuasive ways.

Enhancing peer response credibility

Policy participation is a core part of quality peer responses.

Policy advice from peer responses draws heavily from peer insights, which (as discussed above) are often perceived as less credible than other types of evidence.

Because of this, successful policy advice often requires peer responses to partner with allies (such as researchers and other sector advocates). This enables them to develop a reputation of credibility over time so that policymakers and sector partners increase their trust and confidence to act on peer input.

For peer responses to provide relevant, high-quality input, they need strong relationships and influence within their communities. The W3 Framework can inform the collection and presentation of evidence to monitor and show how peer leadership activities and peer leaders:

• Authentically engage with their communities
• Draw on high-quality engagement to identify key insights about emerging issues
• Package these in a way that justifies the need for the changes while also acknowledging pressures faced by other actors in the policy system
• Propose effective, practical, sustainable, and feasible changes

In other words, the W3 Framework can help peer responses generate evidence to improve their credibility within the health system and policy environment.

Case example

Harm Reduction Victoria’s Peer Network Program

The goal of the Peer Network Program is to engage people who inject drugs (PWID)
who are not being reached by mainstream needle and syringe programs (NSP).

Staff knew they were achieving this but the program’s data collection tools were not sensitive enough to record it in a meaningful way.

They used the W3 Framework to adapt and improve the program’s existing data collection tools and processes.

The improved tools can are helping the program:

• Produce data about the numbers of PWID who would not have obtained sterile injecting equipment if not for the Peer Network Program
• Demonstrate its reach and impact on the community
• Provide robust and compelling evidence of its value

Where to next?

To sum up, peer responses can use the W3 Framework to help guide evaluation of the full breadth and impact of their work.

Our Essential guide to improving peer responses talks about how peer responses can use the results of evaluations to help improve their work.

If you work in a peer response and would like to use the W3 Framework to help improve your evaluation processes, check out the W3 Framework Guide and tools.