Stage 3 of the W3 Project is a national study of the impact of Australian peer-led HIV and hepatitis C programs.
In this stage, we are pooling resources and data from selected peer-led responses in multiple states across Australia. We are hoping to build an evidence bank that shows how peer-led responses achieve and sustain impact in their work across the four W3 Functions. This will help us generate stronger and clearer evidence of what works, and why, and provide insights and guidance for the investment and scale-up of peer programs in priority populations.
Background to Stage 3 of the W3 Project
The Australian National HIV and Hepatitis C Strategies affirm the importance of community-based and peer-led approaches. These approaches have a unique and important role in the HIV and hepatitis C response. This comes from their strong, positive influence on their communities and on the health systems and policies that affect community health.
Despite this, the evaluation that funders ask for often focuses only on individual-level factors that cannot measure system-level impacts and synergies. This makes it hard for peer-led responses to show the full extent of their impact and value.
The W3 Project’s third stage is a national study of how peer-led responses achieve and sustain the four W3 Functions in their work. We will pool resources and data from selected peer-led responses in multiple states of Australia to create an evidence base of their impact.
We believe this will support peer-led responses to:
- Enhance the implementation and impact of their programs
- Describe and demonstrate their full impact and value
- Advocate for targeted investment and scale-up of peer-led programs with priority populations
- Adapt quickly and confidently to rapid changes
Objectives
During Stage 3 of the W3 Project, we aim to:
- Draw together evidence from multiple organisations to show the role and impact of peer-led responses
- Compare peer-led-program models in different organisational, geographic, and policy contexts
- Identify useful and practical indicators of effectiveness for peer-led programs
- Provide guidance for those funding, managing, and evaluating peer-led responses
Methods
To guide our work, we are using the Locate evidence, Evaluate evidence, Assemble evidence, inform Decisions (LEAD) framework for assembling evidence and informing decisions.
We chose this framework due to its relevance in guiding the identification and collection of diverse, heterogeneous, meaningful evidence to help inform decisions around complex public health problems using a systems-thinking approach.
The LEAD Framework outlines five key tasks:
Although depicted as a linear process, it is not linear in practice. Due to the nature of locating evidence within and across diverse contexts and data sources, the process should be undertaken in a cyclic, iterative way.
Current progress
Specifying questions
Specifying questions centres around defining a set of indicators against the W3 Functions.
In order to achieve any of the objectives for this project, we need a way to determine how peer-led responses achieve and sustain impact in their work across each of the four W3 Functions.
Having a complete set of W3 Indicators provides a framework to help us collect evidence and answer questions, such as:
- Is the peer-led response achieving this indicator?
- Can the peer-led response demonstrate that they are achieving this indicator?
- What are the barriers and enablers to the peer response achieving (or demonstrating achievement of) this indicator?
Defining initial indicator lists
During 2021, we worked with our partners to develop evaluation indicators to measure each of the W3 Functions.
It was important to us that the list of indicators be able to capture:
- The full scope of the work that peer-led organisations and programs undertake
- The role that they play within both their communities and the health sector and policy environment
It was also important that they be practical and useful for both the peer responses themselves and for their funders.
In order to achieve this, we invited peer workers and representatives from our partner organisations and from state government health departments.
In order to develop a list that everyone agreed upon and felt was useful, we used a modified Delphi Method.
The initial step was to develop draft lists of indicators based on what we learned from piloting the W3 Framework during Stage 2 of the W3 Project.
Participants provided their feedback on the draft indicators via online surveys, which we analysed and used to create a revised list of draft indicators. We repeated this process once more, after which we invited our partner organisations to provide a last round of feedback through face-to-face meetings.
The lists include broad indicators:
- For use at both organisation and program levels
- To measure quality/process and impact
- That cover all the W3 Functions
We described this work in our report ‘W3 Project: Creating a set of evaluation indicators for peer-led work’.
Please wait while flipbook is loading. For more related info, FAQs and issues please refer to DearFlip WordPress Flipbook Plugin Help documentation.
Ongoing ‘specifying questions’ work
As we locate evidence against the indicators, some gaps are emerging in our indicator lists.
We are, therefore, tweaking and refining the lists to ensure they can truly capture the scope and role of peer work.
Locating evidence
Locating evidence involves working collaboratively with peer-led programs and organisations to identify and gather the types of evidence that are potentially relevant to each of the W3 Indicators.
We are conceptualising ‘evidence’ as very broad, including:
- Evidence originating from any source (i.e. not just information produced by the peer organisations)
- Relevant records or information existing in any form within the peer-led organisations (i.e. including information not created or usually used for evaluation purposes)
- Expert knowledge (extensive understanding and experience of the peer organisation’s communities, systems, processes, activities, and/or impacts)
Locating evidence against the organisation-level W3 Indicators
We began identifying potential sources of evidence against each indicator while we were developing the indicators. (You can see these initial lists in the report about that work in the ‘examples of potential metrics and sources of evidence’ columns of the indicator tables).
We have continued to develop these lists through meetings and workshops with our partner organisations.
These meetings involve peer staff in a diverse range of roles, including:
- Executive leadership
- Program management/coordination
- Communications
- Program evaluation
- Service delivery
Our report ‘W3 Project: Locating evidence against the W3 Indicators for peer-led work’ outlines the types of evidence we are most commonly locating against the organisation-level W3 Indicators. It also identifies some common enablers and barriers to locating this evidence.
Please wait while flipbook is loading. For more related info, FAQs and issues please refer to DearFlip WordPress Flipbook Plugin Help documentation.
Ongoing ‘locating evidence’ work
We are continuing to work in close collaboration with our partners to locate and gather all the available evidence against the W3 Indicators.
Evaluating evidence
Evaluating evidence involves assessing the evidence we have gathered for its descriptive power and rigour.
This process is happening in tandem with the locating and assembling evidence processes. This means that as we:
- Locate evidence, we are assessing its quality and coding it against a list of all indicators
- Continue building a bank of evidence, we are constantly assessing what evidence best demonstrates the extent to which the indicator is or is not being achieved
Assembling evidence
Assembling evidence involves selecting and summarising the evidence we have gathered and evaluated. We are using a framework synthesis method to do this, with the W3 Framework and W3 Indicators as our framework.
This process is happening in tandem with ongoing specifying questions and locating and evaluating evidence work. This means that, as we assemble the evidence we return to:
- ‘Specify questions’ if we find evidence of impact or quality that is not reflected by the current W3 Indicator lists
- ‘Locate evidence’ if indicators are lacking evidence regarding quality or impact
- ‘Evaluate evidence’ if there is a large amount of evidence against particular indicators and need to identify the strongest evidence
Informing decisions
We will draw on framework synthesis, complemented by thematic synthesis and textual narrative synthesis, to generate:
- A consolidated evidence base of the impact of peer-led work within Australia’s BBV response
- Recommendations to support:
- Peer organisations to evaluate and demonstrate the full impact of their work
- The strategic use of peer insights to inform peer work and policy participation
- Funders and policymakers to create enabling environments for peer-led work
This work will begin once we are confident that we have located most of the evidence our partners can provide.
Publications
Partners
PLHIV-led organisations
PWUD-led organisations
Funding
Stage 3 of the W3 Project receives funding from the Australian Government Department of Health
Our other research projects
This post was first published on 10 June 2021. We have edited the post to keep it up to date.
