V
Evaluating Outcomes and Enhancing Effectiveness Summary, Panel Discussion
The final panel, moderated by Marco Ferroni, executive director of the Syngenta Foundation for Sustainable Agriculture, examined the ways in which partnerships measure their progress, particularly as it affects the human and natural environment. Following the panel discussion, workshop participants offered additional insights into ways that partnerships might operate more effectively. Despite the increasing body of experience with partnerships for sustainability, there is still substantial room for improvement in the area of evaluating their outcomes. There is a tendency on the part of partnerships to celebrate the efforts that go into the partnership, rather than focusing on an assessment of results. By extension, there is a tendency for partnerships to develop metrics that focus on processes both internal and external to the partnership with less attention paid to other measureable outcomes. This final session focused on the results frameworks that various partnerships have developed to evaluate their impacts. Panelists were asked to reflect on four specific points: outcomes in the context of the partnerships’ results frameworks; outcomes as actually achieved; the partnering process as an outcome; and lessons going forward from the evaluations.
To develop a results framework, partners must formulate a hypothesis of how their partnership will affect reality. This may be referred to as an “impact pathway” or a “results chain,” and requires that partners consider how a partnership’s actions will affect external actors and processes. Panelists noted that this step warrants careful attention. Taking stock of the various pathways and potential pitfalls may lead the partnership to reconsider its mission, its membership, and its mode of operating. It is important to
begin developing this framework early on, understanding that it can and likely will change as the partnership matures. This is where it is helpful for a partnership to understand where it fits within a typology; many of the metrics it might use for evaluation can be widely applied across a group of partnerships, such as those that focus on advocacy and information dissemination. A number of panelists stressed the need for partners to work on a “value proposition” and be results-oriented instead of purely process-driven.
Several panelists also drew distinctions among outputs, outcomes, and impacts, and noted that a partnership will have progressively less control over the results as one moves from outputs (e.g., a new vaccine), to outcomes (e.g., vaccine is widely deployed), and then impacts (e.g., mortality is reduced and incomes increase). Partnerships must also consider alternative scenarios—what would have happened in the absence of the partnership—as they evaluate outcomes and impacts. In doing this, a partnership must also baseline its program, assessing the current conditions that it seeks to improve. One final point that panelists made was that the partnership process in many cases can also be considered an outcome, particularly when it influences future cross-sector work or changes within partners’ home institutions.
The East Coast Fever Vaccine project identified one simple output, an experimental subunit vaccine. Partners were drawn together in the hopes of creating this vaccine, which would next be developed into a deployable vaccine that was safe, effective, and affordable for East African farmers. Over the longer term, such a vaccine should help increase livestock productivity and reduce losses to farmers. However, the initial output was never achieved, prompting the partners to dissolve the partnership after they had agreed that they could not attain a proof of concept. Nonetheless, the partnership effort did affect the agricultural research system within which it operated. It led to a new platform (GALVmed) for livestock research, and there is evidence that both private sector and academic research partners are now approaching joint ventures differently, and with more enthusiasm.
The Sustainable Forest Products Global Alliance’s expected outcome was not quantified, although the “expansion in the proportion of internationally traded forest products sourced from sustainably managed certified forests” is one which could be, and has since been, calculated in terms of hectares and monetary value. The World Wildlife Fund’s projects increased in value from US$6 billion in 2003 to US$42 billion in 2007. Hectares of sustainably managed forest more than doubled, from 10.5 million to 26.5 million, and the number of participating companies more than tripled. However, other expected outcomes that focused on the partnership process were not achieved. Ultimately, the U.S. Agency for International Development (USAID) did not develop relationships with the private sector
members (which worked through NGOs as intermediaries), and even the relationships with the NGOs were universally viewed as being a traditional donor–grantee relationship, as opposed to a partnership. Ultimately, it seemed that the existing bureaucratic structure in particular did not allow USAID to participate as a partner.
Interestingly, these two cases (ECF and SFPGA) were singled out in discussions as examples of “failed” partnerships even though practitioners noted several “successes” resulting from the partnership efforts. In the former case, the failure was not unexpected (partners could not provide proof of concept for a viable vaccine), and was not attributed to the fact that partners had formed this alliance to try and address the challenge. In the latter case, SFPGA partners were critical of the notion that it was indeed a partnership. Instead, the arrangement did not stray far from traditional donor–grantee relations, though partners noted that outcomes could have been more significant and could have had more impact had they been able to stretch that boundary and work more closely with one another.
Global Water Challenge’s (GWC’s) mission—access to clean water for everyone—is one that is quantifiable, but so ambitious that the partners instead have chosen to focus on scalable activities that add up to large, dramatic outcomes. An example of these “micro-outcomes” includes a program focused on clean water access and sanitation practices in Kenyan schools, which demonstrated drops in absentee rates. Once partners feel that they have demonstrated a proof of concept, they set about scaling these projects up. Scalable solutions often require partnering with governments and other large groups that recognize that they might benefit from partnering with GWC.
The Common Code for the Coffee Community (4C) held a visioning exercise early on to map out its results framework. Partners posed the question, “What does ‘good’ look like?” within the coffee industry, taking account of all links along the value chain. Specialty standards already existed and were improving, but were not reaching everybody in the field. Therefore, 4C set up working groups based on the three pillars of sustainability (economic, social, and environmental). Eventually, these three groups created 4C’s principles, which pointed to activities whose outcomes could be measured. In order to achieve these measurable outcomes, the partners agreed that the missing ingredient to date had been collective knowledge. Thus, everyone with a piece of that knowledge was invited to the table to shape the specific goals of 4C, which also encouraged the new partners to buy in to the results framework. Indicators are being developed with on-the-ground partners. Though this requires that 4C establish several regional offices to interface with the local partners, it has also helped the partnership use verification as a development tool, as opposed to a policing action. A support component was established so that farmers can receive
training to help them improve and satisfy the 90 criteria that 4C uses when evaluating compliance with 4C’s principles. These criteria are not viewed as “pass/fail,” but as a dashboard that highlights strengths, weaknesses, and areas in need of improvement.
The Common Code set a goal of certifying 50 percent of the coffee market by 2015, and this led participants into a broader discussion of modest “realistic” targets versus ambitious “stretch” targets. A panelist pointed out that the ability to set ambitious targets is actually an advantage of partnerships. Nonetheless, prospective partners can sometimes be concerned with highly ambitious objectives that run counter to a home institution’s risk management strategy. Many institutions will be selective when engaging in partnerships and concerned about their ability to live up to the partnership’s obligations without assuming too much individual risk to their reputations (or finances). Once a partnership is well established, though, it may be easier for a risk-averse partner to accept ambitious or “stretch” targets. The Common Code’s response has been that it is better to aim for perfection and miss than to aim low and achieve something ordinary. The partnership’s targets are not being defined by existing resources or capacity. Instead, partners identify stretch targets, and then the secretariat continues to push the partners to aggressively pursue these and acquire the necessary resources. Since the 4C partners have come together to support the emergence of sustainability within the entire sector, one of their stated outcomes is to define a path toward sustainability, which entails setting ambitious targets and then working backwards.
In charting such a path to sustainability, it is also useful to identify the indicators and metrics that will help chart and report progress. Some potential outcomes may manifest themselves during the course of the partnership and were not conceived of at the beginning, so midcourse evaluations or even partner surveys can be a useful means of identifying and incorporating these. Partnerships need not expend great time and energy developing unique sets of indicators; there are several sets of existing, well-defined indicators and metrics from different sectors that may be applicable. As an example, infrastructure public–private partnerships have undergone a change in the way they report their outcomes. Initially, partnerships focused on the inputs; e.g., dollars spent on a project; or on basic outputs; e.g., kilometers of roadway constructed. Possibly influenced by private sector involvement, however, these partnerships are increasingly reporting outcomes in terms that are better understood by consumers and end users, e.g., amount of time saved on a commute. And while indicators are an important measure for reporting and accountability, their value as a learning tool within the partnership should not be underestimated.
In concluding the final session, panelists and participants were asked to identify the most salient ideas for enhancing the effectiveness of part-
nerships for sustainability, based on their own experiences in analyzing or engaging in partnerships. To do this, participants were divided into four discussion groups, each with a rapporteur, and asked to present their top three ideas for enhancing the effectiveness of partnerships. Although groups often identified more than three ideas, these ideas addressed three overarching themes, roughly categorized and summarized below: establishing a vision, creating decision support tools, and replicating success. Establishing a vision is a critical process in which a partnership will engage early on, where partners seek to align their different visions and identify key stakeholders, as well as objectives on which they can find common ground, or a common viewing point. Most participants agreed that everyone ought to be clear on what individual partners bring to the table, what they need from the table, and what the value-added is of everyone working together.
This clear understanding of incentives and rewards also relates to an important decision support tool: metrics. Since partnerships are voluntary activities, it is important for a partnership to design or utilize existing metrics that reflect the incentives of the partnering institutions and will allow the partnership to progress. Identifying the impact pathways up front will also help a partnership develop appropriate metrics. A previous review of agricultural research partnerships1 revealed that most were not taking the time to do this, for example, to identify how a new crop variety would lead to positive impacts for farmers. And while metrics and indicators are valuable decision support tools, partnerships can also benefit from tools that help guide them through issue framing, expansion, self assessments, and other general processes common to these sorts of efforts.
Finally, this idea of lessons from which generalizations can be drawn relates to the third issue, that of replicating success. Many participants emphasized that partnerships will benefit themselves and other efforts by improving their reporting, and by involving external parties to do this analysis, documentation, and dissemination of both “best” and “worst” practices. These “lessons learned” are not easily located at present, and the transaction costs for an individual partnership to do so are generally too high. Yet early efforts to mine this extant knowledge, like the work of the United Nations Commission on Sustainable Development, or the Partnering Initiative, seem to demonstrate that there is value in such exercises.
Looking ahead, the lessons learned should inform existing and nascent partnership efforts, but there is also a need for “partnership basics” to be taught in a way similar to how general management is taught, so that more and more people have the skills necessary to work in a partnership.
