This chapter discusses the rationale for designing an effective evaluation process. Subsequently, this chapter reviews the U.S. Fish and Wildlife Service’s (FWS’s) approach to evaluating the Landscape Conservation Cooperatives (LCCs) and provides guidance on what constitutes reasonable short-, medium-, and long-term metrics to evaluate the LCC Network. In this chapter, the committee responds to the following parts of the statement of task: What is the FWS’s strategy to assess the effectiveness (outputs and outcomes) of the LCC Network? What are reasonable short-, medium-, and long-term metrics for the effectiveness of the LCC Network in achieving its stated purpose and goals?
The stated purpose or vision statement for the LCC program is to achieve “[l]andscapes capable of sustaining natural and cultural resources for current and future generations.”1 To achieve the prescribed vision and mission, the LCC Network has articulated its strategy (also described in Chapters 1 and 3), which contains four key goals: conservation strategy, collaborative conservation, science, and communications (LCC, 2014). Each goal is accompanied by a set of objectives, and each objective is further supported by “example tactics.” Some of the tactics are essentially metrics used to measure progress in meeting individual goals for the network as a whole. The LCC Network Strategic Plan (hereafter referred to as the strategic plan; LCC, 2014) and its goals and objectives are described in Chapter 3, where we also offer a framework for setting clear means, process, and ends objectives. Here we review the FWS’s evaluation approach and offer guidance to measure progress toward achieving the objectives of the LCCs and the LCC Network. This chapter begins with a discussion of the need for and challenges of program metrics.
In general, and for the LCCs in particular, the challenges of establishing shared metrics and outcomes in a landscape-scale context (meaning multijurisdictional and multistakeholder) are well documented (see Box 4.1 for definitions). Typical challenges include, but are not limited to, the following (noting that some are explicit challenges to setting appropriate metrics, whereas others are challenges in process or outcome that may create difficulty both in setting appropriate metrics and/or later evaluating whether those metrics were correctly set):
- Defining an appropriate scale at which to measure conservation outcomes
- Unifying conservation objectives in situations where the leadership setting those objectives is distributed
- Integrating scientific information with management decisions
- Engaging multiple stakeholders, leading to diverse conservation objectives that may not work synergistically (e.g., Peterson et al., 2014)
- Considering the inherent complexity of the natural systems involved, in which apparent positive outcomes in any part of an ecosystem may involve trade-offs and/or create negative consequences in other parts of the ecosystem (NRC, 2005b; Game et al., 2013)
- Involving differences among focal questions, leading to varying methods of evaluation depending on the question being asked (Mascia et al., 2014)
- Distinguishing failure of a process or evaluation approach from failure of outcome (Peterson et al., 2014)
- Considering whether and under what conditions commonly espoused management techniques such as adaptive management are applicable (Allen and Gunderson, 2011; Doremus, 2011; NRC, 2011)
Given the presence of multiple stakeholder groups with multiple objectives and the absence of universally accepted approaches to monitoring and evaluating outcomes, it is not yet possible in a landscape-scale context to consistently evaluate whether the process, and associated metrics of the process, is directly linked to the outcomes later achieved and associated metrics of the outcome. In other words, establishing causal links between collaborative conservation planning and actions and the ultimate outcome is very difficult if not nearly impossible (such challenges are also described in Appendix A). Furthermore, because the LCCs primarily function as conveners and facilitators of collaborative conservation with the goal of “improving the management of fish, wildlife, and habitats,” any evaluation will primarily demonstrate the LCCs’ contribution to developing conservation strategies, delivering the science to inform management, and/or the quality of collaboration. Such evaluation challenges, however, are not unique to LCCs; nor does their recognition render irrelevant the need to articulate conservation measures for the purpose of outcome evaluation and program success. Indeed, a 2008 document prepared by the FWS titled Strategic Habitat Conservation describes the need for evaluating conservation programs in terms of inspiring “investor confidence.”
Different approaches for evaluating conservation programs have been identified together with a set of focal questions to help determine the appropriate framework for evaluation (Mascia et al., 2014). Out of those, the performance measurement approach, which involves “the process of measuring progress toward specific project, program, or policy objectives, including desired levels of activities, outputs, and outcomes,” appears most relevant here (Mascia et al., 2014). Its associated focal question is: “To what extent is a conservation intervention making progress toward its specified objectives for activities, outputs, and outcomes?” This approach is also consistent with the report titled Performance Measurement and Evaluation: Definitions and Relationships (GAO, 2011). It is important to note that this type of program performance evaluation can be effective in evaluating and improving conservation programs even without establishing a causal link between conservation actions of a program and the outcome in the health of a resource. Establishing such causal links is very difficult given the complexity of the system and the many uncontrolled variables.
Without suggesting specific individual performance measures, the Government Accountability Office (2011) identifies and suggests three categories of performance measures that are relevant across federal agencies: those that measure process, those that measure outputs, and those that measure outcomes (see Box 4.1). Furthermore, these performance measures meet the requirements set forth by the Office of Management and Budget (OMB). OMB developed guidance for program evaluations, in part to help agencies meet their requirements under the Government Performance and Results Act of 1993 (GPRA, 1993) and the Government Performance and Results Modernization Act (GPRA, 2010).2 This law requires agencies to develop strategic plans and goals against which to measure performance. Annual reporting to OMB requires agencies to provide performance measures to assess outputs, service levels, and outcomes of each program activity as well as a comparison of accomplishments with performance goals set by agencies.
Review of the FWS’s Evaluation: Process, Outputs, and Outcomes Assessment of the SIAS Document
The FWS Science Investment and Accountability Schedule (SIAS) is the current approach of the FWS to assess the effectiveness of the LCC program. The SIAS was developed by FWS
to express the Service’s vision for, and to inform the Service’s investment in, the suite of activities, actions and outcomes that an LCC would accomplish as it develops as a collaborative conservation forum; and, to help respond to Congressional direction that “the Service establish clear goals, objectives, and measurable outcomes for LCCs that can be used as a benchmarks of success in the program.”
Benchmarks serve as a point of reference relative to which progress is measured (i.e., metrics). The SIAS consists of “eight, interrelated Conservation Activity Areas and twenty-two associated benchmarks that are guided by the Strategic Habitat Conservation Handbook (SHC; National Technical Assistance Team, 2008) in support of the LCC Network’s Vision and Mission” (SIAS, 2013) (see Table 4.1). The foundation for the SIAS is the SHC handbook and several of the SIAS Conservation Activity Areas are directly identifiable in the SHC handbook (National Technical Assistance Team, 2008).
Each LCC coordinator completes the SIAS evaluation form for its respective LCC, which is designed to measure progress toward each of the eight SHC Conservation Activity Areas and their related progress benchmarks by means of associated metrics. An extract from the North Atlantic LCC SIAS 2.0 for 2014 (see Table 4.2) is provided to illustrate both the SIAS framework and a sample result from a completed evaluation form. The identification of benchmarks is important because they are reference points for measuring progress, and these benchmarks comply with the OMB requirement for performance goals.
Breaking the SIAS evaluation form down and reorganizing it according to the “process, outputs, and outcomes” categories in the Government Accountability Office report (GAO, 2011) enables a closer exploration of the organization of the SIAS document.
Challenges of Metric-Setting in the LCC Context
Since the development of the SIAS, the LCC Network has developed its strategic plan (LCC, 2014). Furthermore, the individual, self-directed LCCs have engaged in strategic planning efforts with their respective steering committees and many have developed their own strategic plans. Both the network-level strategic plan and the individual LCC plans tend to be written for internal, within-network audiences. It appears that they were not developed for the purpose of, and are not intended for, establishing metrics for program assessments.
TABLE 4.1 The 22 Benchmarks Under the Eight Conservation Activity Areas
|Conservation Activity Area||Benchmarks|
|1. Organizational operations||1A. Engagement and coordination;|
|1B. Leveraging resources;|
|1C. Evaluating progress; and|
|1D. Engaged technical community and dedicated technical staff.|
|2. Landscape conservation planning foundation||2A. Assess existing conservation efforts;|
|2B. Identify priority resources;|
|2C. Collate and establish conservation goals and measurable objectives; and|
|2D. Refining landscape conservation planning foundation.|
|3. Landscape Conservation Design||3A. Vulnerability and landscape assessments;|
|3B. Adaptation strategies; and|
|3C. Integration of multiple priority resources and associated measurable objectives into Landscape Conservation Designs.|
|4. Informing conservation delivery||4A. Provide decision support;|
|4B. Information delivery;|
|4C. Assessment of information delivery;|
|4D. Collaborative conservation delivery to realize resource objectives; and|
|4E. Tracking delivery on the landscape.|
|5. Decision-based monitoring||5A. Collaborative monitoring; and|
|5B. Monitoring change of the landscape and priority resources.|
|6. Research to support adaptive management||6A. Testing underlying assumptions.|
|7. Data management and integration||7A. Data management and integration.|
|8. LCC Network function||8A. Participation in the LCC Network enterprise; and|
|8B. Function as part of integrated network of LCC partnerships.|
TABLE 4.2 Science Investment and Accountability Schedule (SIAS 2.0) for the North Atlantic Landscape Conservation Cooperative (NALCC) for Fiscal Year 2014
|SIAS 2.0 (FY 2014)||North Atlantic LCC|
|Conservation Activity Areas and Benchmarks||Metrics||Metric Score||Bench-mark||Metric Score||Justification (limited to <4,000 characters)|
|1. Organizational Operations: Addresses fundamental organizational and administrative components necessary to establish and maintain an LCC as part of the national LCC Network. The LCC Partnership is composed of participating organizations (LCC Partners), is directed by the LCC Steering Committee (LCC SC), and is supported by the LCC Staff as well as science, technical, and other work teams. The LCC Staff and LCC SC and their associated organizations actively engage other relevant individuals, organizations, and partnerships creating collaborative relationships with key decision makers who are able to influence current and future landscape conditions. The LCC Staff maintains strong professional contacts and connections, networking to keep LCC Partners abreast of current conservation issues, techniques, etc. The LCC Staff also identifies partner capabilities to address the LCC mission and works with partners to address capacity gaps by adding key positions, relying on partner capacities, utilizing contracts, or by training appropriate to the size and complexity of the LCC geographic region (LCC Geography). LCCs must work closely with other conservation science and delivery activities to ensure efforts are coordinated and integrated. The LCC participates in development of common national LCC Network messages to relevant state, regional, and national entities. The LCC works to ensure its activities are coordinated and integrated with those of the Climate Science Centers, Cooperative Fish and Wildlife Research Units, Cooperative Ecosystem Studies Units, Forest Service Research Centers, Joint Ventures, Fish Habitat Partnerships, and similar key players.|
|SIAS 2.0 (FY 2014)||North Atlantic LCC|
|Conservation Activity Areas and Benchmarks||Metrics||Metric Score||Bench-mark||Metric Score||Justification (limited to <4,000 characters)|
|1.A - Engagement and Coordination - LCC Staff and Steering Committee are actively fostering strategic engagement, collaboration, and coordination with a diversity of entities that influence landscape conservation decisions, including state and federal agencies, tribes, universities, NGOs, regional partnerships (e.g., JVs, NFHPs, AFWA regions), and regional and local community planners.||No||0||1A||1||The North Atlantic LCC has a broad and active partnership of more than 100 partners including a Steering Committee with 33 members representing federal and state agencies, tribes, Canadian partners, and NGOs. Three technical teams and several, project teams bring together technical expertise from agencies, universities, and organizations across the LCC geography. A science delivery team links to decision makers at regional, subregional, state, and local (land trust and community) scales. Regional partnerships including the Atlantic Coast Joint Venture, Eastern Brook Trout Joint Venture, Atlantic Coastal Fish Habitat Partnership, Northeast Partners in Amphibian and Reptile Conservation, Northeast Regional Ocean Council, Mid Atlantic Council on the Ocean, and others are linked to the LCC through team members and/or projects. The LCC has a particularly strong relationship with the Northeast Association of Fish and Wildlife Agencies at the director, administrator, and technical levels including a joint effort to support regional work in support of regional context for State Wildlife Action Plans.|
|1.B - Leveraging Resources - LCC Partners contribute resources (e.g., staff, funding, infrastructure, tools, expertise, etc.) to fill administrative and technical capacity, and information gaps necessary to achieve the LCC mission.||0% of total FWS annual investments leveraged by partner contributions (cash and/or in kind).||0||1B||2||North Atlantic LCC partners contribute resources in numerous ways including staffing, in-kind participation and travel, complementary projects, and match. The U.S. EPA contributed a full-time liaison to the northeast LCCs (North Atlantic and Appalachian) through the first part of 2014 and the National Park Service contributes a portion of their coastal landscape adaptation coordinator’s time to support LCC activities. The LCC and The Nature Conservancy shared a GIS analyst position to ensure spatial data are available to partners during part of 2014. About 50 non-FWS steering committee, technical team, and science delivery team members provide in-kind time and travel in support of LCC activities for several days a year. The close working relationship with the Northeast Climate Science Center results in a number of leveraged, complementary projects (e.g., integrated stream science projects). The NEAFWA RCN program provides directly complementary project support toward common goals in the Northeast Conservation Framework. Science Delivery partners use LCC funds to leverage their existing partner network capacity. The Connecticut River Landscape Conservation Design Pilot required significant in-kind participation by 30 FWS and non-FWS partners. The North Atlantic LCC successfully competed for > $5 million in DOI Hurricane Sandy Resiliency funds to address LCC priorities related to marsh, beach, and stream resiliency - about 5% of these funds were leveraged in 2014. LCC projects do not require match but several projects providing matching funds or in-kind services. The total of these contributions in 2014 was about 50% of the annual North Atlantic LCC budget.|
|1% to 33% of total FWS annual investments leveraged by partner contributions (cash and/or in kind).||1|
|34% to 66% of total FWS annual investments leveraged by partner contributions (cash and/or in kind).||2|
|67% to 100% of total FWS annual investments leveraged by partner contributions (cash and/or in kind).||3|
|>100% of total FWS annual investments leveraged by partner contributions (cash and/or in kind).||4|
|SIAS 2.0 (FY 2014)||North Atlantic LCC|
|Conservation Activity Areas and Benchmarks||Metrics||Metric Score||Bench-mark||Metric Score||Justification (limited to <4,000 characters)|
|1.C - Evaluating Progress – The LCC Steering Committee has established metrics and processes for identifying, collaboratively pursuing, and evaluating actions in support of the LCC’s mission, goals, and objectives. The LCC develops a comprehensive strategic action plan, updated on a regular defined time period, that describes their science agenda, approach, monitoring, and communications strategy and progress in collaboratively achieving the LCC mission.||Part i: Has the LCC started a comprehensive strategic action plan?|
|The LCC has not started a comprehensive strategic action plan.||0||1C(i)||2||The North Atlantic LCC completed a comprehensive Conservation Science Strategic Plan in 2011 as well as a draft science delivery plan and communications framework in 2013-2014.|
|The LCC has started a comprehensive strategic action plan.||1|
|The LCC has completed a comprehensive strategic action plan.||2|
|Part ii: Has the LCC Steering Committee started a process for evaluating progress?|
|The LCC Steering Committee has not started a process for evaluating progress at regular intervals toward established goals and updating the identification and prioritization of the most important science and capacity needs to support LCC goals.||0||1C(ii)||2||LCC staff provide a state of the LCC presentation annually at each April Steering Committee meeting and the Steering Committee reviews and provides input on shifting of priorities. In 2013–2014 the Steering Committee recommended a shifting of resources toward science delivery that was reflected in a new team, strategy, and grant program for science delivery. The LCC technical teams review science priorities each year and provide recommendations on needs to address and update the science needs matrix portion of the strategic plan. In 2013–2014, that review resulted in adaptive actions to fund work that would be responsive to and complementary to Hurricane Sandy resiliency funded work and ongoing Landscape Conservation Design projects. In 2014, the Steering Committee and staff initiated a review of the strategic plans with the intent of consolidating and updating them in 2015–2016.|
|The LCC Steering Committee has started this process||1|
|At least one iteration of this process, resulting in an updated strategic action plan, has been completed. Note: Report (in narrative form) on the identified adaptive actions taken as a result of the process.||2|
|1.C Summary Score||1C||4|
|1.D - Engaged Technical Community and Dedicated Technical Staff - The LCC has organized the technical capacity, including dedicated partner staff, needed to address priority conservation science needs. Further, the LCC has established a working relationship with USGS regional Climate Science Center(s) and other entities to ensure that science and conservation activities involving the LCCs have access to the best regional technical information and that priorities are coordinated and integrated.||The LCC has not organized technical capacity nor established relationships with the broader science community.||0||1D||2||The North Atlantic LCC has three engaged technical teams addressing coastal and marine, terrestrial, and wetland and aquatic science needs in the LCC science strategic plan. The LCC has a science coordinator, science delivery coordinator, data manager, and GIS analyst to provide technical staff capacity and staff support through EPA and NPS liaisons and a shared position with TNC. The North Atlantic LCC has strong working relationships with both the university and USGS components of the Northeast Climate Science Center (located 1 mile from the LCC office) and has been directly involved in developing and ranking the results of all CSC RFPs. Project oversight teams and peer reviewers ensure the LCC projects achieve their stated goals.|
|The LCC has established science teams or technical committees to assess science and technical needs for the LCC.||1|
|The LCC’s science teams or technical committees are addressing the LCC’s priority conservation science needs.||2|
NOTES: Shown is an extract of the SIAS 2.0 table for the NALCC with a description of the Activity Areas to be evaluated, titled “Organizational Operations” with corresponding benchmarks, metrics, and options of metric scores in the gray, left side of the table. On the blue-colored, right side of the table, the NALCC assigned itself a metric score and provided a justification for the metric score. The yellow highlighted area indicates the score the NALCC assigned to each benchmark. AFWA = Association of Fish and Wildlife Agencies; CSC = Climate Science Center; EPA = U.S. Environmental Protection Agency; FWS = U.S. Fish and Wildlife Service; FY = Fiscal Year; GIS = geographic information system; JV = Joint Venture; NEAFWA = Northeast Association of Fish & Wildlife Agencies; NFHP = National Fish Habitat Partnership; NGO = nongovernmental organization; NPS = National Park Service; RFP = request for proposals; TNC = The Nature Conservancy.
The SIAS is currently the only evaluation tool being used by the FWS and generally is focused on FWS-centric metrics, but, as the committee learned during discussions with LCC staff, it is not necessarily adaptable to the network and individual LCCs from the broader perspective necessary to measure complex landscape- and partner-driven outcomes. A single SIAS or sets of SIAS-focused metrics that encompass all important program elements will be complicated and perhaps dampen innovation in design. For this reason, the SIAS framework can meet objectives of the FWS independently from an evaluation framework that assesses the LCC Network and individual LCCs as long as some consistency is maintained at the network level. The discussions in this chapter do not assume the SIAS is the only metrics option, and indeed the LCC Network may wish to consider developing a separate framework outside of the SIAS.
Reflective of the challenges noted above, the structure of the LCC Network poses some unique challenges with regard to setting metrics. In most contexts, the focus of program assessment and metric-setting is to demonstrate how a respective agency’s investments have resulted in achieving the particular agency’s goals. In the LCC Network, where goals are to be achieved through collaborations across programs and agencies, subsequent outcomes must also be viewed in the context of these partnerships. This challenge is acknowledged by the FWS in its discussion of the SIAS (2013) and includes the recognition that additional measures must be forthcoming, stating that:
The partnership of LCCs provides the opportunity for significant conservation progress, but also demands a complex set of interactions in which different agency and group authorities and priorities must be respected. This version of the SIAS is referred to as SIAS 2.0 in recognition that it is the next evolution of the original SIAS performance system. . . . SIAS 2.0 is another step along the path to develop an incentive based approach to Service investments in LCCs. Therefore, SIAS 2.0 will require further refinement after it is implemented and eventually a new SIAS 3.0 will result.
This statement also acknowledges the intent to carefully use metrics that reflect only FWS effectiveness in implementing the network of LCCs, and not the outcomes that should result from the collaborative process.
Appropriate Short-Term Metrics
One of the LCC Network’s greatest strengths is its ability to respond to stakeholder priorities within a given region and its flexibility to collaborate across LCCs in response to conservation challenges that are multijurisdictional. However, this characteristic can also be a weakness in that it can create the appearance of a program serving a diversity of missions and objectives, and lacking focus. During a discussion with the committee, it was noted that the diversity of LCCs has led to a bewildering diversity of LCC science “requirements,” formats, priorities, and communication mechanisms. In general, the LCC Network needs to be able to demonstrate that it operates both efficiently and as a coherent program, and that the program’s constituent elements work together toward the program’s overall stated purpose and objectives and do so cost-effectively. For this reason, most of the short-term metrics listed below are process oriented, and are intended to refine FWS-relevant benchmarks associated with the SIAS, as well as to strengthen and clarify alignment of the SIAS with the strategic plan.
- Extent to which costs for each SIAS Activity Area can be identified and returns on investment relative to those costs identified. Because LCCs work through partnering with relevant agencies and much of the implementation authority lies outside the LCCs, it is important that they can demonstrate how investments were either leveraged or resulted in outputs or outcomes through activities or actions undertaken by LCC partners. Furthermore, because facilitating the development of shared conservation priorities is an important component, it is important that the SIAS also evaluate the breadth and scope of partner engagement (ACCCNRS, 2015). At present, the SIAS 2.0 contains a section titled “Leveraging Resources,” which is the only benchmark that identifies resources as an issue. This benchmark defines the contribution of the total FWS annual investments that have been leveraged by partner contributions, either cash, in kind, or both, under the Activity Area of “Organizational Operations.” For example, to reduce cost of office space, LCC coordinators for two LCCs work in offices provided by state agencies. Additionally, in the case of the Desert LCC, the Bureau of Reclamation provides the staff support to coordinate the LCC. Many LCCs receive technical and staff support from other state or federal agency partners on the steering committees. To the extent possible, it would be appropriate for the next iteration of the SIAS to more fully describe—either qualitatively or quantitatively—the returns achieved by the LCC Network for both the FWS and partner dollars invested.
- Extent to which benchmarks are appropriately assigned and distributed to achieve progress in each SIAS Conservation Activity Area. In the SIAS, benchmarks are more fully assigned to some Activity Areas than to others. For instance, “Landscape Conservation Design” contains numerous benchmarks addressing a wide range of strategies and services, while “Research to Support Adaptive Management” contains one benchmark. This uneven distribution of benchmarks could be interpreted as assigning value to the various Activity Areas and may or may not address fully the intent of that Activity Area. Although the benchmarks identified in some of the Conservation Activity Areas are appropriate, consideration might be given to whether the number of benchmarks needs to be evened out and whether
additional benchmarks may be needed in other areas to better define and measure progress.
- Degree to which SIAS benchmarks are associated with the appropriate short-, medium-, and long-range context. Because the SIAS progress benchmarks are not couched in the context of short-, medium-, and long-term progress, the accompanying metrics are also not framed in that context. Not all progress toward meeting benchmarks could proceed at the same pace or not all outcomes could be achieved at the same time.
- Extent of alignment at the network level between the SIAS and other key documents, i.e., the strategic plan. When developing program metrics, there usually is a fundamental framework that the metrics address. The basic framework of the LCC Network is the strategic plan. The strategic plan is based on a vision and mission with four strategic goal areas: (1) conservation strategy, (2) collaborative conservation, (3) science, and (4) communications. Nineteen objectives provide direction on how these goals will be accomplished (see also Chapter 3). When attempting to cross-reference the goals and objectives of the strategic plan to the metrics of the SIAS, a direct match of metrics to goals and objectives is not obvious. Many of the 22 benchmarks can be paired to single or multiple goals and objectives, but it is certainly not a set of metrics designed to align with the strategic plan—in part, because the SIAS uses the SHC framework for metrics while the strategic plan is not based on the SHC framework directly; and also in part because this version of the SIAS was completed before strategic plan development. This sequence of events could lead to some confusion and difficulty in communicating LCC effectiveness.
Assigning Appropriate Medium- and Long-Term Metrics
What constitutes medium and long term is a matter of judgment. Generally speaking, conservation outcomes are context dependent, and what might be appropriate as a medium-term metric in one situation or scale might be more appropriate as a short- or long-term metric in another. Nonetheless, with the intent of defining program phases, since the LCC Network was initiated 5 years ago (in 2010), it would be appropriate to assign a working definition to short term of 0–2 years from today (or 7 years since program inception), to classify medium term as 7–10 years from present, and to refer to long term as more than 10 years from present. This classification acknowledges that it takes a long time to achieve ends objectives (see discussion in Chapter 3), and that measuring some of the outcomes of a program might take much longer than the next 10 years, whereas others will not.
Using the process, output, and outcome assessment approach identified above, it will be helpful to conceptualize medium- and long-term metrics in terms of those that measure the outputs delivered and the outcomes they help achieve as well as those that measure the effectiveness of the collaborative process itself (see also discussion in Chapter 3 on ends and means objectives). As the LCC Network evolves, the SIAS metrics and those of any other LCC evaluation framework that might be developed need to be refined to more fully address the following factors.
Measuring Outputs and Outcomes: Outputs and outcomes are not needed for every action that an LCC takes; however, the LCC Network needs to be able to identify accomplishments that have occurred as a result of the LCC Network being in place, and also needs to be able to easily identify one or more clear successes for individual LCCs. Thus, in the committee’s assessment, measuring the medium- and long-term effectiveness of the LCC Network needs to include the extent to which funds can be associated with each of the SIAS Activity Areas (see Conclusion section). Measuring effectiveness also needs to be tied to appropriate medium-term outputs (products and services delivered) and longer-term outcomes (results achieved for the products and services delivered).
From this standpoint, reasonable medium- and long-term metrics for individual LCCs, which could then be aggregated at the network level, would assess the following:
- Extent to which both agency-unique contributions and their costs can be clearly identified and tied—qualitatively, quantitatively, or both—to program outputs and longer-term outcomes. As the LCC Network evolves, and in light of increasing emphasis on government performance and results, it is important for each of the public agencies involved to attempt, to the extent feasible and practicable, to identify its unique contribution and the costs of that contribution to the collective impacts achieved by the LCC Network. Metrics ideally include results achieved for both financial and human capital investments.
- Extent to which partner contributions and the costs of those contributions can be clearly identified and tied—qualitatively, quantitatively or both—to both outputs and outcomes.
- Extent to which both successes and failures can be meaningfully analyzed in ways that result in program corrections. As Baylis et al. (2015) wrote, “Understanding why conservation programs succeed or fail is essential for designing cost-effective initiatives and for improving the livelihoods of natural resource users.”
Measuring Collaboration: In addition, medium- and long-term metrics also need to measure the outputs and outcomes delivered by the collaborative process itself. Because the main goal of the Secretarial Order was to create a mechanism for facilitating collaborations across jurisdictional boundaries, it is important to measure the quality of and the deliverables from the collaborative process. As discussed in Chapter 3, this means objective is, at the same time, an ends
objective. However, measuring collaborations and outcomes from these collaborations is difficult for several reasons, as discussed above: (1) the FWS needs to meet the expectations from both OMB and Congress and demonstrate outcomes directly relevant to its core mission; and (2) measuring collaborations at the individual LCC level cannot be easily aggregated to demonstrate effectiveness of collaboration at the network level. Therefore, the committee provides a review and guidance on this measurement topic.
The work of Thomson et al. (2007) may be helpful to the LCC Network on an illustrative basis, because it identifies five key dimensions contributing to an overall construct of collaboration. Thomson et al. (2007) define collaboration as “a process in which autonomous or semi-autonomous actors interact through formal and informal negotiation, jointly creating rules and structures governing their relationships and ways to act or decide on the issues that brought them together; it is a process involving shared norms and mutually beneficial interactions.”
The authors suggest five factors relevant to measuring collaboration, briefly recited below. In seeking empirical data to measure the model’s validity, the authors structured survey questions that were sent to 1,382 managers of a large national organization; the survey approach may be something the LCC Network would want to consider in order to gather feedback on the effectiveness of the collaboration process. The collaboration metrics associated with each of the five factors could be rated using the format currently applied in the SIAS document, that is, a progressive, scaled rating system consistent with the current SIAS approach to measurement. The five factors are shown below, together with some suggested metrics that the committee finds relevant to the LCC Network in terms of measuring the effectiveness of its own collaborations.
- Governance involves “developing sets of working rules about who is eligible to make decisions, which actions are allowed or constrained, what information needs to be provided, and how costs and benefits are to be distributed” (Ostrom, 1990). Illustrative metrics, appropriate to LCC Network governance could include (a) identifying where overlapping objectives exist, and if such overlaps exist, (b) identifying the extent to which policy or procedural platforms exist that allow one organization to take a lead for a given issue to allow interagency collaboration to continue.
- Administration means that “some kind of administrative structure must exist that moves from governance to action . . . as public mangers [sic] know all too well, decentralized administrative structures still require a central position for coordinating communication, organizing and disseminating information, and keeping partners alert to the jointly determined rules for governing relationships—what Freitag and Winkler (2001, p. 68) describe as social coordination.” Illustrative metrics appropriate to administration could include (a) whether there is an administrative structure in place to allow agencies to identify who will coordinate, communicate, and disseminate information throughout the network, and (b) the extent of its effectiveness.
- Organization autonomy recognizes and acknowledges that collaborating organizations “maintain their own distinct identities and organizational authority separate from a collaborative identity . . . when collaboration’s goals conflict with the autonomous goals of individual partner organizations, identities are at stake . . . it is likely that individual missions will trump collaboration missions” (Thomson and Perry, 2010, p. 293). Because the individual LCCs set their own goals and priorities through a collaborative process (i.e., the steering committees), each LCC develops a distinct identity based on the joint goals of the partners. Therefore, the metrics appropriate for each individual LCC may or may not be identical. Furthermore, metrics for individual LCCs ideally assess the contributions of partners to the individual LCC’s goals, and vice versa. Thus, illustrative metrics appropriate for the LCC Network could include metrics to measure the contributions of partners.
- Mutuality is closely tied to complementarity and describes a situation where “parties to a network agree to forgo the right to pursue their own interests at the expense of others” and accommodation serves as the modus operandi of interaction (Powell, 1990, p. 303). “It occurs when one party has unique resources (e.g., skills, expertise, and money) that another party needs or could benefit from (and vice versa). As long as collaborative partners can satisfy one another’s differing interests without hurting themselves, collaboration can occur” (Wood and Gray, 1991, p. 161). Illustrative metrics appropriate for the LCC Network could include the extent to which LCC Network partners have been able to leverage contributions made by other partners in the network.
- Trust is a central component of collaboration because it reduces complexity of transactions. Developing trust takes time and needs repeated interaction among partners to build the credible commitment necessary for collective action to occur (Axelrod, 1984, 1997; Ostrom, 1990). The Comprehensive Everglades Restoration Plan and its implementation—and to some extent, the lack of its implementation—is a study in trust-based collaboration (NRC, 2014b). A challenging restoration effort is taking place in the California Bay-Delta, which also involves complex institutional arrangements requiring trust building (NRC, 2012b). Illustrative metrics appropriate for the LCC Network—reciprocity and trust—could include the extent to which trust among parties has led to reduced complexity of transactions.
Establish and Aggregate Individual LCC Metrics to Measure LCC Network-Level Goals
In presentations to the committee and public, LCC staff acknowledged difficulties scaling up from individual LCC SIASs to a network-level SIAS assessment. In addition, other
than completion of the SIAS by each LCC coordinator, most LCCs have yet to develop their own set of metrics beyond the SIAS, although many have some requirements for measuring success. For example, the strategic plans for the Eastern Tallgrass Prairie and Big Rivers LCC, the Aleutian and Bering Sea Islands LCC, and the North Pacific LCC have expressed the intent to develop metrics that mimic either the SIAS or some portions of the SIAS. In open discussion with individual LCC participants, it was clear that while metrics for individual LCCs have only been developed by a few LCCs to date, the need for such metrics is generally acknowledged as a priority by most LCCs. Annual reports include benchmarks per se, but those reviewed provided accomplishments without first identifying the need for the accomplishment to occur or including metrics to evaluate progress toward those accomplishments.
The FWS staff has mapped the individual LCCs’ goals onto the goals of the LCC Network Strategic Plan (see Appendix G). This illustrates the complexity of developing metrics for individual LCCs that can assess the effectiveness of the LCC Network as a whole. It further demonstrates the need for compatible metrics that cross all LCCs and provide some level of outcome measurement at the LCC Network level.
The committee concludes that the SIAS document provides a useful initial assessment of the FWS components of the LCC Network. Specifically, (1) the categories of performance metrics (process, outputs, and outcomes) outlined by OMB were addressed (although not labeled as such); (2) each of the eight Conservation Activity Areas guided by the SHC framework establishes progress benchmarks; (3) metrics were linked to the progress benchmarks in the SHC framework; (4) benchmarks and their accompanying measures were set at a level of aggregation that appropriately transcends the conservation issues specific to any one LCC (or subset of LCCs) and provides a common system for programmatic evaluation that is network wide; and (5) some of the metrics used a “percent complete” approach, whereas others were expressed narratively; however, the context for each was appropriate. Thus, the committee concludes that the SIAS helps the FWS meet its reporting requirement to Congress and OMB.
The committee concludes that the current approach and focus on measuring FWS investments and outcomes do not adequately measure all aspects of the LCCs and the LCC Network. In particular, the committee concludes that the evaluation process currently falls short in two important ways: SIAS metrics are not aligned with the goals of the LCC Network or with those of the individual LCCs; and the SIAS does not measure the value of the network to its partners.
Aligning SIAS Metrics and Benchmarks with Network Goals: Because the SIAS benchmarks are based on the goals in the FWS’s SHC plan, the SIAS is not designed to measure progress toward the goals in the LCC Network Strategic Plan. Also, some SIAS metrics account for efforts across multiple LCCs, but a process is missing to effectively measure various efforts at the network scale, or to aggregate results from the individual SIAS documents to the network scale. To improve the SIAS, an important short-term need will be the alignment of the next iteration of the SIAS to better reflect the mission, goals, and objectives of the strategic plan, as well as the goals and objectives of each self-directed LCC.
As discussed in Chapter 3, there are opportunities to improve upon the strategic plan, and based on discussions with FWS staff, the committee expects that a new iteration of the LCC Network plan will be forthcoming. In doing so, the FWS has to consider whether the SIAS can assess the goals of the LCCs and the LCC Network, while still serving the intended purpose of the SIAS, which states,
the FWS [(SIAS)] will help guide our support for individual Landscape Conservation Cooperatives (LCCs) and the National Landscape Conservation Cooperative Network. In pursuit of our agency’s mission and our vision for science, the following Activity Areas and associated benchmarks will help specify our investment and participation in the LCC Network to ensure effectiveness, efficiency, and support for the LCC Network vision and mission.3
We conclude that establishing solid and defensible metrics that are clearly aligned to processes, outputs, and outcomes at the individual LCC level is a key step toward creating a more understandable and defensible LCC Network as a whole. For that reason, it is important to direct LCC resources into developing sound metrics at the individual LCC level. The cost (staff time) required to complete such evaluations needs to be considered in the development of further evaluation processes and tools.
Recommendation: The FWS, in its next iteration of the SIAS, should (1) identify how and where the SIAS relates to elements of the LCC Network strategy; (2) identify (as it has done for SIAS 2.0) the benchmarks associated with each Activity Area, and continue that exercise by (a) classifying benchmarks as short, medium, or long term; (b) ensuring that benchmarks are adequately developed for and assigned to each SIAS Activity Area; and (3) begin the process of identifying, to the extent feasible and practicable, costs relative to returns on investment associated with achieving each benchmark.
Recommendation: Establishment of metrics at the individual and network-wide scales should become a high priority.
- Metrics should be developed to measure each LCC’s unique goals, yet be consistent enough across LCCs to permit aggregation to a network scale.
- The criteria used to rate the performance of the LCC Network as a whole, and its components, should be closely related to the objectives that they are intended to evaluate, and articulated clearly enough that any evaluator with access to the same information about the LCC Network could apply those criteria consistently.
- To more clearly demonstrate relevance to the stated purpose and goals of the LCC Network, as well as to better define the FWS role in support thereof, the SIAS Activity Areas and benchmarks should be written in a manner that clearly aligns with the LCC Network’s purpose and goals, as captured in the strategic plan or its next iteration.
Measuring Partner Contributions and Benefits: Although the SIAS tracks how partners contribute to the FWS-specific goals of the individual LCCs, currently the evaluation process does not yet account for how these contributions further the goals of the network in general or the goals of the key partners in particular. Thus, the current evaluation approach will not be able to account for the outcomes that result from convening partners, and it might fail to properly measure partners’ investment in or benefits from joint activities. As a result, it might lead to misinterpretation of accomplishments and/or an inability to express the value-added outcomes of the LCCs. Developing metrics for the network as a whole may best be incorporated into a frame of reference that is complementary to, but separate from, the SIAS. Because the SIAS is an FWS product that is reflective of the agency’s own goals and objectives, the LCC Network as whole may consider a framework that is complementary with, but separate from, the SIAS in order to better capture goals, objectives, and measurements toward network-wide progress. In so doing, the observations in the above section on short-, medium-, and long-term metrics will be useful and applicable. In short, the requirement for short-term “process” metrics never disappears—an organized, coherent program is necessary to justify government funding, and the process organization should carry through and become increasingly refined as the program evolves. In this sense, monitoring the extent of and continuing to strive for alignment between the SIAS and other key documents, such as the strategic plan and the individual strategic plans of the respective LCCs, should continue. Furthermore, the process of better organizing the SIAS, using the short-term metrics identified above, will likely give rise to additional medium- and long-term metrics beyond those identified below. While providing a deliberate, useful, and meaningful initial assessment tool, the SIAS document and its accompanying metrics would benefit from supplementing in ways that would refine the use of metrics to better show the returns achieved for investments made by partners and design and apply FWS-relevant metrics at the individual LCC level that can be aggregated to measure and demonstrate outcomes at the LCC Network level.
Recommendation: The LCC Network should improve its evaluation process to better capture the contributions made by all partner agencies or groups toward common objectives. In particular, to demonstrate the effectiveness of the individual LCCs and the LCC Network, the evaluation process should measure how resources invested in any portion of the LCC Network further the goals of the LCC Network and its partners. The efforts invested in the LCCs and the LCC Network consists of (1) federal funding allocation via the FWS; (2) partners’ in-kind contributions via staff time or technical expertise; and (3) funding from other state/federal agencies or private partners.
In particular, the committee recommends:
- Measure contributions and results—distinguish among process, output, and outcome metrics—and attempt to ensure that the outputs (the products and/or services rendered) and the outcomes (results achieved for those products and services) are clearly identified where feasible to do so. Specifically,
- seek to identify agency-unique contributions as well as partner contributions and the outputs/outcomes achieved for those contributions;
- seek to identify costs of agency-unique and/or partner contributions relative to the outputs/outcomes obtained for those investments; and
- continue to use both qualitative and quantitative measures to evaluate outputs/outcomes.
- Where feasible, measure collaboration, that is, governance, administration, organizational autonomy, mutuality, reciprocity, and trust.
- Where feasible, develop impact and collaboration metrics at the individual LCC level in ways that aggregate to the network level.
This page intentionally left blank.