Evaluating Research Efficiency in the U.S. Environmental Protection Agency (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix C Program Assessment Rating Tool (PART) Questions1 OFFICE OF MANAGEMENT AND BUDGET Section I. Program Purpose and Design 1.1: Is the program purpose clear? 1.2: Does the program address a specific and existing problem, interest, or need? 1.3: Is the program designed so that it is not redundant or duplicative of any other Federal, State, local or private effort? 1.4: Is the program design free of major flaws that would limit the pro- gram’s effectiveness or efficiency? 1.5: Is the program design effectively targeted so that resources will ad- dress the program’s purpose directly and will reach intended beneficiaries? Section II. Strategic Planning 2.1: Does the program have a limited number of specific long-term per- formance measures that focus on outcomes and meaningfully reflect the purpose of the program? 2.2: Does the program have ambitious targets and timeframes for its long- term measures? 1 OMB (Office of Management and Budget). 2007. Program Assessment Rating Tool Guidance No. 2007-02. Washington, DC. 86 

Appendix C 87 2.3: Does the program have a limited number of specific annual perform- ance measures that can demonstrate progress toward achieving the program’s long-term goals? 2.4: Does the program have baselines and ambitious targets for its annual measures? 2.5: Do all partners (including grantees, sub-grantees, contractors, cost- sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program? 2.6: Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need? 2.7: Are budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program’s budget? 2.8: Has the program taken meaningful steps to correct its strategic plan- ning deficiencies? Specific Strategic Planning Questions by Program Type 2.RG1: Are all regulations issued by the program/agency necessary to meet the stated goals of the program, and do all regulations clearly indicate how the rules contribute to achievement of the goals? (Regulatory) 2.CA1: Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity? (Capital Assets and Service Acquisition) 2.RD1: If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals? (R&D) 2.RD2: Does the program use a prioritization process to guide budget re- quests and funding decisions? (R&D) Section III. Program Management 3.1: Does the agency regularly collect timely and credible performance in- formation, including information from key program partners, and use it to man- age the program and improve performance? 3.2: Are Federal managers and program partners (including grantees, sub- grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results? 3.3: Are funds (Federal and partners’) obligated in a timely manner, spent for the intended purpose, and accurately reported? 

88 Evaluating Research Efficiency in EPA 3.4: Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution? 3.5: Does the program collaborate and coordinate effectively with related programs? 3.6: Does the program use strong financial management practices? 3.7: Has the program taken meaningful steps to address its management deficiencies? Specific Program Management Questions by Program Type 3.CO1: Are grants awarded based on a clear competitive process that in- cludes a qualified assessment of merit? (Competitive Grants) 3.CO2: Does the program have oversight practices that provide sufficient knowledge of grantee activities? (Competitive Grants) 3.CO3: Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful man- ner? (Competitive Grants) 3.BF1: Does the program have oversight practices that provide sufficient knowledge of grantee activities? (Block/Formula Grant) 3.BF2: Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful man- ner? (Block/Formula Grant) 3.RG1: Did the program seek and take into account the views of all af- fected parties (e.g., consumers; large and small businesses; State, local and tribal governments; beneficiaries; and the general public) when developing significant regulations? (Regulatory) 3.RG2: Did the program prepare adequate regulatory impact analyses if required by Executive Order 12866, regulatory flexibility analyses if required by the Regulatory Flexibility Act and SBREFA, and cost-benefit analyses if re- quired under the Unfunded Mandates Reform Act; and did those analyses com- ply with OMB guidelines? (Regulatory) 3.RG3: Does the program systematically review its current regulations to ensure consistency among all regulations in accomplishing program goals? (Regulatory) 3.RG4: Are the regulations designed to achieve program goals, to the ex- tent practicable, by maximizing the net benefits of its regulatory activity? (Regu- latory) 3.CA1: Is the program managed by maintaining clearly defined deliver- ables, capability/performance characteristics, and appropriate, credible cost and schedule goals? (Capital Assets and Service Acquisition) 3.CR1: Is the program managed on an ongoing basis to assure credit qual- ity remains sound, collections and disbursements are timely, and reporting re- quirements are fulfilled? (Credit) 

Appendix C 89 3.CR2: Do the program’s credit models adequately provide reliable, con- sistent, accurate and transparent estimates of costs and the risk to the Govern- ment? (Credit) 3.RD1: For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain pro- gram quality? (R&D) Section IV. Program Results/Accountability 4.1: Has the program demonstrated adequate progress in achieving its long-term performance goals? 4.2: Does the program (including program partners) achieve its annual performance goals? 4.3: Does the program demonstrate improved efficiencies or cost effec- tiveness in achieving program goals each year? 4.4: Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals? 4.5: Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results? Specific Results Questions by Program Type 4.RG1: Were programmatic goals (and benefits) achieved at the least in- cremental societal cost and did the program maximize net benefits? (Regulatory) 4.CA1: Were program goals achieved within budgeted costs and estab- lished schedules? (Capital Assets and Service Acquisition)