Promoting Innovation: 2002 Assessment of the Partnership for Advancing Technology in Housing (2003)

Ideally, an assessment process should have baseline measures and evaluation questions that judge the efficacy of a program. Where there are not previous measurements, baseline performance targets need to be based on experience and expertise. The committee believes that future evaluative work should address the following questions.

The committee drafted the following questions as a starting point for future assessment. It is expected that questions would be applied as appropriate to specific activities and revised as additional data become available. This set of questions addresses PATH administrative issues.

Process: Is the PATH process comprehensive, complete, and effective for its intended purpose?

Scope: Do the activities include a representative mix (size, location, housing type, housing cost) of housing projects? Do the activities include a representative mix (systems, materials, costs) of technologies?

Structure: Is PATH structured to achieve its mission and meet its goals?

Participants: Do the participants represent an appropriate mix of stakeholders?

Documentation: How well are the results of the activities documented?

Dissemination: How likely is it that the information from the activities will reach a broad range of stakeholders and housing segments?

Content quality: Is the information accurate, credible, objective, current, and comprehensive?

Graphic quality: Does the graphic presentation enhance dissemination of the information?

Focus: Are the content and format appropriate for the intended audiences?

Linkages: Are links to other documents or Web sites appropriate to the topic, helpful, and accurate?

Annual planning: Are PATH annual planning and resource allocation effective for meeting its goals and mission objectives?

Long-range planning: Are PATH long-range planning and resource allocation effective for meeting its goals and mission objectives?

Program evaluation: Is the PATH evaluation process effective?

The following questions are applied as appropriate to the administrative and support activities that deal directly with external stakeholders and audiences.

Communications: Is PATH maintaining communications with all partners?

Recognition: Is a broad base of stakeholders aware of the PATH mission? Is a broad base of stakeholders aware of PATH accomplishments?

Marketing: Are PATH efforts effective in increasing the depth and breadth of awareness of PATH and its accomplishments?

Partnerships: Is PATH creating and maintaining effective partnerships with industry (manufacturers, researchers, marketers, builders, tradesmen, architects, engineers, fanciers, insurance professionals, realtors, inspectors, and appraisers), government at all levels (researchers, administrators, and regulators from federal, state, and local agencies), and academic institutions (researchers and educators) in both regular academic programs and extension services?

The following questions are applied as appropriate to all activities to assess progress toward achieving the program’s goals, with an emphasis on assessing programmatic outcomes.

New knowledge: Is the activity likely to produce new knowledge?

Rate of diffusion: How likely is it that the activity will affect the rate of diffusion of new technologies?

Barrier reduction: How likely is it that the activity will reduce barriers (regulatory, market, industry) to adoption of new technologies?

Rate of innovation: How likely is it that the activity will directly or indirectly increase the rate of development of new technologies?

Housing performance: Is the activity likely to contribute to an improvement in performance of one or more housing characteristics (affordability, sustainability, durability, or safety)?

The following questions are applied as appropriate to assess progress toward achieving specific PATH goals.

Program activities: Is PATH implementing an appropriate variety of activities (subject, size, issue, stakeholders) given its total funding?

Goal 1: Is PATH making progress toward barrier reduction—Will its activities reduce barriers (regulatory, market, industry) to adoption of new technologies?

Goal 2: Is PATH making progress toward improving technology transfer, development, and adoption through information dissemination?

Goal 3: Is PATH making progress toward advancing housing technologies research and fostering development of new technology?

Goal 4: Is PATH providing administrative support that makes it possible for the program activities to make progress toward achieving PATH’s goals?

Overall performance: Considering the inputs, outputs, and performance assessments, does the PATH program achieve its mission, goals, and objectives, enhance the development and diffusion of technologies, and improve housing performance?

For each evaluation question there should be performance targets. Ideally these targets would be derived from baseline data of the outcomes and expected output of the activities. The targets should define a range of performance from unacceptable to acceptable and the highest expected level of performance. Because this is a new program and new assessment process, interim targets will need to be established and then refined as more data become available.

Some performance targets can be easily quantified, such as the number and variety of technologies in a demonstration, or the number and variety of housing types or geographic locations in the demonstration program. Checklists can be used to assess how thoroughly an activity has been executed, but some targets will require subjective evaluation of quality. Often the performance target will need to be defined by the activity’s planned objectives, with an evaluation of how well these objectives were met.

Some activities will require extensive effort to collect the data needed to answer assessment questions and determine the program outcome. Some data can be obtained from existing sources, but additional surveys using questionnaires and interviews will be needed to fully assess the PATH program and provide direction for future improvements.