9
PATH Forward—Program Plan and Performance Measures
WRAP-UP COMMENTS
MR. GONZALEZ: The final segment of the workshop addresses the synthesis of PATH’s three goals, nine objectives, and 53 outcomes that are presented in the draft document. One objective of the workshop is to give PATH some sense of priorities in order to maximize the impact of its limited resources. Up until now we have addressed the three goals as having equal weight, but now we will discuss possible priorities while keeping in mind what is feasible and practicable.
This discussion will be led by Dr. Melvin Mark.
DR. MARK: PATH has been extremely responsive to previous NRC recommendations. The draft strategy, operating plan, and performance measures are a direct response to what PATH heard from the NRC and others. There have been a number of comments along the lines that there is a need for focus, especially given resource constraints and a budget that is smaller now than when the planning took place. The workshop group has recognized the need to focus, but then there were discussions that pulled the possible focus in most of the directions that are presented in the draft and some additional ones. We will not try to develop a consensus, but we should be able to develop a sense of priorities and the optimum balance of a diffuse versus a focused approach.
We also need to get back to some issues of specific metrics to determine which measures are most important to PATH’s stakeholders, and what information HUD needs to manage the program. It is clear from today’s discussions that there are stakeholders with tangible interests and these interests are diverse. We are faced with the question of whether or not PATH can be all things to all people.
This discussion will start with introductory statements from Mike Chapman, president of Chapman Homes and chairman of the PATH Industry Committee; Ross Heitzmann, from NAHBRC; and Anny Wong, a political scientist at RAND and a coauthor of the Building Better Homes report.
MR. CHAPMAN: Everybody participating in this workshop is familiar with PATH and how far it has come. When PATH was first proposed most people thought it would only result in a report that would sit on a shelf to gather dust. Most people were not sure it would be worth an investment of their time. Several years were spent wrangling over which government agency would take the lead role. PATH started with goals assigned by the administration that the NRC committee found to be overly ambitious for a small program, influenced by many factors outside the purview of PATH, and very difficult to measure. The program staff worked with all of its partners to develop more appropriate goals and respond to many other suggestions to improve the program. At the same time its budget was shrinking and its supporters had to fight to keep it going. To see PATH come from that kind of beginning to develop the strategy and operating plan we are reviewing shows that if it survives, it can overcome any obstacles to its success. A key factor that will influence success will be the continued coordination of a private/public partnership.
PATH may have some lessons to learn from a HUD program in the late 1970s and early 1980s called the Joint Venture for Affordable Housing. I think it may be one of the best programs that I have ever participated in. The goal of the program was to demonstrate how innovative builders and developers could create affordable housing if they are not constrained by zoning and building codes and standards. All exceptions to the codes and standards needed to be well documented, based on sound engineering, and maintain the health and safety of the house and community. Also, as a HUD program, the availability of
FHA loans was assured.
At that time, affordability was the major concern in the housing industry. I do not think we have given up on that objective, but it is getting more difficult. My company’s demonstration project was phase one of a three-phase development in Santa Fe, New Mexico.
The exceptions to the codes and standards allowed us to cut development and building costs by more than 25 percent per house. There was an earlier reference to using innovation to save one dollar per square foot. I do not know the cost of the houses in that case, but it is probably a savings of one percent or less. The 25 percent savings was accomplished by addressing every detail such as the placement of electrical plugs and eliminating door bells and using door knockers. The trigger for action was the opportunity for saving $75 per house.
The demonstration was a tremendous success; people were waiting in line to purchase the houses. All 50 houses were sold almost immediately. Unfortunately, this success was not extended to the second phase because of political objections to affordable housing in the community. Without the regulatory exceptions, second-phase prices jumped 25 percent.
The city of Santa Fe now wants affordable housing. The city requires 30 percent of the houses in all new developments to be affordable and meet certain other requirements regarding the mix of units for different housing needs. Affordability is now achieved with a subsidy, which is about $100,000 per house. The subsidy results in a corresponding increase in the cost of the other 70 percent of the houses. Furthermore, the city maintains an ownership position in the subsidized houses that limits the equity growth of the owners. There are probably many more efficient ways to provide affordable housing. This situation is not unique to Santa Fe. PATH focuses on using new technologies to solve housing problems, but there are many other issues that builders and consumers need to deal with.
MR. HEITZMANN: Speaking for NAHBRC, the shift of PATH’s goals to focus on the process of innovation instead of its outcome was the right thing to do. The draft program plan is clearly thought out, very detailed, and focused in the three most appropriate areas: barriers, dissemination of information, and advancing R&D. Even with the limited funding, all three should be maintained as part of the PATH program. The metrics should have more differentiation by giving them weights. Some of the outcomes are more important than other outcomes.
Benchmarks for the performance measures should be consistent with the amount of program funding. PATH also needs to ensure that the measures actually measure the desired outcome. Many of the outcomes are very difficult to measure or the cost of an effective measure would be prohibitive. It is very difficult to measure changes in the rate of technology diffusion. The number of people accessing PATH Web sites and the number of documents that are down loaded can be measured, but that does not show how much difference PATH has made.
DR. WONG: The draft PATH program strategy, operating plan, and performance measures document indicates that PATH is making changes and becoming a better program. The logic model delineating the program activities and measures of the inputs, outputs, and outcomes should lead to an even more effective program.
The draft document is, overall, a good first effort, but there are some problems caused by the way certain terms are used. For example, the use of the term “innovation” is expectably pervasive. But innovation can and does refer to many different things. To begin with there are product innovations and process innovations. The various audiences that PATH is addressing approach these types of innovations in different ways. A builder might be concerned about an efficient process for installing windows, but a consumer might only be concerned about its durability or energy efficiency.
The term “barriers” is another example. Again, the perspective of the audience is very important. The barrier may affect any of the PATH participants, so there is a need to address them all on their own terms. PATH participants are all motivated by their own incentives to do the things they do, and make the choices that they make. Understanding barriers includes knowing where the barriers are in order to identify the opportunities to start changing the incentive structure.
Another concern is the term “housing.” The term is used to refer to a wide array of possible audiences, often without distinguishing their characteristics. The concerns of first-time home buyers are
very different from those of luxury home buyers, and the needs and concerns of people building or buying single-family homes are very different from those involved with multifamily dwellings. New housing issues differ from those of existing housing stock.
The amount of funding allocated to PATH cannot adequately support all of the goals and activities listed in the plan. The program would be difficult to support even with significant contributions from PATH partners. The list of activities in the draft is an excellent start, but it is more of a menu than a plan. The plan needs to also have priorities and show how those priorities are likely to change over time. It may be more effective to address all of the goals by focusing on each of them consecutively rather than addressing them all at the same time.
Performance measures are important but the cost of collecting and maintaining performance data should be a key consideration in determining which metrics to use. The number of proposed activities and measures magnifies this problem. There should also be consideration of a centralized database to support the information needed by management as well as support the reporting and oversight functions. As mentioned earlier, practicability should be the first criterion for performance measures.
The discussion question should be revised from: Are the goals realistic? to, Are the priorities realistic? I agree that research at universities is important. Basic research is important; however, that research should be applicable to the needs of PATH’s partners. It is the responsibility of the government to fund the growth of a knowledge base as a public good. Part of this knowledge base is the university system that also trains architects, engineers, and homebuilders who will be charged with achieving PATH’s goals.
The value of research in academic institutions or industry is not a matter of basic or applied research. The issue should be whether the research is mission driven. PATH as a government program has the role of a convening authority. It needs to bring people together to define problems, and then create the opportunities to initiate research to solve those problems.
It is generally more effective to advance technology by using demand to pull than to use R&D to push. This is due to the way the housing market works. This puts a priority on outreach and information dissemination and research that identifies current problems. PATH should consider working with the media and trade organizations to let them know that PATH is a resource for housing innovation information. PATH can be an effective source for reporters as well as provide information directly to end users.
The original PATH goals of durability, energy efficiency, environmental soundness, and so forth, are consistent with the values of home buyers and homebuilders. They are not effective performance measures, but they are a connection with PATH’s audiences and should not be lost. The values, mission, and goals should not change over time. However, priorities should change as conditions and specific problems change from year to year.
DISCUSSION
DR. MARK: I want to thank the members of the panel and start the workshop discussion. To start, does the draft program plan need to have fewer activities or at least specify priorities? One way of conceptualizing priorities is in terms of scheduling. This is important, because communicating to OMB that PATH is going to accomplish all of these objectives without clear priorities may be a recipe for failure.
DR. O’BRIEN: FIATECH has a capital projects technology roadmap, which is not exactly analogous to an operating plan, but it is a research plan for the future that goes beyond what FIATECH itself hopes to accomplish. The roadmap helps to create a centralized agenda that other organizations can use to coordinate their activities with FIATECH. There is also value in just articulating a broader program.
DR. MARK: That is often called an aspirational model. Such models have real value, but they may not be useful as plans for success.
DR. JACKSON: The FIATECH roadmap is an aspirational model. It is a vision of the future, as well as a description of where the organization is at present. Instead of 161 projects laid out against a seven year time frame, it defines what the organization wants to be in seven years. The roadmap includes a vision, a starting point, an ending point, and a path to get from here to there.
FIATECH expects to achieve that vision even though it does not have the resources. It is not, nor should it be, the only organization that will contribute to the realization of the vision. The roadmap is a document we use to communicate with industry and other organizations to help coordinate activities and find opportunities for collaborations.
The roadmap is a research, development, and deployment agenda to achieve a vision. For example, when you want to build something, you have a plan for what you want to do. You know in what time you are going to get it done and you know what resources you need to do it.
DR. VANEGAS: It would be a disservice to eliminate things from the PATH operating plan that diminish the overall vision. In order to take this approach there needs to be vision that defines priorities and a focus that communicates those priorities.
DR. SLAUGHTER: Aspirational roadmaps and operating plans are very different. This workshop is about a plan that OMB is going to review to determine if PATH is an appropriate use of government funds. This is very different from a document for coordinating activities of disparate organizations. OMB is not concerned with whether or not PATH has a vision. The draft document is very important to the future of PATH. Its specificity, achievability, and appropriateness are critical. It is not a visioning document. It is akin to a contract between PATH, the administration, and Congress.
MR. KASTARLAK: PATH should not be viewed as a panacea. It was never intended to be. It was simply the prescription for innovation in the housing industry. But we have to also keep in mind that innovation is not an end in itself. PATH also needs to have a social purpose and social benefit. The program is based on the assumption that more innovation and better products will improve housing. That is a big assumption.
It is also possible for PATH to have more than one plan—one to use for its contract with OMB and the other to use as an aspirational model to communicate with its various audiences and stakeholders. PATH developed an elaborate plan for improving research and increasing the development and diffusion of innovative technology. Keep that as a reference. Keep it in sight. But then present a plan for how it should be done with the resources that are provided.
MR. ENGEL: Setting priorities for PATH is important, but very difficult. From the discussion today, I do not think this workshop could achieve a consensus. The needs are so great for all three goals that there is a case for each of them being a priority. Priorities cannot be determined by a vote; the outcome will depend on who is voting. It has to be based on a sound intellectual analysis and resource management.
DR. MARK: Nevertheless, a straw vote could provide some useful information. Each participant has two votes that can be used to identify two priority goals or to identify the importance of one. This is important because there was discussion earlier about the reaction to the research goal being eliminated. [There was a show of hands to indicate priority preference for each goal.] There was some difference but it does not appear to be significant.
Since we have established that all three goals need to be addressed, are there any linkages or crosscutting issues that can help set priorities? Is there an integrated set of activities that can address core concerns and core stakeholders?
MR. SPEAR: The NRC 2002 study noted that it was important, just as important as these three goals, that PATH have effective program administration. This is addressed at the end of the draft background document, but has not been addressed in the operating plan and performance measures. Effective administration and predicable funding commitments are crosscutting issues that affect all the components of the mission. Without effective program administration the goals and activities are merely hypothetical. The limited funding that PATH has been allocated places greater emphasis on collaborative arrangements with other government and private organizations. The absence of effective administration and predicable funding is a disincentive for partners, who need to know that their contributions will have
a reasonable chance for a useful outcome.
MR. CHAPMAN: The characteristics of the housing industry (size, cyclical nature, diversity, complexity of supply chain, and so forth) make it very difficult to measure incremental changes and the effects of specific factors. Performance measurement of the housing industry is not nearly as straightforward as a typical manufacturing process that has more control of production goals and processes. Chances are that PATH’s goals and metrics will never be as defined as we want them to be.
Previous discussion of prioritization suggests that PATH will continue to work on all of its goals simultaneously. The scheduling of activities will be dictated more by opportunities to leverage PATH’s efforts than by an intellectual assessment of the program. Plans are important, but flexibility that can respond to opportunity can be a larger factor in determining success.
PATH has done a great job in terms of its ability to leverage activities. A lot of the effort of the Industry Committee has focused on creating opportunities for PATH to leverage its investments. But it is a complex process that is neither linear nor direct.
MR. KASTARLAK: The current plan seems to represent three simultaneous linear processes without any linkages. In reality, it is not going to happen that way. It may be necessary to develop a model of how the activities will work together over time.
DR. MARTIN: There was an earlier version of the chart that indicated the linkages between activities, but it became too complex and impossible to communicate.
DR. MARK: That is a common problem for performance measurement and logic modeling. As mentioned earlier, PATH may need several versions of its operating plan for different audiences: a simple, straightforward version for external audiences and a more complicated version for internal purposes
MR. ASDAL: The objective of this exercise is to communicate that PATH has created value for its $5 million funding so that the people who control the budget will fund the program in the future. PATH spent a lot of time in 2001, 2002, and 2003 building technology roadmaps. But these roadmaps do not communicate what PATH has accomplished or the progress toward its goals. PATH has sponsored research on cement panels and whole-house remodeling that have provided valuable information, but there is no way to show their contributions toward the program objectives. The roadmap should provide a means of measuring progress. It should also be able to show the best routes as priorities and the activities needed to get from point A to point B, starting with basic research, to demonstration projects, and diffusion in the market. There may be new projects that do not fit on PATH’s map that need to be delegated to others.
MR. EMRATH: I want to return to the earlier discussion of the difficulty of measuring PATH’s effect on the housing industry. It is easy to track how many people visit a Web site, but there is a quantum leap between that and tracking diffusion of innovation in housing. PATH is not unique in this respect. Other HUD programs have faced similar problems in developing performance metrics. Other programs have been allowed to settle for less than perfect measures. The overseers, administrators, and stakeholders got together to identify the measures that were practicable. PATH has a $5 million budget, which is not enough to do involved assessments such as surveys. Many surveys can cost more than $5 million. Perhaps determining performance measures for PATH will require some negotiation.
MR. ENGEL: I think that point is well taken. Even if an outcome is measurable, the cost of applying the measure may not be justified. The size of PATH’s budget is certainly a limit on what can be measured. There are many beneficiaries of PATH who would testify that the program has made a difference, but that difference cannot be cost-effectively measured. Should OMB consider the notion that the industry thinks it made a difference to be an acceptable outcome measure?
MR. WEBER: Many industry partners have performance data to justify their expenditures. PCA conducts homebuilder and homeowner surveys. There may be a way to show how PATH influences the outcomes of those surveys.
MS. SHIPMAN: Performance measures are secondary to the strategic plan. The strategic plan is about where the program should be going, and where the opportunities and barriers are. The annual plan and metrics are more pedestrian. The measured outcomes should have a direct relationship beyond the
intermediate goal and have a link to the mission.
DR. SLAUGHTER: PATH performance measures need to relate to the values of the people who are determining its budget. PATH is contributing to creating safe shelter for everybody. It is not about innovation for innovation’s sake. It will be difficult to get people passionate about a reduction in the rate of diffusion or a reduced barrier to the adoption of a new technology.
If PATH wants to have a champion and a population that turns to PATH for solutions, then it needs to identify and address the critical issues that are important to that population. The measures need to show if PATH’s activities provide immediate benefits to homeowners and homebuilders.
DR. MARK: There have been a number of comments that take this discussion back to the issue of alignment with mission.
MR. HATTIS: We heard that PATH does not have enough money to both develop an effective program and implement effective performance measures. There is general agreement, at least among the workshop participants, that PATH does a good job of allocating its $5 million to a variety of contracts. The people who are in the best position to measure the results of this effort are the contractors, who know more about what they have been doing than anyone else does.
Instead of having a separate track for developing performance measures, it can be incorporated into each activity. It may add to the cost of activities, but that approach is probably more efficient than a separate activity. Contractors can propose performance measures that assess their work and provide performance data as part of the contract deliverables. It would be part of their scope of work. That will help contactors manage their activities as well as help PATH manage its programs. If the contractor has subcontractors they could apply the same approach. PATH could provide guidance and help to develop a consistent methodology for all contractors to use by identifying the performance measures that are most cost-effective.
DR. MARK: There are a number of workshop participants who could help guide a contractor-based performance measurement system.
DR. WONG: The idea of the annual plan needing to be coherent with the strategic plan is important. The three goals are part of the strategic plan. Priorities should be part of the annual plan that changes over time. Both plans, and the interrelationship of activities, need to be presented, but they do not need to be in the same document.
The draft document presents a full menu of what can be done. It can be used to convey this message to OMB or any other interested party. PATH then needs to present its priorities and schedule for accomplishing its established goals.
The concept of having contractors provide performance data is appealing, but PATH needs to take an active role in defining the metrics to ensure that they are valid and not susceptible to gaming. It can be a collaborative process.
MR. ASDAL: It would be easier for PATH to develop program performance measures and develop a branding program if there were a housing rating system to assess housing performance criteria other than energy efficiency, such as a durability rating system and a value rating system. Consumers do not know what a good house looks like. NAHB publishes square footage costs and the number of housing starts. The government tells us about homeownership rates. The realtors will tell us the average price of housing. But nobody is assessing the quality or telling the consumer how to recognize a good house. A simple rating system that anyone can understand could be used by insurance underwriters, bankers, and tax assessors, and tied to incentives. If people got behind building better houses and consumers knew what one looked like, not just higher ceilings, more bathtubs, more square footage, or more density, then American homes would quickly improve.
The components of a unified house rating system could include measures such as moisture control, energy efficiency, durability, maintenance cycles, and operating costs. The various criteria could be weighted and combined into a single number that represents a house value. The consumers can know that if a builder is producing houses that are rated 84 it is better than others that are rated 67, and they can compare both cost and value. If a remodeler can take a house rated 64 and increase its rating to 72, he or she can show the added value for the completed work.
DR. MARK: I want to thank all workshop participants for a very constructive and thoughtful discussion. I would also like to thank the panelists for their contributions.
MR. GONZALEZ: It may have sounded like there was a lot of conversation and no real definitive answers, but we have provided PATH feedback on its draft strategy, operating plan, and performance measures that PATH can use to complete its plans. There were a lot of wonderful ideas put forward that will be captured in the workshop report to help create future opportunities.
Just because the workshop has concluded does not mean the dialogue needs to end. If anyone has additional ideas or suggestions please send them in writing to the NRC or directly to PATH.
Thank you for a very productive workshop.
