Proceedings of a Workshop to Review PATH Strategy, Operating Plan, and Performance Measures (2006)

Presented in this appendix are the post-workshop comments of several of the participants, supplied here for the additional perspectives they provide.

Manuel Gonzalez

KTGY Group, Inc.

I have summarized my observations as follows:

• Although refinements are needed, the draft PATH Review and Strategy, Performance Measures, and Operating Plan is essentially a good plan for managing the program. It responds to the findings and recommendations of earlier NRC reports and suggestions from stakeholders for improving the program. The main concern is that it contains too much relative to the program’s current funding.

• The three program goals are interrelated, but this does not seem to be visible in the operating plan or measures.

• The plan has a valid logic model, but the performance measures need additional consideration to ensure that they are valid, causally connected, and practicable. Considerations should be given to incorporating performance measures into requirements for all contracts and grants. This will require care in selecting appropriate measures and validating them. Budget limitations may require acceptance of less than ideal measures. Anecdotal comments could be valid measures if their objectivity can be validated.

• The operating plan is aspirational in that it goes beyond the current level of funding. Aspirational plans are good for long-range planning but may cause unrealistic expectations if combined with short-term plans. Two separate plans may be necessary. Further, the observations that follow need to be considered and weighed in light of the need to more narrowly focus the short-term operating and evaluation plans.

• PATH needs a plan to become a sustainable program. The Construction Industry Institute and FIATECH were cited as possible models.

• PATH could be considered a virtual housing laboratory and serve as a portal to programs sponsored by PATH and others.

• The barriers to innovation are real, and a program, such as PATH, is needed to address them.

• There is a need for basic and non-proprietary applied research in housing. Undertaking this research in academic institutions leverages PATH funding and provides educational opportunities that further PATH goals.

• PATH has a better opportunity to achieve its mission by combining the forces of technology push and demand pull.

• Dissemination of information could follow the supply chain, with outreach activities targeted to the various participants, including researchers, manufacturers, distributors, architects, engineers, builders (both large and small), code officials, consumers, financiers, insurers, and others.

• The Concept Home, Web sites, trade show participation, and magazine and newsletter articles have been effective and could be expanded especially in areas that address consumers.

David Conover

International Code Council

My observations are as follows:

• The PATH program is placing increased and significant emphasis on addressing technology acceptance and barriers to innovation—something recommended by the National Research Council in its review of the PATH program in 2002.

• The PATH program is promoting innovation that will provide for the public good as long as it can help facilitate technology acceptance and in doing so provide for better and more affordable housing for the public.

• Innovation must be considered from the standpoint of the consumer because the consumer is a key driver in technology acceptance. To support their driving innovation, ways to measure and express innovation performance need to be available to them.

• Innovation includes technology as well as processes.

• Standards and conformity assessment can foster acceptance of new technology and must be included in programs to address technology acceptance.

• The Internet and electronic means of making information available provide a formidable vehicle for communication, which in turn can support technology acceptance.

• Risk is a barrier to innovation that must be addressed, recognizing that it presents itself differently to each of the entities involved in the building process.

• The task of addressing barriers and facilitating technology acceptance is formidable and can be more easily addressed by larger participants in the process rather than smaller entities that may need assistance to create a level playing field.

• All actors in the technology supply chain need information so they can take appropriate action to address more timely acceptance in areas of the supply chain they control.

• The building regulatory system, while typically viewed as a barrier, can very effectively work as a catalyst for acceptance of innovation.

• Designers support development and implementation of systems that will facilitate electronic submittal of construction documents and their automated review and approval as a means of reducing the time for regulatory approval and increasing uniformity and reliability.

• There are limited housing technology research sources, and PATH is one of those sources and should continue to partner with industry to address research on building technology.

Paul Emrath

National Association of Home Builders

Here are my thoughts on main issues from the workshop to emphasize:

1. As the NRC concluded in 2002, PATH’s original mission was inappropriate, primarily because it was too ambitious given its resources. Nevertheless, there are many barriers to innovation in residential construction (a diffuse set of buyers who often appear more concerned about appearance than performance, unwillingness of insurers or appraisers to reward improved performance, a relatively unskilled and untrained work force, etc.), so there is an important role for PATH to play.

2. One significant role for PATH is to understand what motivates all the players in the process. For example, why do so many believe suppliers are a bottleneck to innovation? Why do architects feel they have little control? Why do none of the purchasing agents for the large publicly traded homebuilders care about innovation? Why do local governments resist proven technologies that allow homes to be built at lower cost?

3. There was widespread agreement among workshop attendees that the current, revised PATH goals—removing barriers and facilitating technology development and adoption, improving technology transfer, development, and adoption through information dissemination, and advancing housing technologies research and fostering development of new technology—are appropriate and demonstrate that the PATH staff at HUD understood the NRC’s criticism and effectively implemented its recommendations. To the extent there was any dissent, it involved speculation that even the new goals may be too ambitious, given PATH’s limited budget.

4. Metrics used to measure how well PATH is achieving its goals must be cost-effective. PATH could easily use up its entire, limited budget attempting to measure its own performance. This led to discussions of possible low-cost metrics, such as feedback from users of pathnet.org obtained directly through the Web site.

Bulent I. Kastarlak

AIA Emeritus

Chair, AIA Housing Network for Technology Research

AIA Housing Committee chair Ed Hord asked me to represent the housing committee at the subject workshop. I was honored and obliged. The following reflects my impressions and notes from the workshop.

The meeting was well organized. It has achieved its intended results through the collective efforts of many persons. I would like to mention in particular Michael Cohn (National Academies), Carlos Martin (PATH), Manny Gonzales (AIA), and John Spear (AIA). Their dedication to the cause made the workshop a success. The turnout was very good. There were about 60 participants from government, the private sector, and academia. The objective of the workshop was to review the PATH strategy, operating plan, and performance measures in light of the 2002 assessment of PATH by the National Research Council (NRC). The two subject documents were:

• Promoting Innovation: 2002 Assessment of the Partnership for Advancing Technology in Housing (PATH)

• PATH Program Review & Strategy, Performance Metrics and Operating Plan for 2005-2010.

After proper introductions, keynote speaker Dr. Sarah Slaughter, formerly an associate professor of civil engineering at MIT, now the founder of a private consulting group MOCA Systems, spoke on the value of technological innovation in home construction and the role of government/industry partnership in promoting innovation. Her presentation was eloquent and informative.

Next, Dr. Melvin Mark, professor of psychology and senior scientist at the Institute for Policy Research and Evaluation at Pennsylvania State University, spoke on the subject of defining success and performance measures for the evaluation and management of PATH. His presentation was heavily theoretical.

Three discussion groups (A, B, C) followed. They considered the following subjects and answered the following questions pertaining to the PATH document:

• Is the goal of PATH for its actions communicated accurately?

• How important is the goal to the development and diffusion of new housing technology?

• Who is the audience and how do they define success?

• Are there performance measures to measure success?

1. John Spear led the first group. I participated as one of the four discussion leaders on the subject of removing barriers and facilitating technology development and adoption.

2. Manny Gonzales led the second group that discussed improving technology transfer and adoption through information dissemination.

3. Jorge Vanegas led the third group’s discussions on the subject of advancing housing technologies research and fostering development of new technology.

Finally, the conference wrap-up session was led by four other panel members and reviewed comments on the PATH program and performance measures. The discussion focused on the feasibility of specific program items. The following topics were discussed:

• Is the draft strategy of PATH and operating plan the right paradigm?

• How do we define programmatic success?

• How are priorities established and what should they be?

• What are realistic goals given the current funding level?

• How could these goals change if the funding level changed?

• What are possible strategies for the dissemination of this information to interested parties in the homebuilding industry?

1. The purpose of the workshop was to assist PATH in making improvements in its operating plan that would be necessary for satisfying the requirements of the Office of Budget and Management (OMB) for continued funding of the program. If I am not mistaken, there was no one from OMB attending the workshop. The validity and relevance of OMB’s evaluation standards and procedures were not discussed.

2. PATH’s operating plan was prepared in response to the analysis and recommendations of the National Research Council (NRC) in Promoting Innovation: 2002 Assessment of the PATH Program. I read the NRC report and took extensive notes. I do not agree with everything it said about PATH’s mission. However, I think that PATH was put in an intellectual straight jacket preventing it from engaging in broader, and more relevant, research for solving the housing problem in the USA. Instead the NRC assumed that “innovation in housing technology will realize social benefits” and will help solve the housing problem. By force PATH accepted this assumption. Nevertheless, doubts about pursuing innovation for innovation’s sake, without resulting in “social benefits,” remained. The representatives of

the homebuilding industry strongly objected to technological innovation that would not have “cost reduction and profit enhancement” as its main objective. I shared this opinion.

3. PATH’s operating plan and metrics were organized as a matrix formed by a vertical series of “Goals,” “Objectives,” and “Outcomes” and a horizontal series of “Inputs,” “Activity,” “Output,” and “Outcomes” (short and long term) that would evolve from each “Objective.” Because many work elements were not measurable, cause-effect associations among them were not self-evident. This presented a difficult problem for PATH in satisfying the OMB requirements. Associations were presented as three parallel series of work elements, each element derived from the higher work element. Every parallel series was independent, without connections with one another. This graphic format was misleading and simplistic.

4. The workshop was confined to PATH’s strategy for promoting innovation in housing research. External issues pertaining to solving the housing problem in the USA were not on the agenda. Nevertheless, the workshop often sidetracked from the structured agenda to these external issues and brought a sense of reality to discussions. Very interesting and productive exchanges of ideas took place when discussions drifted to these external issues. In particular, representatives from the housing industry deflated the importance of certain performance standards used in evaluating the PATH operating plan. They were pragmatic. They said that innovation in housing technology is relevant to builders if it helps reduce costs and increase profits. In their opinion, all other motivations were incidental. I agreed, and advocated “cost/profit” as the principal performance standard for evaluating the PATH operating plan.

5. I reviewed both the 2002 NRC report and PATH’s operating plan for 2005-2010. They are well documented and clearly written. Although I do not fully agree with all the statements included in the documents, I have not prepared a critique for either. The direction of PATH is set for the foreseeable future and commitments to the course have been made. I assumed that there is no practical benefit in criticizing the past.

6. I reviewed the “Metrics Map” for action in the PATH operating plan for 2005-2010. A total of 256 work items are listed for achieving the four goals of the plan. This corresponds to an average of 50 work items per year for five years. To implement this work program, PATH received only $5 million for FY 2006 from the federal government. Most of this money will be used by PATH for leveraging actions in the private sector for removing barriers, improving technology transfer, and advancing housing technology research. Therefore, PATH will have to operate with an average budget of $100,000 per work item during 2006. By any measure, this is a laughably (!) small amount of money for getting the job done.

7. The reality of the PATH budget dictates that considerable reduction is necessary in the operating plan. How this reduction could be achieved was the subject of more discussions. Workshop participants offered several solutions.

   1. One solution was to eliminate one of the three goals altogether from consideration—(a) remove barriers and facilitate technology development and adoption; (b) improve technology transfer, development, and adoption through information dissemination; and (c) advance housing technologies’ research and foster development of new technology.

   2. A second solution was to apply “reverse logic” to the sequence of work elements in the plan. The sequence was to start by establishing a budget for the work item first and to work backward from “outcome” expected (short and long term), to “output”; to “activity”; to “input”; and finally to realistic achievable “Goals” and “Objectives.” This approach promised to maintain all work elements by scaling them down and fitting them into the allocated budget.

   3. A third solution was to prioritize work elements randomly and implement as many elements as the budget would allow.

   4. A fourth solution was to eliminate the elements of technological innovation affecting the high end of the housing market altogether and dedicate the resources of PATH to innovation in “affordable housing” exclusively.

   5. A fifth solution was to follow the “money trail” and eliminate work items that would not result in cost reduction and profit enhancement for the housing industry.

8. In several presentations by speakers and in PATH’s operating plan (Work Item I.A) the “Architects’ role in Innovation” was presented almost as an afterthought. The plan originally allocated only $150,000 for this one work item in the budget. That role remains to be defined. Beyond that one work item in the plan, there is no other work specified for architects. This lapse of good judgment suggests that the American Institute of Architects should be more involved and assertive in promoting the importance of its members’ role in the PATH operating plan.

9. Information dissemination was a recurring topic. It became apparent that PATH was assuming a “passive” role and taking the stance of “I am here; you find me.” As a result, many participants have not even heard of PATH’s existence until they were invited to the workshop. A large number of AIA member architects were in this category. I know for a fact that the AIA Palm Beach County chapter in Florida was not aware of PATH. Workshop participants strongly suggested that PATH should take a more active role in making its existence known to the housing industry and others by adopting the motto, “You are there; I will find you.” Since government agencies, by law, cannot advertise, PATH will have to find the proper medium for communicating with its housing constituents.

10. The merits of “radical innovation,” as opposed to “incremental innovation,” were hotly debated. For some advocates, the “dysfunctional” (!) housing industry thrived on “incremental innovation” despite the educational and training deficiencies of its work force. Small inventions were the norm. But these did not appreciably alter the housing product, whereas, “radical innovation” advanced by others advocated major systemic changes in housing production and in its delivery methods. Manufactured housing and changes in professional practice by architects (see Design 21 by Bulent I. Kastarlak) offered promise.

11. The consensus of the workshop participants was that there is no pressing demand for radical innovation in housing technology at this time. When the industry is making good money there is no motivation for innovation, particularly at the low end of the housing market. Everyone agreed that the high-end housing market is being well served regardless of cost factor. But the “affordable housing” market is hard-pressed for technological innovation and cost reduction. The answer for some was to strive for “attainable” housing, meaning making incremental innovations in the product and in the process of housing production provided that the cost of the product and speed of the process represent improvements over what exists now.

12. Administering innovation in housing technology could also benefit from a similar research/innovation/licensing model used by the U.S. Food and Drug Administration. Every food and drug item and medical procedure goes through an R&D process administered by the private sector and culminates with a license, or rejection, by the government before it reaches the market. This model is worth exploring for PATH.

Theodore Koebel

Virginia Polytechnic Institute and State University

Overall, I think PATH has moved in the direction that I suggested in my article quoted by the NRC report. And while I’m gratified that the committee was influenced by my work, I also want to emphasize that I was looking at technology diffusion through one particular lens in that article, that of diffusion theory and research. One of the benefits of the PATH program to me personally is that I now have a greater appreciation of the complexity of the multi-faceted processes that lead to technological advancement in this industry, as well as a greater appreciation of how little we know about that complexity. This is also true for our understanding of residential building construction, the systems complexity of which has been documented in our process modeling in the Industrializing the Construction Site projects.

PATH has ambitious goals; some might argue they have been too ambitious. But I think we know more today from facing the challenges of those goals and falling short of attaining them than we would have if PATH pursued a more modest agenda. Would PATH be better if we had clearer, surer

models of technology development, technology testing, commercialization, and diffusion? Of course. But we don’t have those models. So some stumbling and false starts should be expected. And much to its credit, PATH has learned from those and has refined its approach.

PATH and NSF-PATH are doing more to advance our understanding of technology diffusion in the residential industry than any pervious effort. PATH is helping establish the fundamental knowledge required to develop clearer, surer models to successfully introduce new technologies into housing. The PATH work plan addresses key areas where we first need to know more before we can do more. These include identifying impediments to innovation and diffusion, modeling the commercialization process and developing commercialization tools, the role of the supply chain in technology diffusion, identifying the networks that advance diffusion, and modeling the diffusion process. This work will lead to a more fundamental understanding of diffusion in this industry, an understanding based on documented research that is replacing a heavy reliance on anecdotes by industry and academic experts.

A casual review of PATH is easily confusing. There are numerous projects and reports, which are not readily ordered by an overarching model. I emphasize again that there is no single model that can (or should) determine PATH’s logic. If we’ve learned one thing from our research, it is that the complexity of the systems involved requires multiple models rather than gross simplifications.

As PATH has matured, it is becoming increasingly necessary to clarify its contributions by providing integrative summaries and meta-analyses of its individual projects. These overviews would help crystallize what PATH has established, where knowledge gaps are persistent, and where new directions should be established. I would recommend that these overviews address the metrics of PATH’s logic model, but not be constrained by them. Logic models are at best a set of working hypotheses that should be revised to reflect the ongoing learning that PATH promotes.

Matt Syal

Michigan State University

Let me make a couple of points related to research. With PATH’s increasing acceptance as the voice of innovation among industry and researchers, more and more people are looking to PATH for good research—both applied and basic. As the PATH dollars reduce and other pressures start to build, we need to explore innovative ways to accomplish the research goal. Two ideas:

1. PATH should form a panel of experts and researchers to evaluate any research coming out of academia, industry, associations, PATH-supported contractors, etc. (kind of an FDA model). After the research is approved by the panel, it should be recommended by PATH.

2. PATH should serve as a catalyst in forming an industry-funded research consortium (similar to CII or FIATECH). This should be a completely independent consortium and not an add-on to any existing group.

Douglas Thomas

National Institute of Standards and Technology

The PATH goals outlined in PATH Program Review and Strategy, Performance Metrics, and Operating Plan are good principles that create an ideal sense of direction. All of the issues that are addressed in the draft are pertinent to increasing innovation in the residential construction industry. The primary concern that I have with the draft was touched on briefly by Dr. Slaughter at the PATH workshop and in Dr. Koebel’s statement.

The goals may be ambitious for the PATH organization, and it will be difficult to measure their progress. PATH is charged with identifying and reducing barriers, disseminating information to accelerate innovation, and advancing housing technology research and development. Measuring the

progress of PATH toward these goals is as difficult as achieving them. However, given the purpose of PATH these challenges are expected. In many respects, the residential construction industry today is similar to the agricultural industry of the early 20th century. Both industries have a slow rate of technology diffusion, have a preponderance of small firms, have workers with poor skills and training, are subject to high risk, are geographically dependent, and use sales representatives as their primary source of information. The state Agricultural Extension Services program was established in 1914 to diffuse innovations in the agricultural industry, similar to the purpose of PATH. It was not until the 1950s that the agricultural revolution took off. Measuring the progress of the Agricultural Extension Services would have been difficult until at least the 1950s. Similarly, it may take time before the efforts of PATH are clearly visible in the industry. In the meantime, PATH could have a difficult time proving its performance level and its effect on the residential construction industry.

Hassell, S., A. Wong, A. Houser, D. Knopman, and M. Bernstein. 2003. Building Better Homes: Government Strategies for Promoting Innovation in Housing. Rand Corporation, Santa Monica, Calif. Available at www.rand.org/publications/MR/MR1658/MR1658.pdf.

Koebel, C.T., M. Papadakis, E. Hudson, and M. Cavell. 2004. Diffusion of Innovation in the Residential Building Industry. Office of Policy Development and Research, HUD, Silver Spring, Md. Available at www.huduser.org/Publications/PDF/Diffusion_Report.pdf.

National Research Council. 2003. Promoting Innovation: 2002 Assessment of the Partnership for Advancing Technology in Housing. The National Academies Press, Washington, D.C. Available at books.nap.edu/catalog/10688.html.

Partnership for Advancing Technology in Housing (PATH). NSF-PATH Housing Research Agenda Workshop Final Report. Available at www.pathnet.org/sp.asp?id=12201.

U.S. Department of Housing and Urban Development (HUD). 2005. Overcoming Barriers to Innovation in the Home Building Industry. Office of Policy Development and Research, HUD, Silver Spring, Md. Available at www.pathnet.org/si.asp?id=1452.

This page intentionally left blank.