National Academies Press: OpenBook

Rebuilding the Research Capacity at HUD (2008)

Chapter: 3 Evaluation of External Research

« Previous: 2 Background
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 33
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 34
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 35
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 36
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 37
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 38
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 39
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 40
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 41
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 42
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 43
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 44
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 45
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 46
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 47
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 48
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 49
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 50
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 51
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 52
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 53
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 54
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 55
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 56
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 57
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 58
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 59
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 60
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 61
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 62
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 63
Suggested Citation:"3 Evaluation of External Research." National Research Council. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: The National Academies Press. doi: 10.17226/12468.
×
Page 64

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

3 Evaluation of External Research INTRODUCTION Much of the research agenda of PD&R is carried out by outside research organizations that are selected and funded to conduct specific studies. This chapter assesses the quality, timeliness, and usefulness of this external research. Following a brief overview of the processes used to select and supervise external research organizations, the chapter delineates three broad categories of research—large-scale, high-impact research studies; intermediate-scale ­policy and program studies; and small-scale exploratory studies—and defines criteria for evaluating studies in each category. The chapter then addresses each category in turn, first evaluating individual studies in the category and then assessing the overall portfolio of research projects in the category. Following these assessments, the chapter discusses PD&R’s overall agenda-setting process and the overall agenda for external research. The final section presents the committee’s conclusions and recom- mendations for external research. Funds obligated for external research averaged about $30.3 million between 1999 and 2007, ranging from a high of $47.2 million in 2000 to a low of $14.8 million budgeted for 2007 (see Table 2-6 in Chapter 2). Most of this funding comes from research and technology appropriations to PD&R, but additional funding for external research is sometimes provided from either salaries and expense appropriations or program appropriations to other offices in HUD. Research and technology funding obligated for external research dropped precipitously in 2002 and again in 2006. PD&R staff members, working with representatives from other offices 33

34 REBUILDING THE RESEARCH CAPACITY AT HUD of HUD, select the topics, define the research questions to be addressed, determine the basics of the methodology to be implemented, and estimate the likely cost of the research. Generally, a research organization is selected competitively by a panel of HUD staff to conduct each study, often through a formal request for proposals and a structured ranking and selection process. The organizations that compete for and conduct PD&R-funded research include for-profit firms, nonprofit research organizations, and (sometimes) academic institutions. Recently, competition for many studies has been limited to small businesses. Once a contractor has been selected, PD&R staff monitor progress and performance and review the research products. Almost all of the studies funded by PD&R produce reports that are made available to the public. Most of these reports are published in hard copy by PD&R and dissemi- nated through HUD USER. Exceptions include papers funded by small grants (discussed further below), which are intended for publication in journals, and studies that PD&R did not initiate but provided partial fund- ing for, which are usually published by other sponsors. In addition, a small number of external research projects yield findings or reports that PD&R decides cannot be released because of poor quality. PD&R’s standard contract allows the funded research organizations to publish results independently once a study has been completed and follow- ing a set embargo period. Consequently, in addition to HUD publications, PD&R-funded research appears in academic journals, conference presen- tations, book chapters, policy briefs, opinion pieces, and congressional testimony. Types of External Research and criteria for evaluation Three basic types of research studies form part of a comprehensive, policy research agenda: large-scale, high-impact studies; intermediate-scale   n some cases, PD&R has awarded “indefinite quantity contracts” to several research I organizations (selected competitively), which are then tapped for specific, quick-turnaround research projects.   D&R staff identified seven studies funded in recent years that were not published for this P reason. In some cases, PD&R staff made revisions themselves and produced a releasable report despite the fact that the contractor’s report was deemed unsatisfactory.   inal scopes of work typically include the following language: “Contractors may not pub- F lish a report based on this study or otherwise disclose the contents of research reports prepared under this contract to the public for three months following the formal submission of the final report, unless the contracting officer has given written permission. After the three-month period, the Contractor who wishes to publish shall include a clear notice that the research was performed under a contract with the Office of Policy Development and Research, U.S. Department of Housing and Urban Development.”

EVALUATION OF EXTERNAL RESEARCH 35 policy and program studies; and small-scale, exploratory studies. Although these categories of research overlap and the boundaries between them are not always distinct, it is useful to think about PD&R’s sponsored research in this framework. Large-Scale, High-Impact Studies Large-scale, high-impact studies address major enduring policy ques- tions that matter to the public, Congress, and the HUD secretary. Such studies are typically costly (often over $1 million) and take more than a year (sometimes several years) to complete. But more important than their size is the fact that studies in this category are designed and implemented to address fundamental questions about the need for and effectiveness of public interventions—questions that span administrations and help shape long-term public policy development. Important examples of PD&R-funded research of this type include three national studies of the incidence of discrimination against racial and ethnic minorities searching for housing in urban areas. These studies, conducted in 1977, 1989, and 2000, pioneered the use of the “paired-testing” methodol- ogy. Other important examples of PD&R-funded research in this category include the Moving to Opportunity for Fair Housing (MTO) demonstra- tion and the Jobs-Plus demonstration. Both of these studies implemented rigorous, controlled experimental designs to assess the effects of housing interventions on resident self-sufficiency. MTO, which is ongoing, measures the effects of providing vouchers that require low-income families to relo- cate to low-poverty neighborhoods (Orr et al., 2003). Jobs-Plus measures the effects of delivering intensive employment assistance and incentives to residents of public housing developments (Bloom et al., 2005). Intermediate-Scale Policy and Program Studies Moderate-scale studies address significant (but more immediate) issues of program design and implementation or market trends and conditions. Though less costly than the multiyear high-impact projects, these studies still involve substantial investments, often costing hundreds of thousands of dollars and requiring a year or more to complete. PD&R has funded many studies in this category. One example esti- mated the number, characteristics, and risk profile of potential homeown- ers (Galster et al., 1996). This study was the first to use the federal Survey of Income and Program Participation to analyze who among the pool of renters might become homeowners if various conditions were changed by public policies, and how that, in turn, might change the profile of mortgage default risk. Another example was a study of metropolitan areas across the

36 REBUILDING THE RESEARCH CAPACITY AT HUD country where the federal housing voucher program is administered region- ally, or at least across several jurisdictions (Feins et al., 1996). The latter study identified examples of regional voucher administration, described the historical and policy circumstances that led to it, documented how regional administration was carried out, and assessed potential strengths and weak- nesses of regional administration. Small-Scale, Exploratory Studies Small, exploratory studies investigate new issues, expand the use of new data sets, or engage new researchers. Studies in this category typically cost under $100,000 and are completed within 1 year. Given their relatively small size, they may be quite narrowly focused or provide only preliminary answers, but they can also extend the scope of a policy research agenda into new issue areas or explore innovative methodologies. Beginning in 1997, PD&R initiated “small grant competitions,” inviting researchers to suggest studies around a broad policy theme. Two of these competitions related to the mortgage purchase activities of Fannie Mae and Freddie Mac, and a third explored the topic of socioeconomic change in ­cities. In 2003 PD&R instituted a different method for securing smaller- scale scholarly research on specific topics relevant to HUD assisted housing programs: the “research cadre.” In this program, PD&R authorized a private contractor to perform all tasks necessary to select and fund a cadre of as many as 20 social science researchers capable of conducting policy research and analysis using HUD’s program administrative data as well as data from other sources. HUD episodically provides the contractor with research topics, and the contractor authorizes a member of the cadre to conduct the work and provides appropriate oversight and project management. Criteria for Evaluation Studies in all three categories should meet a common set of evaluation criteria: (1) relevance and importance of the topic; (2) rigor and appropri- ateness of methodology; (3) timeliness; (4) qualifications of the research team; and (5) quality of the research products. The specifics of these five basic criteria differ somewhat across the three categories of study; Table 3-1 details those differences. LARGE-SCALE, HIGH-IMPACT RESEARCH STUDIES Over the last decade, PD&R has sponsored a small number of very high-quality research initiatives that have rigorously addressed major policy issues of importance to the nation. All four that have produced interim or

TABLE 3-1  Criteria for Evaluating PD&R’s External Research Evaluation Criteria Large-Scale, High-Impact Studies Moderate-Scale, Program Studies Small-Scale, Exploratory Studies Relevance and Study should address a clearly Study should address an issue of Individual studies need not be directly importance of topic articulated research question or program design or implementation related to HUD’s current agenda, but hypothesis of major importance to important to HUD’s mission. may be foundational, or exploratory. public policy. Rigor and Basic methodology (selected by Basic methodology (selected by Methodology implemented by a appropriateness of PD&R) should be appropriate to PD&R) should be appropriate to particular scholar or team should be methodology address the study’s research question address the study’s basic objectives; appropriate to address the study’s or hypothesis; details (developed by details (developed by the basic objectives and should yield the contractor) should yield contractor) should yield reliable reliable results at a level of statistically significant, reliable, and results at a level of generalizability generalizability appropriate to the generalizable answers to the study appropriate to the topic and time topic and time frame. questions. frame. Timeliness Not applicable. Study should produce results in the Study should produce results within 1 time frame needed by the relevant year. program office or stakeholders. Qualifications of Research team should include Team selected should include Individual or team selected should research team qualified methodologists, analysts, qualified methodologists, analysts, possess the technical and/or policy and project managers and advised by and project managers. qualifications appropriate for the an outside panel of specialized proposed methodology. experts. Quality of research Products should include complete Products should include complete Products should include a complete products documentation of data and methods; documentation of data and documentation of data and methods; comprehensive reporting of results; methods; comprehensive reporting comprehensive reporting of results; and understandable assessment of of results; and understandable and understandable assessment of the the implications. Quality can also be assessment of the implications. implications. Eventual publication in assessed if scholarly papers are Quality can also be assessed if peer-reviewed journals is expected. published after peer review. scholarly papers are published after peer review. 37

38 REBUILDING THE RESEARCH CAPACITY AT HUD final results are described here, applying the evaluation criteria in Table 3-1: (1) the 2000 Housing Discrimination Study (HDS-2000); (2) the MTO demonstration, which evaluates the effects of assisted housing mobility; (3) the evaluation of the effectiveness of housing vouchers for welfare families; and (4) the Jobs-Plus demonstration, which evaluates the effects of work incentives and supports for public housing residents. In addition to evaluat- ing the quality of these individual, high-impact studies, this section assesses the mix of studies sponsored over the years and the extent to which this mix addresses the information needs of HUD and the larger housing and urban development policy community. The 2000 Housing Discrimination Study Since the 1960s, advocates for fair and open housing have used a technique called paired testing to detect and reveal discrimination by real estate and rental agents. In a paired test, two individuals—one white and the other minority—pose as equally qualified home seekers. Both testers are carefully trained to make the same inquiries, express the same prefer- ences, and offer the same qualifications and needs. From the perspective of the housing provider they visit, the only difference between the two is their race or ethnicity, and they should therefore receive the same information and assistance. Systematic differences in treatment—telling the minority customer that an apartment is no longer available when the white customer is told he could move in next month, for example—provide powerful evi- dence, easily understandable by the general public, of discrimination that denies minorities equal access to housing. When a large number of consistent and comparable tests are conducted for a representative sample of real estate or rental agents, the results con- trol for differences between white and minority customers, and directly measure the prevalence of discrimination across the housing market as a whole. PD&R recognized the potential of the paired testing methodology as a research tool and has used it to monitor the incidence of housing dis- crimination nationwide at roughly 10-year intervals. The 1977 Housing Market Practices Study provided the first solid esti- mates of the prevalence of discrimination against African American home seekers (Wienk et al., 1979) and helped build the case for strengthening the enforcement of federal fair housing protections in the 1988 Fair Housing Act Amendments. The 1989 Housing Discrimination Study extended those initial national estimates to cover Hispanics and concluded that overall lev- els of adverse treatment against African Americans had remained essentially unchanged since 1977 (Turner, Struyk, and Yinger, 1991). Most recently, HDS-2000 reported the change since 1989 in discrimination against African Americans and Hispanics and up-to-date estimates of the incidence of dis-

EVALUATION OF EXTERNAL RESEARCH 39 crimination, including the first national estimates of discrimination against Asians and Pacific Islanders and the first rigorous estimates of discrimina- tion against Native Americans searching for housing outside of native lands (Turner et al., 2002b; Turner and Ross, 2003a, 2003b). Funding for HDS-2000 was allocated by Congress from annual appro- priations to the Fair Housing Initiatives Program, and PD&R was assigned responsibility for study design and selection of the research team. The request for proposals (RFP) envisioned three phases of paired testing, with the first phase focusing on estimates of change in the incidence of discrimination against African Americans and Hispanics in metropolitan areas nationwide and subsequent phases focusing on other minority groups or nonmetropolitan communities. In addition, the RFP called for a sample design that would both measure change at the national level and provide reliable estimates of the incidence of discrimination for individual metro- politan areas. PD&R selected a team led by the Urban Institute to con- duct HDS-2000. This team included staff and consultants who had been involved in previous paired testing studies and had extensive expertise in fair housing issues, the paired testing methodology, sampling methodolo- gies, and management of large-scale field data collection. Each of the study’s three phases involved selection of a representative sample of metropolitan areas in which testing was conducted, selection of representative samples of advertised housing units in these metropolitan areas, highly standardized paired testing protocols, and rigorous statisti- cal analysis. Reports for each phase were published by HUD, and include complete documentation of sampling and statistical procedures and paired testing protocols (Turner et al., 1991, 2002b; Turner and Ross, 2003a, 2003b). Findings from HDS-2000 have been presented at academic and practitioner conferences, and summarized in several book chapters and journal articles. Assisted Housing Mobility Authorized by Congress in 1992, the MTO demonstration provided tenant-based rental assistance and housing search and counseling services to families living in high-poverty public and assisted developments, in order to assess the effects of neighborhood conditions on educational and employment outcomes. MTO was inspired by findings from the Gautreaux demonstration, which provided special-purpose vouchers to enable African American families (who either lived in public housing or were eligible for it) to move to predominantly white or racially mixed neighborhoods in the city of Chicago and surrounding suburban communities. This program was designed as part of the court-ordered legal remedy for systematic dis- crimination and segregation of Chicago’s public housing program. Research

40 REBUILDING THE RESEARCH CAPACITY AT HUD on Gautreaux families suggested that many of the families who moved to suburban neighborhoods and stayed there experienced substantial benefits over time. PD&R convened a panel of academics, policy experts, and ­practitioners to help develop the basic demonstration design for MTO. MTO’s experi- mental design randomly assigned eligible families, who volunteered to par- ticipate, to one of three groups. The experimental group received ­Section 8 certificates or vouchers usable only in low-poverty census tracts (defined as under 10 percent poor in 1990) and assistance in finding a unit and moving. The comparison group received regular Section 8 certificates or vouchers, which had no geographical restrictions and which did not pro- vide search assistance. A control group continued to receive project-based assistance. PD&R then competitively selected a contractor (Abt Associates, Inc.) to manage the demonstration operations, including baseline data collec- tion, random assignment, monitoring counseling operations, and track- ing household outcomes. The Abt Associates team was well qualified for this assignment, consisting of sampling and survey specialists, experts in experimental design demonstrations, and staff with extensive experience in the operations of public housing agencies and the voucher program. Dur- ing the early years of the demonstration, small grants were also awarded competitively to academic researchers in the demonstration sites who proposed innovative, exploratory studies of the relocation and neighbor- hood adjustment process. These grants engaged a pool of distinguished academics from fields other than housing in the ongoing demonstration research effort. In 1999, PD&R issued an RFP for an interim evaluation of demonstration impacts. This contract was awarded to a team led by Abt Associates, but also including researchers from the Urban Institute and the National Bureau of Economic Research (NBER). Members of this research team secured substantial additional funding from foundations and the National Institutes of Health for the interim evaluation. Finally, in 2006, PD&R issued an RFP for a final evaluation of MTO. This contract was also awarded to NBER. Both the interim and final evaluations use a combination of admin- istrative data and follow-up surveys of experimental, comparison, and control households. They rigorously measure MTO “treatment” effects by comparing outcomes for experimental, comparison, and control groups over time. The interim evaluation results are fully documented in a report by Abt and NBER researchers published by HUD (Orr et al., 2003). In addition, numerous site-specific studies using a range of data collection and analytic methods have been conducted and continue to be conducted with foundation funding. This research has been published in numerous working papers, policy briefs, a book, and academic journal articles. Links to most

EVALUATION OF EXTERNAL RESEARCH 41 of the published studies and reports are provided on a website (http://www. mtoresearch.org). To date, the evaluation research has found that the MTO treatment enabled families to move to dramatically safer neighborhoods with lower poverty rates and more neighbors who are working. However, most of these neighborhoods are majority-minority and located within central city juris- dictions. The MTO treatment has resulted in significant improvements in the physical and mental health of women and girls. However, no significant gains in employment, earnings, or educational outcomes were found across the five demonstration sites, and delinquent behavior among boys appears to have increased among experimental families. All of these outcomes are currently being reassessed as part of the final evaluation, which should be completed by 2010. Housing Vouchers for Welfare Families In 1999 Congress passed a special appropriation of housing vouchers for a demonstration initiative targeted specifically to families making the transition from welfare to work. Public housing agencies were competitively selected to participate in this demonstration, based on locally designed strategies for coordinating housing assistance with welfare reform and welfare-to-work initiatives. The appropriation for this Welfare to Work Voucher cemonstration provided a 1 percent set-aside for evaluating the effect of housing assistance on welfare families under the demonstration. From the outset, PD&R planned a random assignment, experimen- tal design methodology for this demonstration. First, a contractor was competitively selected from among existing indefinite quantity contract h ­ olders to develop the evaluation methodology, design and conduct random assignment of applicants, develop data collection methods and instruments, and conduct baseline data collection. Then a separate RFP was issued to select a contractor to implement the full evaluation methodology, includ- ing all post-test data collection and analysis. Abt Associates, Inc. won both of these competitive procurements. The Abt team was extremely well qualified to conduct the welfare voucher evaluation; the company had staff with long-standing expertise in the voucher program, random assignment demonstrations, sampling and statistical procedures, survey design and implementation, and administrative data collection. This team designed and implemented a rigorous evaluation methodology, which made excellent use of both household survey data and administrative data on individual demonstration participants. Abt researchers were simultaneously involved in the MTO evaluation and in a panel survey of public housing families

42 REBUILDING THE RESEARCH CAPACITY AT HUD relocating from HOPE VI developments, and they were able to incorporate methods and lessons from these initiatives into the design of the Welfare to Work Voucher demonstration and evaluation. Abt Associates completed two major reports on the Welfare to Work Voucher demonstration—an interim report to Congress in 2004 (Patterson et al., 2004) and a final report in 2006 (Mills et al., 2006). These were both published by PD&R. The findings from these studies provide important new evidence on the effects of housing voucher receipt on key outcomes for welfare families. Specifically, the evaluation found that receiving a housing voucher resulted in small improvements in neighborhood conditions among welfare families, enabled welfare mothers and their children to live indepen- dently rather than doubling up or living in multigenerational households, dramatically reduced the incidence of homelessness, and increased spending on food. The research also found that receiving a voucher initially reduced recipients’ employment and earnings, but after a year or two this negative effect disappeared: over a 3.5-year period, there was no significant effect of voucher receipt on employment and earnings. These findings, and the details of the demonstration design and evaluation methods, are fully docu- mented and clearly explained in the HUD reports. To date, findings from this research have not appeared in academic journals or books. Work Incentives and Supports in Public Housing (Jobs-Plus) In the mid-1990s, as debates over welfare reform were under way, representatives from the Rockefeller Foundation and the Manpower Dem- onstration Research Corporation (MDRC) approached the PD&R assis- tant Secretary to explore ideas for promoting work and self-sufficiency in public housing developments where unemployment and rates of welfare receipt were extremely high. Together, the three organizations developed the basic framework for the Jobs-Plus demonsration, which was ulti- mately implemented in randomly selected public housing developments in five ­ cities, with randomly selected comparison developments in each city to allow for rigorous estimates of the impact of saturation services and incentives. Jobs-Plus was designed to test the impact of a saturation intervention that included work incentives, employment services, and community supports for work on employment and earnings among pub-   aunched in 1992, the HOPE VI Revitalization of Severely Distressed Public Housing L Program replaces severely distressed projects with redesigned mixed-income developments and provides housing vouchers to enable the original residents to rent apartments in the private market. It is the department’s most extensive effort to address the problems in some public housing projects and to reduce concentrations of poverty.

EVALUATION OF EXTERNAL RESEARCH 43 lic housing residents and on employment rates and community health in public housing developments. Because the concept and design of Jobs-Plus were jointly developed and the Rockefeller Foundation was providing substantial funding, PD&R entered into a sole-source, cooperative agreement with MDRC for all phases of the demonstration design, implementation, monitoring, and evaluation. MDRC was well qualified for this role. Although the organization did not have experience with federal housing programs, it had an outstanding track record of designing and implementing controlled experimental design demonstrations of welfare and employment initiatives. The project staff included well-qualified statistical, sampling, and survey methodologists and experts in administrative data assembly and analysis, as well as personnel with extensive experience in the implementation of demonstration initia- tives, and the evaluation design for the demonstration implemented creative methods for using administrative data on residents of both treatment and control sites from before and after the intervention to rigorously measure the impact of “saturating” public housing developments with services and incentives. Over roughly a decade, MDRC published 11 formal reports on Jobs- Plus design, implementation, and results, culminating in a final report on the demonstration’s impacts on employment, earnings, and neighborhood health (Bloom et al., 2005). MDRC researchers have also published sev- eral journal articles on the evaluation design and findings. The MDRC research concludes that, when effectively implemented, the Jobs-Plus model (combining work incentives, employment services, and community sup- ports for work) results in significant increases in individual earnings. These earnings gains stem in part from increased employment rates, but also from increased hours and wages among working adults. Despite the gains in earnings, however, Jobs-Plus had no measurable effects on the overall employment rate in the targeted projects or on other indicators of commu- nity health or quality of life. Assessment of the Overall Portfolio of High-Impact Research Studies Although PD&R has conducted very high-quality studies of this type, the mix of PD&R-funded research includes too few of the ambitious, large- scale studies that answer fundamental questions of impact and effectiveness. This has been true throughout PD&R’s history, and only four or five ­studies sponsored since 1997 fall into this category: those discussed above and possibly a recently initiated evaluation of housing counseling. Only two of   ith W increasing interest in counseling by both the administration and Congress and large percentage increases in funding, PD&R initiated an evaluation of HUD-approved counsel-

44 REBUILDING THE RESEARCH CAPACITY AT HUD these (the MTO evaluation and the housing counseling study) are currently under way and, based on information provided by PD&R, no new large- scale, high-impact research studies appear to be in the planning stages. Over the past decade, PD&R has missed some critical opportunities to launch rigorous studies on high-priority policy issues. For example, in 1996, Congress authorized HUD to conduct a Moving to Work (MTW) demonstration, which allowed a small number of public housing agencies unprecedented flexibility by granting waivers of federal statutes and rules related to the Public Housing and Housing Choice Voucher programs (P.L. 104-134, 204). Participating agencies had the opportunity to design and test new approaches to reducing program costs, encouraging economic residents’ self-sufficiency, and increasing the housing choices of low-income families. Although Congress mandated that MTW be evaluated, PD&R did not insist on building a rigorous evaluation into the design of the dem- onstration. In part, this was because PD&R had already committed sub- stantial resources to public housing demonstrations (including MTO and Jobs-Plus, discussed above) and lacked the resources to launch another. In addition, the participating housing agencies in MTW were extremely vocal in insisting that the demonstration was intended to give them flexibility and independence, not more data collection and evaluation burdens. The Office of Public and Indian Housing supported this argument and joined in opposition to a formal research demonstration. Instead, the Office of Public and Indian Housing used resources of its own to contract for an assessment of MTW (Abravanel et al., 2004). While useful, this assessment did not establish control groups, require baseline data collection, or rigorously measure impact or effectiveness. In fact, because of delays in the adaptation of HUD’s primary system for collecting and maintaining data on public housing residents and voucher recipients, no consistent data on resident outcomes were collected across the MTW sites. Over the last several years, HUD has advanced several public housing and voucher reform proposals, arguing that some form of flat or “stepped” rents (rather than the current percent-of-income formula) would create incentives for assistance recipients to get and keep jobs. Several MTW housing agencies experimented with rent reforms, and if the demonstration had been properly evaluated, it would have provided reliable information on their effects and effectiveness. Unfortunately, however, 10 years after enactment of MTW, the country is in many respects no better informed ing agencies in September 2007, which is being conducted by Abt Associates. The study is discussed in more detail in Chapter 9. The design and scope are still under development: The evaluation has a 5-year schedule, and its course will depend on the availability of funding. It is premature to classify it as a large-scale, high-impact study, though that is PD&R’s intention.

EVALUATION OF EXTERNAL RESEARCH 45 about the benefits of public housing and voucher deregulation than it was before the program. A considerable amount is known about which local policies and practices housing agencies adopted when rule making became decentralized, but too little is known about the outcomes or effectiveness of critical innovations, including rent reforms, work requirements, and time limits. Because it has not carried out rigorous, large-scale evaluations, PD&R has missed opportunities to inform HUD, Congress, and the public about the impacts and cost-effectiveness of major initiatives. For example, beginning in the mid-1990s, the Empowerment Zone Program made huge investments in locally designed strategies for promoting the economic redevelopment of distressed urban and rural communities. This program built on past evi- dence and theory about the challenges of neighborhood revitalization, and it offered participating communities considerable flexibility in the design of local strategies and in the use of federal resources. HUD’s Office of Com- munity Planning and Development strongly opposed the implementation of a rigorous, PD&R-led evaluation, and blocked PD&R’s efforts to design and launch any independent assessment. Ultimately, an intermediate-scale study was launched by a PD&R contractor, in partnership with local evalu- ation teams. However, neither the scale nor the design of this evaluation was sufficient to provide definitive findings regarding either impacts or cost-effectiveness. Moreover, other critical policy questions that merit rigorous analysis have not been addressed. For example, policy makers and practitioners lack reliable information about both short- and long-term impacts for families and children of different types of housing assistance: How does receipt of housing assistance affect residential stability, household budgets, housing quality, child nutrition, physical and mental health, employment, and edu- cational achievement? In addition, while rigorous research documents the persistence of racial and ethnic discrimination in urban housing markets, little is known about how families search for housing and how barriers such as discrimination or unaffordable prices and rents affect their search and decision processes and their ultimate housing outcomes. Similarly, too little is known about the operation of the supply side of today’s city and suburban housing markets, including the filtering process (the extent to which particular zoning and land-use regulations restrict the total volume of production), and the impact on prices and rents of specific regulatory reforms. And finally, too little rigorous research has been conducted to assess how innovative land-use regulations, design standards, and building technologies could affect development patterns, commuting times, housing quality, and housing affordability.

46 REBUILDING THE RESEARCH CAPACITY AT HUD INTERMEDIATE-SCALE POLICY AND PROGRAM STUDIES Intermediate-scale studies—costing $100,000-$499,999—account for the majority of PD&R’s external research. Over the last decade, PD&R has sponsored several high-quality studies that provide important information about program implementation, utilization, and effectiveness, including studies that used creative or innovative methodologies to estimate impacts and effectiveness without the costs of a controlled experimental design. Six recent, high-quality examples are described here, applying the evalua- tion criteria introduced in Table 3-1: (1) a national study of closing costs on home sales; (2) a national survey of public attitudes about federal fair housing protections; (3) a pilot paired testing study of discrimination in home insurance; (4) innovative studies of the effects on property values of place-based investments; (5) an evaluation of the “Mark-to-Market” Program involving Section 8 new construction projects; and (6) actuarial and policy analysis of the Home Equity Conversion Mortgage (HECM) Program. In addition, the committee conducted a systematic assessment of recent studies in this category. This section evaluates the quality of these individual studies, and assesses the extent to which PD&R’s programmatic research has effectively evaluated the effects and cost-effectiveness of HUD’s major programs. Mortgage Closing Costs Since passage of the Real Estate Settlement Procedures Act in 1974, HUD has been responsible for regulating the processes of buying homes and refinancing mortgages. Costs associated with the transaction (mortgage origination, appraisal, title insurance, and others) are large and vary widely among borrowers, resulting in arguments about the causes of the variation. The regulations have been the subject of repeated intense policy debates since the act was passed, and numerous class-action lawsuits have been filed and adjudicated on this topic in the past decade or more. In the course of HUD’s rule-making process, PD&R undertook exten- sive data assembly and systematic analysis of closing costs on FHA-insured home purchase mortgages (Woodward, 2008). It was the first major study of closing costs, other than proprietary work in connection with class- action lawsuits. More than 7,500 mortgages were included in the study, including loans in each state and the District of Columbia. The study analyzed differences in overall closing costs, title insurance, and real estate brokerage fees across a number of dimensions, including by state and by characteristics of the neighborhood (race, ethnicity, and education level of residents); by type of loan originator (mortgage brokers, mortgage bankers, or depositories such as banks and credit unions); and by the way in which

EVALUATION OF EXTERNAL RESEARCH 47 closing costs were paid, whether by cash at settlement or in the form of a higher monthly mortgage payment (known as a yield spread premium). The purpose was to investigate the extent of competitiveness and transparency in the mortgage origination process. Funding for the study was provided largely from the PD&R budget, with some additional support from FHA. The study found that closing costs vary substantially across the states. The national average was $3,400, but there was an average difference of $2,700 per loan between the highest and lowest cost states. Costs were higher in neighborhoods with a high minority population and lower in neighborhoods in which residents were largely college graduates. Costs varied by type of originator, being highest for loans originated by mortgage brokers and lowest for those originated by depository institutions. Costs also varied by mode of payment, being lowest when they were completely financed as part of the interest rate so that borrowers needed only compare loans on the basis of the interest rate. The closing cost study provides—for the first time—rigorous infor- mation about closing costs for home sales nationwide. Its design was not experimental; rather, it used the data from actual home purchase transac- tions and conducted systematic statistical analyses of the data. Conceptually, it is analogous to the housing discrimination studies; it reported on current housing market conditions. In both housing discrimination and home pur- chase transactions, HUD has regulatory and enforcement authority, and the results of the studies may lead to regulatory or legislative action. Public Attitudes Toward Fair Housing Protections HUD’s Office of Fair Housing and Equal Opportunity is responsible not only for enforcing federal fair housing laws, but also for administering programs that expand public awareness of and support for fair housing protections. In order to assess levels and trends in both awareness and support, PD&R commissioned two surveys of public awareness, the first in 2000-2001 and the second in 2005. The contractor for the first survey (the Urban Institute) was competitively selected from several firms holding indefinite quantity contracts. The contractor for the second (M. Davis and Company, Inc., with the Urban Institute as a subcontractor) was competi- tively selected from a pool of small businesses. PD&R’s initial statement of work called for a survey of a nationally representative sample of adults that would assess the extent to which people understand and support federal prohibitions against housing discrimination based on race, ethnicity, national origin, disability status, and family compo- sition. The Urban Institute developed a questionnaire design centered around 10 scenarios, each describing a set of actions by landlords, home sellers, real estate agents, or lenders, eight of which involved conduct that is prohibited

48 REBUILDING THE RESEARCH CAPACITY AT HUD under federal law. Respondents were asked if they approved or disapproved of the actions taken in each scenario and whether they thought it was legal or illegal. The scenarios were carefully worded so as not to signal whether the actions were illegal or undesirable and covered protections against dis- crimination for families with children, racial and ethnic minorities, disabled people, and people of different religions. The survey was administered by telephone to a nationally representative sample of 1,000 adults, as part of the well-established Survey of Consumers, which is conducted monthly by the Survey Research Center at the University of Michigan. Analysis of survey responses found widespread knowledge of and sup- port for many federal fair housing protections, although only a minority of respondents was aware that it is illegal to treat families with children dif- ferently from childless households. For each category of federal fair housing protections, the analysis estimated the shares of people who are supportive but not knowledgeable, knowledgeable but not supportive, and neither knowledgeable nor supportive, highlighting the fact that these groups may require different types of outreach and education strategies. Roughly 5 years after this initial survey was conducted, PD&R com- missioned the follow-up survey, with a statement of work calling primarily for a replication of the original instrument in order to measure changes in both awareness and support for federal fair housing protections. The 2005 survey was administered by telephone to a nationally representative sample of 1,029 adults. In addition, supplemental samples of four targeted populations—African Americans, Hispanics, families with children, and people with disabilities—were interviewed in order to assess the extent to which their awareness or attitudes might differ from the national average. Analysis of the follow-up survey found little change in public awareness of federal fair housing protections, but significant increases in public support. African Americans and Hispanics were somewhat more likely to be aware of protections against racial and ethnic discrimination, and African Ameri- cans, Hispanics, and families with children all exhibited above-­average support for these protections. PD&R issued reports from both survey waves, providing clear and complete documentation of data, methods, findings, and implications. In addition, results from the first survey wave were published in an article in Housing Policy Debate, and results from the second wave appeared as a chapter in an edited volume on federal fair housing enforcement ­(Abravanel, 2002, 2006, 2007; Abravanel and Cunningham, 2002). Incidence of Discrimination in Home Insurance Another excellent example of this category of research study com- missioned by PD&R is Testing for Discrimination in Home Insurance

EVALUATION OF EXTERNAL RESEARCH 49 ( ­ Wissoker, Zimmermann, and Galster, 1998). The study was commissioned by PD&R through an indefinite quantity contract to the Urban Institute in response to growing public concerns about the behavior of the home insurance industry. A spate of studies appearing in the late 1980s and early 1990s had documented that homeowners in minority neighborhoods were less likely to have private home insurance, more likely to have policies that provided less-than-average coverage in case of a loss, and were likely to pay more for such polices. But the central question remained unanswered: Were these differences due to sound, fair business practices associated with inter- neighborhood risk differentials or unfair discrimination against minority homeowners and their neighborhoods? The study addressed the question by employing the paired testing technique in a path-breaking way: both testers and homes were matched in this new design. Ten pairs of real, nearly identical homes (from the perspective of such key insurance variables as size, age, and materials) were found, with one home in each pair located in a predominantly white- occupied neighborhood and its match in a black-occupied neighborhood in New York City; 10 similarly matched pairs of homes were identified in predominantly Latino-occupied and Anglo-occupied neighborhoods in Phoenix. Five pairs of testers in each city were carefully matched on their characteristics except race or ethnicity. Teammates randomly phoned home insurance companies listed in the Yellow Pages seeking quotes for insurance on one of the matched homes “that they were planning on buying.” Testers rotated among the homes about which they were calling, to test for poten- tial differences in treatment depending on the congruence between the race or ethnicity of the caller and that of the neighborhood in question. These new testing methods, and the statistical tests of the data collected thereby, were thoroughly vetted by an external panel of experts (social sci- entists, insurance industry practitioners, community advocates) at several stages in the research. The statistical power of the sample sizes and validity of analysis techniques were confirmed, though of course the results could not necessarily be generalized beyond the two cities. The research was conducted by a team with extensive prior experience with paired testing techniques in both real estate and labor market transactions and that had expertise in both testing field operations and statistical methodology and analysis of test data. The final report provided an overview of the home insurance industry and outstanding research questions related to its behavior toward minority- occupied neighborhoods. The research found very few instances of differ- ential treatment that were consistent with the hypothesis of discrimination based on the racial-ethnic composition of the neighborhood or the home buyer. The report was comprehensive in its documentation of the method- ology and fieldwork protocols, matching procedures, statistical modeling,

50 REBUILDING THE RESEARCH CAPACITY AT HUD and power of the method. Primary and supplemental results were reported in detail, and lessons were drawn about home insurance testing that could be applied in future research. The report was explicit in presenting the strengths, limitations, and caveats associated with the study. Because this research was path breaking, complicated, and subjected to unusual scrutiny, the final report was not published until almost 3 years after the original task order was issued. The report was deemed sufficiently interesting and important that the Urban Institute published its own, edited version in 1998 and distributed it widely (Wissoker et al., 1998). The core methods and findings were later published in the peer-reviewed scholarly journal Urban Studies (Galster, Wissoker, and Zimmermann, 2001). Property Value Impact Studies PD&R has sponsored several intermediate-scale studies that use inno- vative quasi-experimental design method to estimate program impacts and effectiveness without incurring the costs of a controlled experimental design. One such set of studies used an adjusted interrupted time-series econometric model to assess the neighborhood property value effects of various types of assisted housing (Galster et al., 1999, 2000; Johnson and Bednarz, 2002). This work resulted in a series of scholarly publications in peer-reviewed journals: Galster et al. (1999, 2002); Santiago, Galster, and Tatian (2001); and Galster, Tatian, and Pettit (2004). Another example is a study that assesses the degree to which Com- munity Development Block Grant (CDBG) expenditures by cities have any noticeable impact on a variety of neighborhood indicators of interest (Walker et al., 2002). This project first developed a parsimonious set of eas- ily obtained, annually updated indicators that were shown to be a robust proxy for a wide range of community development objectives (published later in Galster, Hayes, and Johnson, 2005). The project then used a differ- ence econometric model embodying threshold effects to estimate what req- uisite intensity of CDBG spending was required before indicators changed appreciably (later published as Galster et al., 2004). The Mark-to-Market Program In 1997 Congress enacted the Multifamily Assisted Housing Reform and Affordability Act (MAHRA), intended to reduce subsidy costs in Section 8 new construction projects insured by the Federal Housing ­Authority (FHA) while still preserving the projects as affordable housing for lower-income families. MAHRA was originally scheduled to terminate in 2001, but was

EVALUATION OF EXTERNAL RESEARCH 51 extended to 2006. In 2002 PD&R funded a study of the program. The contractor was Econometrica, Inc., with Abt Associates as subcontractor. The study is partly an evaluation of the program and partly a descrip- tion of the process by which mortgages were restructured and subsidy costs reduced. It provides a clear description of the complicated Mark-to-Market Program created by MAHRA. It includes a process analysis of program administration, a review of HUD documents, and interviews with numerous program staff, project owners, and other program participants. A retro- spective statistical analysis estimates the subsidy savings to the government from the program, using all 2,400 projects that entered the restructur- ing program by July 2003. This analysis directly addresses the question of whether Mark-to-Market reduces subsidy costs. Finally, a prospective analysis consists of 15 detailed case studies, not selected randomly but designed to indicate the range of project types, restructuring arrangements, and outcomes (including some projects for which the process did not result in restructuring). The case studies were the main source of information on the impact of the program on tenants. This complicated study design nonetheless provided valuable informa- tion to policy makers, including congressional staff, when the program came due for a further legislative renewal in 2006, according to information provided to the committee. Policy makers received well-developed estimates of program savings, with sensitivity analyses to indicate the probable range of future savings, and insight into the complex restructuring process, which had at times appeared to be going more slowly than anticipated. The study has been published by PD&R (Hilton et al., 2004); it has not appeared in whole or part in peer-reviewed journals. Actuarial and Policy Analysis of HECMs The HECM Program has been regularly evaluated since it was enacted in 1990. The fourth of this series of evaluations (Rodda et al., 2003) falls into the set of intermediate-scale studies reviewed by the committee. Like the third evaluation, it was conducted by Abt Associates. The study differs in methodology from others in this category. It con- sisted of an actuarial analysis of the HECM, using HUD program data for the complete set of HECM loans originated between 1990 and 2000. An actuarial model was developed and used to calculate net expected liability to FHA per mortgage, with sensitivity analyses for alternative economic assumptions. The actuarial analysis is much more complicated than the annual analyses of the basic FHA home mortgage insurance program that have been conducted since 1990, because the HECM is a “reverse” mort- gage, in which an elderly borrower draws out the equity in his or her home to be paid back when the property is sold, rather than borrowing first and

52 REBUILDING THE RESEARCH CAPACITY AT HUD paying back over time. The methodology of the actuarial model involves technical refinements from the previous analysis, published in 2000. The study calculates that loan premiums are exceeding the projected costs of defaults. It also addresses three then-current policy issues: (1) replacing local loan limits (as in other FHA home mortgage programs) with a single national limit; (2) reducing the mortgage insurance premium for refinancing HECMs; and (3) waiving the upfront premium on HECMs used exclusively for the payment of long-term-care insurance policies. All three involve balancing a presumed benefit to elderly homeowners against pos- sible higher liabilities for FHA. The study found that the first two changes could be adopted without causing the HECM to be a financial burden on FHA and the taxpayer. Waiving the upfront premium for long-term-care insurance, however, would cost FHA a substantial amount. The study fur- ther concluded that the demand for HECMs dedicated to long-term-care insurance would be low, partly because the financial incentive from the waiver is modest. Although the purpose of this study was directly to inform policy ­makers and program managers, an article reporting the results of lowering the premium for refinances was subsequently published in the peer-reviewed journal Real Estate Economics (Rodda, Lam, and Youn, 2004). Assessment of the Overall Portfolio of Programmatic Studies As mentioned previously, the committee also conducted a systematic assessment of studies in this category carried out in the last 5 to 10 years. Using a PD&R list for all research projects meeting the intermediate-scale definition, the committee reviewed all studies that met the following three criteria: (1) the study was published; (2) the study title suggested research, not just description; and (3) the study did not focus on housing technology. Seventeen studies met these criteria: nine studies by small businesses funded by PD&R in 2002 or later and eight by nonsmall businesses funded by PD&R since 1999 (see Table 3-2). There was no difference in the quality of the small business and nonsmall business studies reviewed by the com- mittee, but it is worth noting that six of the nine small business contracts included a nonsmall business subcontractor. The committee found that virtually all of the 17 studies reviewed do   he first year of our review for small business studies was set at 2002 because HUD put T increased emphasis on the small business goals starting in fiscal 2002 (response to committee question by PD&R received November 20, 2007). In fiscal 2008, HUD’s goal is for small busi- nesses to receive 66 percent of dollars awarded to prime contractors and 57 percent of dollars awarded to subcontractors (see http://www.hud.gov/offices/osdbu/policy/goals.cfm). Note that the committee limited its review of the impact of HUD’s recent emphasis on small business contracting to an examination of the quality of research reports produced.

EVALUATION OF EXTERNAL RESEARCH 53 TABLE 3-2  PD&R Intermediate-Scale Studies Reviewed Date Published Report Title Funded Contractor Subcontractor (Date of Publication) 1999 ICF National Evaluation of the Housing Consulting Opportunities for Persons with AIDS Program (HOPWA) (December 2000) 2000 Abt Associates Urban Institute; Interim Assessment of the Empowerment university Zones and Enterprise Communities affiliates in 18 Program: A Progress Report (November localities 2001) 2000 Urban The Impact of CDBG Spending on Urban Institute Neighborhoods (October 2002) 2001 Abt Associates Refinancing Premium, National Loan Limit, and Long-Term-Care Premium Waiver for FHA’s HECM Program (May 2003) 2001 Westat Johnson, Bassin Housing Choice Voucher Tenant & Shaw, Inc.; Accessibility Study: 2001-2002 (January Cherry 2004) Engineering Support Services, Inc. 2001 Westat Johnson, Bassin Evaluation of the Family Self-Sufficiency & Shaw, Inc. Program: Retrospective Analysis, 1996 to 2000 (April 2004) 2001 Abt Associates Study of Homebuyer Activity Through the HOME Investment Partnerships Program (December 2003) 2002 Sociometrics How Do Prospective First-Time Corp. Homebuyers Search for Housing and Credit? (September 2003) 2002 M. Davis Univ. of Predicting Staying in or Leaving Pennsylvania Permanent Supportive Housing That Center for Mental Serves Homeless People with Serious Health Policy & Mental Illness (March 2006) Services Research 2002 Econometrica Abt Associates Evaluation of the Mark-to-Market Program (August 2004) continued

54 REBUILDING THE RESEARCH CAPACITY AT HUD TABLE 3-2  Continued Date Published Report Title Funded Contractor Subcontractor (Date of Publication) 2002 Mele The Cadmus Energy Star in HOPE VI Homes (October Associates Group 2004) 2002 Abt Associates Implications of Project Size in Section 811 and Section 202 Assisted Projects for Persons with Disabilities (March 2004) 2003 ESI Abt Associates The State of Affordable Housing in the U.S.—2000 (November 2004) (draft final report) 2004 Abt Associates Newport Partners Voucher Homeownership Study (March 2006) 2004 Exceed Corp. RTI International Interim Evaluation of HUD’s Homeownership Zone Initiative (March 2007) 2004 Econometrica Abt Associates Multifamily Properties: Opting In, Opting Out and Remaining Affordable (January 2006) 2005 Building ARES; M. Green A Methodology for Identifying, Discussing Technology, & Assoc.; Koffel and Analyzing the Costs and Benefits of Inc. Assoc.; SPA Risk; Code Changes That Impact Housing and Institute for (March 2007) Building Technology and Safety provide important insights into the operations of the programs that were studied. Nine of the 17 studies meet generally accepted research standards, and the study’s approach and findings are appropriately described in the report’s narrative, foreword, and preface. In 8 of the 17 studies, however, one or more of the findings go beyond the limits of the study design with the potential to mislead readers. For example, in six cases, there are refer- ences to the “effects” or “impacts” of the program or overly general state- ments about the functioning of the program or client satisfaction with the program based on a small number of case studies or on cross-sectional data

EVALUATION OF EXTERNAL RESEARCH 55 with no control or comparison group. In another case, the design of the convenience sample may have overstated the positive features of the pro- gram. In the eighth case, the report treats a small and unrepresentative set of site visit interviews as if it were a statistically valid sample. Yet the report contains no documentation of the site visits that supplement the quantita- tive analysis, the rationale for visiting “high performing” programs only, or how these visits improved the core analysis and interpretation. One of these studies also did not cite any of the substantial literature on the topic, nor review this body of knowledge.10 The committee sees two issues here. The first—overreaching in describing the study’s goals and findings, and ignoring an existing body of literature—could be corrected if PD&R required adherence to established research standards before the research begins and greater accuracy and precision in reports by everyone who participates in their writing, review- ing, and editing. The second issue—conducting a small number of site visits and interviews—raises a more fundamental question. The committee appreciates that a well-conceived and implemented qualitative research component can provide important information that is useful to policy makers and others interested in understanding how programs are implemented and function and can also help with interpretation of quantitative analysis (see Moffitt, 2000). But qualitative information can also be expensive to collect (e.g., when site visits are required), and it is easily misused. PD&R could some- times obtain better information at lower cost by limiting its customary addi- tion of a small number of site visits and interviews to quantitative analysis projects and making better use of administrative data. This cost-effective alternative would also become more attractive with the continual improve- ment in the scope and quality of administrative data (see Chapter 7).   hese T studies include Interim Evaluation of HUD’s Homeownership Zone Initiative ( ­ Kirchner et al., 2007); Voucher Homeownership Study (Locke et al., 2006); Multifamily Properties: Opting In, Opting Out and Remaining Affordable (Finkel et al., 2006); National Evaluation of the HOPWA Program (Pollack et al., 2000); Study of Homebuyer Activity Through the HOME Investment Partnerships Program (Turnham et al., 2003); and Implica- tions of Project Size in Section 811 and Section 202 Assisted Projects for Persons with Dis- abilities (Locke, Nagler, and Lam, 2004).   nergy Star in HOPE VI Homes (MELE Associates and The Cadmus Group, 2004). E   valuation of the Family Self-Sufficiency Program: Retrospective Analysis, 1996 to 2000 E (Ficke and Piesse, 2004). 10  mplications of Project Size in Section 811 and Section 202 Assisted Projects for Persons I with Disabilities (Locke, Nagler, and Lam, 2004).

56 REBUILDING THE RESEARCH CAPACITY AT HUD SMALL-SCALE, EXPLORATORY STUDIES PD&R has used three vehicles that have the ability in principle to pro- duce small-scale, exploratory studies to highlight emerging issues, test inno- vative data sources and methods, and engage a wider diversity of researchers (primarily in academia). They are small grants on selected topical areas, the “research cadre” initiative described above, and support for dissertations and postdoctoral study, all of which are awarded competitively. These vehicles have produced some outstanding new research. A fine example is a study of changes in local segregation in selected metropolitan areas between 1980 and 2000 by Wong (2006, 2008). Wong developed a series of new measures for measuring diversity in neighborhoods and in nearby neighborhoods, using innovative geographic information systems methods to focus on spatial relationships. He then used the advanced indices in computations of complex changes in segregation from 1980 to 2000 in 30 major metropolitan areas. His work reveals important, intra- metropolitan variations in the level and stability of segregated and diverse neighborhood contexts and advances the understanding of segregation. A number of recent exploratory studies have drawn on HUD pro- gram data, typically as part of the research cadre. Feins and Patterson (2005) described the mobility of voucher recipient families with children during 1995-2002, tracking the same households over the period. This is probably the first study to investigate second and subsequent moves of voucher recipients: previous studies, including major evaluations, have only been able to describe the initial decision, when a family first receives the voucher. The study would probably not have been feasible, or even possible, except perhaps at prohibitive expense, without access to admin- istrative data. The study found that families’ first moves after receiving a voucher tended to be to neighborhoods with slightly higher poverty rates, while subsequent moves tended to be to neighborhoods with lower rates. African Americans were more likely to make such moves than members of other racial and ethnic groups. Other studies have combined HUD program data with other data sources to investigate the effect of housing assistance on earnings and employment. They have found different effects among programs, with voucher recipients having better work experiences than households in public housing or privately owned projects (Olsen et al., 2005; Susin, 2005).11 These studies are exploratory, not definitive. They also could probably not have been conducted without access to administrative data, and they suggest directions for future work using such data. 11  lsen O et al. (2005) used the Labor Department’s Panel Study of Income Dynamics; Susin (2005) used the Census Bureau’s Survey of Income and Program Participation.

EVALUATION OF EXTERNAL RESEARCH 57 It is not possible to systematically assess all the small-scale work sup- ported by HUD over the last decade, or even a representative sample, because HUD does not keep track of all the products (particularly aca- demic journal articles resulting from the research). However, because these projects are small and exploratory, the success of individual projects is less important than the overall effect of the efforts. What is important is that PD&R is using at least a small share of its resources to catalyze research on emerging topics and potentially to draw new researchers into the field of housing and urban development. In recent years, PD&R has made only limited use of these vehicles. For example, since their inception in 1996, the small grants competitions have become more sporadic. Indeed, since 2001 there have been only two such competitions, both in 2004 (see Table 3-3). Moreover, since 2001 there has been steady erosion (in both real and nominal terms) in the amount of funding that PD&R has allocated to support emerging housing and urban scholars in the form of dissertation and postdoctoral grants (see Table 3-4). In 2006, no awards were made, and the planned awards for 2007 represent a decline of 25 percent in nominal dollars when compared with 2005 for both doctoral and early doctoral support combined. TABLE 3-3  PD&R Small Grant Competitions, by Year, Amount, and Focus Year Amount Research Area 1996 $450,000 Fair lending small grants 1997 $350,000 Studies of mortgage purchases 1997 $263,000 NSF small grants 1997 $500,000 Spatial patterns of assisted housing 1998 1999 $750,000 PATH grants 2000 2001 $750,000 NSF-PATH academic grants 2002 2003 2004 $438,000 Changes in urban areas 2004 $507,000 Home ownership grants 2005 2006 2007 NOTE: NSF = National Science Foundation, PATH = Partnership for Advancing Technology in Housing.

58 REBUILDING THE RESEARCH CAPACITY AT HUD TABLE 3-4  Dissertation Grant Awards by Type, Number, and Amount (in thousands of dollars) Doctoral Early Doctorate Postdoctorala Amount Amount Amount Year (000s) Number (000s) Number (000s) Number 1994 225 15 1995 218 15 1996 224 15 1997 253 17 0 0 0 0 1998 225 15 0 0 0 0 1999 240 16 0 0 0 0 2000 450 30 0 0 0 0 2001 588 21 150 10 1,515 ? 2002 385 17 144 10 0 0 2003 400 17 147 10 795 ? 2004 393 16 120 8 400 ? 2005 396 17 143 10 0 0 2006 0 0 0 0 0 0 2007 296 12 103 7 0 0 NOTE: Doctoral and early doctoral grants were made directly by PD&R; postdoctoral grants were made through a contractor. aAmount obligated to the contractor; it does not necessarily indicate the years of award to the ultimate recipient. Postdoctoral awards were for approximately $55,000 each. SOURCE: Unpublished data from HUD, Office of Policy Development and Research. THE AGENDA-SETTING PROCESS AND OVERALL AGENDA The processes used to develop HUD’s funded research agenda limit input from outside the department and constrain PD&R’s access to creative and innovative thinking about both research issues and methodologies. Each year, PD&R staff engage in a structured process for establishing the agenda of external research to be funded. This process includes both formal (from the assistant secretary) and informal (staff level) outreach to HUD’s program offices, inviting ideas for needed studies. The staff then assembles a list of potential studies, including preliminary descriptions of approach, scale, and cost. These “candidate” projects are then assigned priority rank- ings by the PD&R assistant secretary, based in part on meetings with assistant secretaries for the department’s major programs. Finally, the list of studies to be funded is determined on the basis of the priority rankings and available funding. Because funds for external research are limited, the final list may exclude some high-priority studies that are expensive in favor of lower-priority studies that are more affordable.

EVALUATION OF EXTERNAL RESEARCH 59 Although this process provides ample opportunities for input from the program offices within HUD about needed research, it does not include any systematic outreach to congressional staff, the Office of Management and Budget, other federal agencies, advocacy and industry groups, philan- thropic foundations, or academics and other researchers. Broader outreach of this kind would certainly pose some challenges. The process could be very time consuming; it might raise expectations among external audiences that PD&R could not satisfy; some of the research topics identified might be irrelevant to or inconsistent with priorities of the department; and there might be a risk of inappropriately disclosing information to potential bid- ders on competitive procurements. Despite these difficulties, however, a process of broader and more open outreach could potentially broaden the range of research and policy issues addressed, identifying high-priority policy questions relevant to HUD’s mission that go beyond the immediate concerns of the program offices. In addition, outreach of this type might also yield new funding partnerships or increased resources. For example, consultations with congressional staff might result in a supplemental appro- priation to support a high-cost, high-impact study, like the HDS-2000. Consultations with other federal agencies might identify opportunities to jointly fund a project with cross-cutting policy implications, like the Jobs- Plus demonstration. And consultations with major foundations might iden- tify opportunities to leverage PD&R’s resources in support of innovative demonstrations or surveys, like the ­MTO demonstration. PD&R’s research agenda could also be strengthened through more stra- tegic engagement in relevant academic conferences. Although some staff do attend these conferences, PD&R is not systematically represented, either to present the results of its research or to learn more about evolving research methods or emerging findings. If PD&R had a policy of encouraging, finan- cially supporting, and perhaps assigning staff to attend selected conferences on a regular basis, it could help PD&R staff stay up to date on evolving research and methods, find out about promising scholars, gain insight on emerging policy questions, and generate fresh ideas about potential research that HUD should be supporting. PD&R’s shrinking budget constrains staff from conceptualizing a more ambitious, high-impact research agenda. It is entirely understandable that PD&R staff decide not to spend time conceptualizing or planning research projects that the office lacks the resources to support. Paradoxically, how- ever, it is just this kind of research that has the potential to attract addi- tional support from Congress and from foundation partners. To illustrate, the MTO demonstration was mandated by Congress in 1992, with a special appropriation of $70 million to cover the cost of vouchers and accompany- ing mobility counseling. The cost of evaluating MTO is estimated at $22.7 million, of which PD&R has contributed about half ($11.3 million) and the

60 REBUILDING THE RESEARCH CAPACITY AT HUD remainder has been leveraged from foundations and other outside funders. Similarly, the Jobs-Plus demonstration involved about $4.7 million in fund- ing to participating housing agencies for rent incentives, and $25.3 million in research and technical assistance costs. PD&R contributed substantially less than half of the research and technical assistance funding ($9.5 mil- lion), with the remainder coming from the Department of Labor ($0.5 mil- lion), the Department of Health and Human Services ($1.2 million), and foundations ($14.2 million). Like MTO, HDS-2000 was congressionally mandated, and a special allocation of fair housing enforcement funding was earmarked to cover the research costs, which totaled $16.5 million. Although HDS did not leverage funding from foundations or other agen- cies, it did elicit sufficient congressional interest to generate the needed funds for a very ambitious project. In sum, very substantial resources can potentially be mobilized from Congress, other federal agencies, and philan- thropic foundations when PD&R conceptualizes and launches high-impact research initiatives that address fundamental policy issues of importance to the nation. More broadly, HUD’s research agenda over this period has failed to produce rigorous analyses of the effects or cost-effectiveness for many important programs. Although intermediate-scale studies like those that dominate PD&R’s external research portfolio can provide useful informa- tion, this level of funding will almost always be inadequate to answer the core policy questions of a program’s effects and its cost-effectiveness. Throughout its history, HUD has lacked a tradition or expectation that its major programs would be rigorously evaluated on a routine basis. For each of the department’s primary program areas, Table 3-5 lists studies that produced what independent scholars would consider to be reliable estimates of program effects, involving some form of counterfactual (control or com- parison groups) or other statistical controls. Although PD&R has conducted useful studies in all of HUD’s program areas, only a few have used method- ologies that yield rigorous impact or cost-effectiveness estimates, and most of those were focused quite narrowly on a single site or a single outcome of a multifaceted program. For example, the only rigorous evaluation of HUD’s supportive housing programs addressed the effects of supportive housing developments on neighborhood property values and crime in Denver. The only program for which rigorous evaluations have systematically (and repeatedly) produced impact estimates is the housing voucher pro- gram. Devoting substantial evaluation resources to the voucher program is reasonable; it is currently the largest program in the HUD budget.12 How- ever, a number of other important, though smaller, programs and initiatives 12  he T second and third largest programs, public housing and Section 8 new construction, are no longer producing additional housing units on an annual basis.

EVALUATION OF EXTERNAL RESEARCH 61 TABLE 3-5  History of Rigorous Evaluation for HUD’s Major Program Areas Rigorous Evaluations of Program Impacts or Cost Program Area Effectiveness (Year) Public housing Jobs-Plus demonstration—controlled experimental design evaluation of a public housing employment initiative (2005) Statistically controlled estimates of HOPE VI impacts on neighborhood property values (2003) Statistically controlled estimates of impacts of scattered- site public housing on neighborhood property values and crime in Denver (1999) Subsidized rental production Comparative cost study of alternative housing subsidy programs programs, controlling for unit quality (1980) Statistically controlled estimates of impacts of supportive housing developments on neighborhood property values in Denver (2000) Housing vouchers MTO demonstration—random assignment evaluation of relocation to low-poverty neighborhoods (ongoing) Welfare voucher study—random assignment evaluation of impacts for welfare families (2006) Statistically controlled estimates of impacts of voucher families on neighborhood property values in Baltimore County (1999) Experimental Housing Allowance Program—random assignment evaluation of demand-side subsidies (1983) Single-family mortgage insurance programs Homeless assistance programs Community Development Block Statistically controlled estimates of CDBG impacts on Grant (CDBG) and Home property values (2003) Investment Partnerships Program Empowerment Zone and Enterprise Community Program Fair housing grant programs Low-Income Housing Tax Statistically controlled estimates of LIHTC project Credits (LIHTC) impacts on property values (2002) NOTE: This table includes only studies that produced what independent scholars would consider to be reliable estimates of program impacts, involving some form of control or com- parison groups or other statistical controls.

62 REBUILDING THE RESEARCH CAPACITY AT HUD have not been the subject of rigorous evaluations, among them such key programs as empowerment zones and enterprise communities, family self- sufficiency, and home ownership vouchers. PD&R’s ability to implement rigorous studies of program effects and cost-effectiveness is constrained by two important factors. The first is PD&R’s limited funding for external research; rigorous impact evaluations are generally expensive. In the absence of sufficient funding, PD&R staff may have opted for descriptive implementation assessments instead of rig- orous evaluations. In addition, because evaluation mandates are not built into HUD programs, PD&R may have been constrained—at least in some cases—to be able to implement the necessary data collection protocols. CONCLUSIONS AND RECOMMENDATIONS PD&R’s funded research includes many high-quality studies, includ- ing excellent examples in three key categories: (1) large-scale, high-impact s ­ tudies; (2) intermediate-scale policy and program studies; and (3) small- scale exploratory studies. Many studies reviewed by the committee in all three categories meet high standards of relevance, methodological quality, and understandability. PD&R’s external research provides key insights about the demographic, social, and market challenges that HUD programs seek to address as well as about the implementation and operation of these programs. However, for most of the office’s history, the mix of PD&R-funded research has included too few of the ambitious, large-scale studies that answer fundamental questions about impact and effectiveness. Also, too few have produced rigorous estimates of impacts or cost-effectiveness for HUD’s major programs. As a consequence, PD&R has missed opportuni- ties to inform HUD, Congress, and the public about emerging housing and urban development challenges or about the impacts and cost-effectiveness of alternative strategies for addressing these problems. PD&R’s portfolio of funded research is profoundly constrained by both limited appropriations and limited expectations. And due to persistent bud- get constraints, PD&R too often opts for descriptive process evaluations or qualitative assessments of program implementation instead of conducting significant, high-impact studies and evaluations. In part because of its shrinking financial and staff resources, PD&R’s processes for establishing its funded research agenda limit access to creative and innovative thinking about emerging policy challenges, research issues, and methodologies. One way that PD&R has reached out to capture this kind of input has been to invite proposals for small-scale, exploratory s ­ tudies involving a wider diversity of researchers. This has proven to be

EVALUATION OF EXTERNAL RESEARCH 63 a very effective strategy in the past. However, the use of this approach in recent years has been very limited, closing off an important avenue for PD&R to engage with the policy and research community. Major Recommendation 1: PD&R should regularly conduct rigorous evalu- ations of all HUD’s major programs. Recommendation 3-1: Congress and the secretary should assign PD&R responsibility for conducting rigorous, independent evaluations of all major programs and demonstrations and should ensure that the necessary data collection protocols and controls are built into the early stages of program implementation. Recommendation 3-2: Congress should allocate a small fraction of HUD program appropriations to support rigorous evaluations designed and con- ducted by PD&R. Recommendation 3-3: PD&R should design and fund more ambitious, large-scale studies that answer fundamental questions about housing and mortgage markets and about the impact and effectiveness of alternative programs and strategies. As part of this effort, PD&R should launch at least two new large-scale studies annually, partnering with other federal agencies and philanthropic foundations when appropriate. Recommendation 3-4: PD&R should ensure that its research reports adhere to established research standards before the research begins and greater accuracy and precision by everyone who participates in the writing, review- ing, and editing of its reports. Recommendation 3-5: When PD&R designs intermediate-scale studies that do not involve large-scale data collection from a statistically representative sample of agencies or individuals, it should make more effective use of administrative data and limit its use of small (nonrepresentative) samples of site visits and interviews. Recommendation 3-6: PD&R should conduct more small grant compe- titions that invite new research ideas and methods and should increase funding to support emerging housing and urban scholars in the form of dissertation and postdoctoral grants.

Next: 4 Evaluation of Technology Research »
Rebuilding the Research Capacity at HUD Get This Book
×
Buy Paperback | $69.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Today, the nation faces an array of housing and urban policy challenges. No federal department other than HUD focuses explicitly on the well-being of urban places or on the spatial relationships among people and economic activities in urban areas. If HUD, Congress, mayors, and other policy makers are to respond effectively to urban issues, they need a much more robust and effective Office of Policy Development and Research (PD&R).

PD&R conducts independent research and program evaluation, funds data collection and research by outside organizations, and provides policy advice to the Secretary and to other offices in HUD. Most of PD&R's work is of high quality, relevant, timely, and useful. With adequate resources, PD&R could lead the nation's ongoing process of learning, debate, and experimentation about critical housing and urban development challenges.

Rebuilding the Research Capacity at HUD makes seven major recommendations about PD&R's resources and responsibilities, including more active engagement with policy makers, formalizing various informal practices, strengthening surveys and data sets, and more. Acknowledging that the current level of funding for PD&R is inadequate, the book also makes several additional recommendations to help enable PD&R to reach its full potential.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!