3
Evaluation of External Research

INTRODUCTION

Much of the research agenda of PD&R is carried out by outside research organizations that are selected and funded to conduct specific studies. This chapter assesses the quality, timeliness, and usefulness of this external research. Following a brief overview of the processes used to select and supervise external research organizations, the chapter delineates three broad categories of research—large-scale, high-impact research studies; intermediate-scale policy and program studies; and small-scale exploratory studies—and defines criteria for evaluating studies in each category. The chapter then addresses each category in turn, first evaluating individual studies in the category and then assessing the overall portfolio of research projects in the category. Following these assessments, the chapter discusses PD&R’s overall agenda-setting process and the overall agenda for external research. The final section presents the committee’s conclusions and recommendations for external research.

Funds obligated for external research averaged about $30.3 million between 1999 and 2007, ranging from a high of $47.2 million in 2000 to a low of $14.8 million budgeted for 2007 (see Table 2-6 in Chapter 2). Most of this funding comes from research and technology appropriations to PD&R, but additional funding for external research is sometimes provided from either salaries and expense appropriations or program appropriations to other offices in HUD. Research and technology funding obligated for external research dropped precipitously in 2002 and again in 2006.

PD&R staff members, working with representatives from other offices



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 33
3 Evaluation of External Research INTRODUCTION Much of the research agenda of PD&R is carried out by outside research organizations that are selected and funded to conduct specific studies. This chapter assesses the quality, timeliness, and usefulness of this external research. Following a brief overview of the processes used to select and supervise external research organizations, the chapter delineates three broad categories of research—large-scale, high-impact research studies; intermediate-scale policy and program studies; and small-scale exploratory studies—and defines criteria for evaluating studies in each category. The chapter then addresses each category in turn, first evaluating individual studies in the category and then assessing the overall portfolio of research projects in the category. Following these assessments, the chapter discusses PD&R’s overall agenda-setting process and the overall agenda for external research. The final section presents the committee’s conclusions and recom- mendations for external research. Funds obligated for external research averaged about $30.3 million between 1999 and 2007, ranging from a high of $47.2 million in 2000 to a low of $14.8 million budgeted for 2007 (see Table 2-6 in Chapter 2). Most of this funding comes from research and technology appropriations to PD&R, but additional funding for external research is sometimes provided from either salaries and expense appropriations or program appropriations to other offices in HUD. Research and technology funding obligated for external research dropped precipitously in 2002 and again in 2006. PD&R staff members, working with representatives from other offices 

OCR for page 33
 REBUILDING THE RESEARCH CAPACITY AT HUD of HUD, select the topics, define the research questions to be addressed, determine the basics of the methodology to be implemented, and estimate the likely cost of the research. Generally, a research organization is selected competitively by a panel of HUD staff to conduct each study, often through a formal request for proposals and a structured ranking and selection process. The organizations that compete for and conduct PD&R-funded research include for-profit firms, nonprofit research organizations, and (sometimes) academic institutions.1 Recently, competition for many studies has been limited to small businesses. Once a contractor has been selected, PD&R staff monitor progress and performance and review the research products. Almost all of the studies funded by PD&R produce reports that are made available to the public. Most of these reports are published in hard copy by PD&R and dissemi- nated through HUD USER. Exceptions include papers funded by small grants (discussed further below), which are intended for publication in journals, and studies that PD&R did not initiate but provided partial fund- ing for, which are usually published by other sponsors. In addition, a small number of external research projects yield findings or reports that PD&R decides cannot be released because of poor quality.2 PD&R’s standard contract allows the funded research organizations to publish results independently once a study has been completed and follow- ing a set embargo period.3 Consequently, in addition to HUD publications, PD&R-funded research appears in academic journals, conference presen- tations, book chapters, policy briefs, opinion pieces, and congressional testimony. TYPES OF ExTERNAL RESEARCH AND CRITERIA FOR EVALUATION Three basic types of research studies form part of a comprehensive, policy research agenda: large-scale, high-impact studies; intermediate-scale 1 In some cases, PD&R has awarded “indefinite quantity contracts” to several research organizations (selected competitively), which are then tapped for specific, quick-turnaround research projects. 2 PD&R staff identified seven studies funded in recent years that were not published for this reason. In some cases, PD&R staff made revisions themselves and produced a releasable report despite the fact that the contractor’s report was deemed unsatisfactory. 3 Final scopes of work typically include the following language: “Contractors may not pub- lish a report based on this study or otherwise disclose the contents of research reports prepared under this contract to the public for three months following the formal submission of the final report, unless the contracting officer has given written permission. After the three-month period, the Contractor who wishes to publish shall include a clear notice that the research was performed under a contract with the Office of Policy Development and Research, U.S. Department of Housing and Urban Development.”

OCR for page 33
 EVALUATION OF EXTERNAL RESEARCH policy and program studies; and small-scale, exploratory studies. Although these categories of research overlap and the boundaries between them are not always distinct, it is useful to think about PD&R’s sponsored research in this framework. Large-Scale, High-Impact Studies Large-scale, high-impact studies address major enduring policy ques- tions that matter to the public, Congress, and the HUD secretary. Such studies are typically costly (often over $1 million) and take more than a year (sometimes several years) to complete. But more important than their size is the fact that studies in this category are designed and implemented to address fundamental questions about the need for and effectiveness of public interventions—questions that span administrations and help shape long-term public policy development. Important examples of PD&R-funded research of this type include three national studies of the incidence of discrimination against racial and ethnic minorities searching for housing in urban areas. These studies, conducted in 1977, 1989, and 2000, pioneered the use of the “paired-testing” methodol- ogy. Other important examples of PD&R-funded research in this category include the Moving to Opportunity for Fair Housing (MTO) demonstra- tion and the Jobs-Plus demonstration. Both of these studies implemented rigorous, controlled experimental designs to assess the effects of housing interventions on resident self-sufficiency. MTO, which is ongoing, measures the effects of providing vouchers that require low-income families to relo- cate to low-poverty neighborhoods (Orr et al., 2003). Jobs-Plus measures the effects of delivering intensive employment assistance and incentives to residents of public housing developments (Bloom et al., 2005). Intermediate-Scale Policy and Program Studies Moderate-scale studies address significant (but more immediate) issues of program design and implementation or market trends and conditions. Though less costly than the multiyear high-impact projects, these studies still involve substantial investments, often costing hundreds of thousands of dollars and requiring a year or more to complete. PD&R has funded many studies in this category. One example esti- mated the number, characteristics, and risk profile of potential homeown- ers (Galster et al., 1996). This study was the first to use the federal Survey of Income and Program Participation to analyze who among the pool of renters might become homeowners if various conditions were changed by public policies, and how that, in turn, might change the profile of mortgage default risk. Another example was a study of metropolitan areas across the

OCR for page 33
 REBUILDING THE RESEARCH CAPACITY AT HUD country where the federal housing voucher program is administered region- ally, or at least across several jurisdictions (Feins et al., 1996). The latter study identified examples of regional voucher administration, described the historical and policy circumstances that led to it, documented how regional administration was carried out, and assessed potential strengths and weak- nesses of regional administration. Small-Scale, Exploratory Studies Small, exploratory studies investigate new issues, expand the use of new data sets, or engage new researchers. Studies in this category typically cost under $100,000 and are completed within 1 year. Given their relatively small size, they may be quite narrowly focused or provide only preliminary answers, but they can also extend the scope of a policy research agenda into new issue areas or explore innovative methodologies. Beginning in 1997, PD&R initiated “small grant competitions,” inviting researchers to suggest studies around a broad policy theme. Two of these competitions related to the mortgage purchase activities of Fannie Mae and Freddie Mac, and a third explored the topic of socioeconomic change in cities. In 2003 PD&R instituted a different method for securing smaller- scale scholarly research on specific topics relevant to HUD assisted housing programs: the “research cadre.” In this program, PD&R authorized a private contractor to perform all tasks necessary to select and fund a cadre of as many as 20 social science researchers capable of conducting policy research and analysis using HUD’s program administrative data as well as data from other sources. HUD episodically provides the contractor with research topics, and the contractor authorizes a member of the cadre to conduct the work and provides appropriate oversight and project management. Criteria for Evaluation Studies in all three categories should meet a common set of evaluation criteria: (1) relevance and importance of the topic; (2) rigor and appropri- ateness of methodology; (3) timeliness; (4) qualifications of the research team; and (5) quality of the research products. The specifics of these five basic criteria differ somewhat across the three categories of study; Table 3-1 details those differences. LARGE-SCALE, HIGH-IMPACT RESEARCH STUDIES Over the last decade, PD&R has sponsored a small number of very high-quality research initiatives that have rigorously addressed major policy issues of importance to the nation. All four that have produced interim or

OCR for page 33
TABLE 3-1 Criteria for Evaluating PD&R’s External Research Evaluation Criteria Large-Scale, High-Impact Studies Moderate-Scale, Program Studies Small-Scale, Exploratory Studies Relevance and Study should address a clearly Study should address an issue of Individual studies need not be directly importance of topic articulated research question or program design or implementation related to HUD’s current agenda, but hypothesis of major importance to important to HUD’s mission. may be foundational, or exploratory. public policy. Rigor and Basic methodology (selected by Basic methodology (selected by Methodology implemented by a appropriateness of PD&R) should be appropriate to PD&R) should be appropriate to particular scholar or team should be methodology address the study’s research question address the study’s basic objectives; appropriate to address the study’s or hypothesis; details (developed by details (developed by the basic objectives and should yield the contractor) should yield contractor) should yield reliable reliable results at a level of statistically significant, reliable, and results at a level of generalizability generalizability appropriate to the generalizable answers to the study appropriate to the topic and time topic and time frame. questions. frame. Timeliness Not applicable. Study should produce results in the Study should produce results within 1 time frame needed by the relevant year. program office or stakeholders. Qualifications of Research team should include Team selected should include Individual or team selected should research team qualified methodologists, analysts, qualified methodologists, analysts, possess the technical and/or policy and project managers and advised by and project managers. qualifications appropriate for the an outside panel of specialized proposed methodology. experts. Quality of research Products should include complete Products should include complete Products should include a complete products documentation of data and methods; documentation of data and documentation of data and methods; comprehensive reporting of results; methods; comprehensive reporting comprehensive reporting of results; and understandable assessment of of results; and understandable and understandable assessment of the the implications. Quality can also be assessment of the implications. implications. Eventual publication in assessed if scholarly papers are Quality can also be assessed if peer-reviewed journals is expected. published after peer review. scholarly papers are published after peer review. 

OCR for page 33
 REBUILDING THE RESEARCH CAPACITY AT HUD final results are described here, applying the evaluation criteria in Table 3-1: (1) the 2000 Housing Discrimination Study (HDS-2000); (2) the MTO demonstration, which evaluates the effects of assisted housing mobility; (3) the evaluation of the effectiveness of housing vouchers for welfare families; and (4) the Jobs-Plus demonstration, which evaluates the effects of work incentives and supports for public housing residents. In addition to evaluat- ing the quality of these individual, high-impact studies, this section assesses the mix of studies sponsored over the years and the extent to which this mix addresses the information needs of HUD and the larger housing and urban development policy community. The 2000 Housing Discrimination Study Since the 1960s, advocates for fair and open housing have used a technique called paired testing to detect and reveal discrimination by real estate and rental agents. In a paired test, two individuals—one white and the other minority—pose as equally qualified home seekers. Both testers are carefully trained to make the same inquiries, express the same prefer- ences, and offer the same qualifications and needs. From the perspective of the housing provider they visit, the only difference between the two is their race or ethnicity, and they should therefore receive the same information and assistance. Systematic differences in treatment—telling the minority customer that an apartment is no longer available when the white customer is told he could move in next month, for example—provide powerful evi- dence, easily understandable by the general public, of discrimination that denies minorities equal access to housing. When a large number of consistent and comparable tests are conducted for a representative sample of real estate or rental agents, the results con- trol for differences between white and minority customers, and directly measure the prevalence of discrimination across the housing market as a whole. PD&R recognized the potential of the paired testing methodology as a research tool and has used it to monitor the incidence of housing dis- crimination nationwide at roughly 10-year intervals. The 1977 Housing Market Practices Study provided the first solid esti- mates of the prevalence of discrimination against African American home seekers (Wienk et al., 1979) and helped build the case for strengthening the enforcement of federal fair housing protections in the 1988 Fair Housing Act Amendments. The 1989 Housing Discrimination Study extended those initial national estimates to cover Hispanics and concluded that overall lev- els of adverse treatment against African Americans had remained essentially unchanged since 1977 (Turner, Struyk, and Yinger, 1991). Most recently, HDS-2000 reported the change since 1989 in discrimination against African Americans and Hispanics and up-to-date estimates of the incidence of dis-

OCR for page 33
 EVALUATION OF EXTERNAL RESEARCH crimination, including the first national estimates of discrimination against Asians and Pacific Islanders and the first rigorous estimates of discrimina- tion against Native Americans searching for housing outside of native lands (Turner et al., 2002b; Turner and Ross, 2003a, 2003b). Funding for HDS-2000 was allocated by Congress from annual appro- priations to the Fair Housing Initiatives Program, and PD&R was assigned responsibility for study design and selection of the research team. The request for proposals (RFP) envisioned three phases of paired testing, with the first phase focusing on estimates of change in the incidence of discrimination against African Americans and Hispanics in metropolitan areas nationwide and subsequent phases focusing on other minority groups or nonmetropolitan communities. In addition, the RFP called for a sample design that would both measure change at the national level and provide reliable estimates of the incidence of discrimination for individual metro- politan areas. PD&R selected a team led by the Urban Institute to con- duct HDS-2000. This team included staff and consultants who had been involved in previous paired testing studies and had extensive expertise in fair housing issues, the paired testing methodology, sampling methodolo- gies, and management of large-scale field data collection. Each of the study’s three phases involved selection of a representative sample of metropolitan areas in which testing was conducted, selection of representative samples of advertised housing units in these metropolitan areas, highly standardized paired testing protocols, and rigorous statisti- cal analysis. Reports for each phase were published by HUD, and include complete documentation of sampling and statistical procedures and paired testing protocols (Turner et al., 1991, 2002b; Turner and Ross, 2003a, 2003b). Findings from HDS-2000 have been presented at academic and practitioner conferences, and summarized in several book chapters and journal articles. Assisted Housing Mobility Authorized by Congress in 1992, the MTO demonstration provided tenant-based rental assistance and housing search and counseling services to families living in high-poverty public and assisted developments, in order to assess the effects of neighborhood conditions on educational and employment outcomes. MTO was inspired by findings from the Gautreaux demonstration, which provided special-purpose vouchers to enable African American families (who either lived in public housing or were eligible for it) to move to predominantly white or racially mixed neighborhoods in the city of Chicago and surrounding suburban communities. This program was designed as part of the court-ordered legal remedy for systematic dis- crimination and segregation of Chicago’s public housing program. Research

OCR for page 33
0 REBUILDING THE RESEARCH CAPACITY AT HUD on Gautreaux families suggested that many of the families who moved to suburban neighborhoods and stayed there experienced substantial benefits over time. PD&R convened a panel of academics, policy experts, and practitioners to help develop the basic demonstration design for MTO. MTO’s experi- mental design randomly assigned eligible families, who volunteered to par- ticipate, to one of three groups. The experimental group received Section 8 certificates or vouchers usable only in low-poverty census tracts (defined as under 10 percent poor in 1990) and assistance in finding a unit and moving. The comparison group received regular Section 8 certificates or vouchers, which had no geographical restrictions and which did not pro- vide search assistance. A control group continued to receive project-based assistance. PD&R then competitively selected a contractor (Abt Associates, Inc.) to manage the demonstration operations, including baseline data collec- tion, random assignment, monitoring counseling operations, and track- ing household outcomes. The Abt Associates team was well qualified for this assignment, consisting of sampling and survey specialists, experts in experimental design demonstrations, and staff with extensive experience in the operations of public housing agencies and the voucher program. Dur- ing the early years of the demonstration, small grants were also awarded competitively to academic researchers in the demonstration sites who proposed innovative, exploratory studies of the relocation and neighbor- hood adjustment process. These grants engaged a pool of distinguished academics from fields other than housing in the ongoing demonstration research effort. In 1999, PD&R issued an RFP for an interim evaluation of demonstration impacts. This contract was awarded to a team led by Abt Associates, but also including researchers from the Urban Institute and the National Bureau of Economic Research (NBER). Members of this research team secured substantial additional funding from foundations and the National Institutes of Health for the interim evaluation. Finally, in 2006, PD&R issued an RFP for a final evaluation of MTO. This contract was also awarded to NBER. Both the interim and final evaluations use a combination of admin- istrative data and follow-up surveys of experimental, comparison, and control households. They rigorously measure MTO “treatment” effects by comparing outcomes for experimental, comparison, and control groups over time. The interim evaluation results are fully documented in a report by Abt and NBER researchers published by HUD (Orr et al., 2003). In addition, numerous site-specific studies using a range of data collection and analytic methods have been conducted and continue to be conducted with foundation funding. This research has been published in numerous working papers, policy briefs, a book, and academic journal articles. Links to most

OCR for page 33
 EVALUATION OF EXTERNAL RESEARCH of the published studies and reports are provided on a website (http://www. mtoresearch.org). To date, the evaluation research has found that the MTO treatment enabled families to move to dramatically safer neighborhoods with lower poverty rates and more neighbors who are working. However, most of these neighborhoods are majority-minority and located within central city juris- dictions. The MTO treatment has resulted in significant improvements in the physical and mental health of women and girls. However, no significant gains in employment, earnings, or educational outcomes were found across the five demonstration sites, and delinquent behavior among boys appears to have increased among experimental families. All of these outcomes are currently being reassessed as part of the final evaluation, which should be completed by 2010. Housing Vouchers for Welfare Families In 1999 Congress passed a special appropriation of housing vouchers for a demonstration initiative targeted specifically to families making the transition from welfare to work. Public housing agencies were competitively selected to participate in this demonstration, based on locally designed strategies for coordinating housing assistance with welfare reform and welfare-to-work initiatives. The appropriation for this Welfare to Work Voucher cemonstration provided a 1 percent set-aside for evaluating the effect of housing assistance on welfare families under the demonstration. From the outset, PD&R planned a random assignment, experimen- tal design methodology for this demonstration. First, a contractor was competitively selected from among existing indefinite quantity contract holders to develop the evaluation methodology, design and conduct random assignment of applicants, develop data collection methods and instruments, and conduct baseline data collection. Then a separate RFP was issued to select a contractor to implement the full evaluation methodology, includ- ing all post-test data collection and analysis. Abt Associates, Inc. won both of these competitive procurements. The Abt team was extremely well qualified to conduct the welfare voucher evaluation; the company had staff with long-standing expertise in the voucher program, random assignment demonstrations, sampling and statistical procedures, survey design and implementation, and administrative data collection. This team designed and implemented a rigorous evaluation methodology, which made excellent use of both household survey data and administrative data on individual demonstration participants. Abt researchers were simultaneously involved in the MTO evaluation and in a panel survey of public housing families

OCR for page 33
 REBUILDING THE RESEARCH CAPACITY AT HUD relocating from HOPE VI4 developments, and they were able to incorporate methods and lessons from these initiatives into the design of the Welfare to Work Voucher demonstration and evaluation. Abt Associates completed two major reports on the Welfare to Work Voucher demonstration—an interim report to Congress in 2004 (Patterson et al., 2004) and a final report in 2006 (Mills et al., 2006). These were both published by PD&R. The findings from these studies provide important new evidence on the effects of housing voucher receipt on key outcomes for welfare families. Specifically, the evaluation found that receiving a housing voucher resulted in small improvements in neighborhood conditions among welfare families, enabled welfare mothers and their children to live indepen- dently rather than doubling up or living in multigenerational households, dramatically reduced the incidence of homelessness, and increased spending on food. The research also found that receiving a voucher initially reduced recipients’ employment and earnings, but after a year or two this negative effect disappeared: over a 3.5-year period, there was no significant effect of voucher receipt on employment and earnings. These findings, and the details of the demonstration design and evaluation methods, are fully docu- mented and clearly explained in the HUD reports. To date, findings from this research have not appeared in academic journals or books. Work Incentives and Supports in Public Housing (Jobs-Plus) In the mid-1990s, as debates over welfare reform were under way, representatives from the Rockefeller Foundation and the Manpower Dem- onstration Research Corporation (MDRC) approached the PD&R assis- tant Secretary to explore ideas for promoting work and self-sufficiency in public housing developments where unemployment and rates of welfare receipt were extremely high. Together, the three organizations developed the basic framework for the Jobs-Plus demonsration, which was ulti- mately implemented in randomly selected public housing developments in five cities, with randomly selected comparison developments in each city to allow for rigorous estimates of the impact of saturation services and incentives. Jobs-Plus was designed to test the impact of a saturation intervention that included work incentives, employment services, and community supports for work on employment and earnings among pub- 4 Launched in 1992, the HOPE VI Revitalization of Severely Distressed Public Housing Program replaces severely distressed projects with redesigned mixed-income developments and provides housing vouchers to enable the original residents to rent apartments in the private market. It is the department’s most extensive effort to address the problems in some public housing projects and to reduce concentrations of poverty.

OCR for page 33
 EVALUATION OF EXTERNAL RESEARCH lic housing residents and on employment rates and community health in public housing developments. Because the concept and design of Jobs-Plus were jointly developed and the Rockefeller Foundation was providing substantial funding, PD&R entered into a sole-source, cooperative agreement with MDRC for all phases of the demonstration design, implementation, monitoring, and evaluation. MDRC was well qualified for this role. Although the organization did not have experience with federal housing programs, it had an outstanding track record of designing and implementing controlled experimental design demonstrations of welfare and employment initiatives. The project staff included well-qualified statistical, sampling, and survey methodologists and experts in administrative data assembly and analysis, as well as personnel with extensive experience in the implementation of demonstration initia- tives, and the evaluation design for the demonstration implemented creative methods for using administrative data on residents of both treatment and control sites from before and after the intervention to rigorously measure the impact of “saturating” public housing developments with services and incentives. Over roughly a decade, MDRC published 11 formal reports on Jobs- Plus design, implementation, and results, culminating in a final report on the demonstration’s impacts on employment, earnings, and neighborhood health (Bloom et al., 2005). MDRC researchers have also published sev- eral journal articles on the evaluation design and findings. The MDRC research concludes that, when effectively implemented, the Jobs-Plus model (combining work incentives, employment services, and community sup- ports for work) results in significant increases in individual earnings. These earnings gains stem in part from increased employment rates, but also from increased hours and wages among working adults. Despite the gains in earnings, however, Jobs-Plus had no measurable effects on the overall employment rate in the targeted projects or on other indicators of commu- nity health or quality of life. Assessment of the Overall Portfolio of High-Impact Research Studies Although PD&R has conducted very high-quality studies of this type, the mix of PD&R-funded research includes too few of the ambitious, large- scale studies that answer fundamental questions of impact and effectiveness. This has been true throughout PD&R’s history, and only four or five studies sponsored since 1997 fall into this category: those discussed above and possibly a recently initiated evaluation of housing counseling.5 Only two of 5 Withincreasing interest in counseling by both the administration and Congress and large percentage increases in funding, PD&R initiated an evaluation of HUD-approved counsel-

OCR for page 33
 REBUILDING THE RESEARCH CAPACITY AT HUD TABLE 3-2 Continued Date Published Report Title Funded Contractor Subcontractor (Date of Publication) 2002 Mele The Cadmus Energy Star in HOPE VI Homes (October Associates Group 2004) 2002 Abt Associates Implications of Project Size in Section  and Section 0 Assisted Projects for Persons with Disabilities (March 2004) 2003 ESI Abt Associates The State of Affordable Housing in the U.S.—000 (November 2004) (draft final report) 2004 Abt Associates Newport Partners Voucher Homeownership Study (March 2006) 2004 Exceed Corp. RTI International Interim Ealuation of HUD’s Homeownership Zone Initiatie (March 2007) 2004 Econometrica Abt Associates Multifamily Properties: Opting In, Opting Out and Remaining Affordable (January 2006) 2005 Building ARES; M. Green A Methodology for Identifying, Discussing Technology, & Assoc.; Koffel and Analyzing the Costs and Benefits of Inc. Assoc.; SPA Risk; Code Changes That Impact Housing and Institute for (March 2007) Building Technology and Safety provide important insights into the operations of the programs that were studied. Nine of the 17 studies meet generally accepted research standards, and the study’s approach and findings are appropriately described in the report’s narrative, foreword, and preface. In 8 of the 17 studies, however, one or more of the findings go beyond the limits of the study design with the potential to mislead readers. For example, in six cases, there are refer- ences to the “effects” or “impacts” of the program or overly general state- ments about the functioning of the program or client satisfaction with the program based on a small number of case studies or on cross-sectional data

OCR for page 33
 EVALUATION OF EXTERNAL RESEARCH with no control or comparison group.7 In another case, the design of the convenience sample may have overstated the positive features of the pro- gram.8 In the eighth case, the report treats a small and unrepresentative set of site visit interviews as if it were a statistically valid sample. Yet the report contains no documentation of the site visits that supplement the quantita- tive analysis, the rationale for visiting “high performing” programs only, or how these visits improved the core analysis and interpretation.9 One of these studies also did not cite any of the substantial literature on the topic, nor review this body of knowledge.10 The committee sees two issues here. The first—overreaching in describing the study’s goals and findings, and ignoring an existing body of literature—could be corrected if PD&R required adherence to established research standards before the research begins and greater accuracy and precision in reports by everyone who participates in their writing, review- ing, and editing. The second issue—conducting a small number of site visits and interviews—raises a more fundamental question. The committee appreciates that a well-conceived and implemented qualitative research component can provide important information that is useful to policy makers and others interested in understanding how programs are implemented and function and can also help with interpretation of quantitative analysis (see Moffitt, 2000). But qualitative information can also be expensive to collect (e.g., when site visits are required), and it is easily misused. PD&R could some- times obtain better information at lower cost by limiting its customary addi- tion of a small number of site visits and interviews to quantitative analysis projects and making better use of administrative data. This cost-effective alternative would also become more attractive with the continual improve- ment in the scope and quality of administrative data (see Chapter 7). 7 These studies include Interim Ealuation of HUD’s Homeownership Zone Initiatie (Kirchner et al., 2007); Voucher Homeownership Study (Locke et al., 2006); Multifamily Properties: Opting In, Opting Out and Remaining Affordable (Finkel et al., 2006); National Ealuation of the HOPWA Program (Pollack et al., 2000); Study of Homebuyer Actiity Through the HOME Inestment Partnerships Program (Turnham et al., 2003); and Implica- tions of Project Size in Section  and Section 0 Assisted Projects for Persons with Dis- abilities (Locke, Nagler, and Lam, 2004). 8 Energy Star in HOPE VI Homes (MELE Associates and The Cadmus Group, 2004). 9 Ealuation of the Family Self-Sufficiency Program: Retrospectie Analysis,  to 000 (Ficke and Piesse, 2004). 10 Implications of Project Size in Section  and Section 0 Assisted Projects for Persons with Disabilities (Locke, Nagler, and Lam, 2004).

OCR for page 33
 REBUILDING THE RESEARCH CAPACITY AT HUD SMALL-SCALE, ExPLORATORY STUDIES PD&R has used three vehicles that have the ability in principle to pro- duce small-scale, exploratory studies to highlight emerging issues, test inno- vative data sources and methods, and engage a wider diversity of researchers (primarily in academia). They are small grants on selected topical areas, the “research cadre” initiative described above, and support for dissertations and postdoctoral study, all of which are awarded competitively. These vehicles have produced some outstanding new research. A fine example is a study of changes in local segregation in selected metropolitan areas between 1980 and 2000 by Wong (2006, 2008). Wong developed a series of new measures for measuring diversity in neighborhoods and in nearby neighborhoods, using innovative geographic information systems methods to focus on spatial relationships. He then used the advanced indices in computations of complex changes in segregation from 1980 to 2000 in 30 major metropolitan areas. His work reveals important, intra- metropolitan variations in the level and stability of segregated and diverse neighborhood contexts and advances the understanding of segregation. A number of recent exploratory studies have drawn on HUD pro- gram data, typically as part of the research cadre. Feins and Patterson (2005) described the mobility of voucher recipient families with children during 1995-2002, tracking the same households over the period. This is probably the first study to investigate second and subsequent moves of voucher recipients: previous studies, including major evaluations, have only been able to describe the initial decision, when a family first receives the voucher. The study would probably not have been feasible, or even possible, except perhaps at prohibitive expense, without access to admin- istrative data. The study found that families’ first moves after receiving a voucher tended to be to neighborhoods with slightly higher poverty rates, while subsequent moves tended to be to neighborhoods with lower rates. African Americans were more likely to make such moves than members of other racial and ethnic groups. Other studies have combined HUD program data with other data sources to investigate the effect of housing assistance on earnings and employment. They have found different effects among programs, with voucher recipients having better work experiences than households in public housing or privately owned projects (Olsen et al., 2005; Susin, 2005).11 These studies are exploratory, not definitive. They also could probably not have been conducted without access to administrative data, and they suggest directions for future work using such data. 11 Olsen et al. (2005) used the Labor Department’s Panel Study of Income Dynamics; Susin (2005) used the Census Bureau’s Survey of Income and Program Participation.

OCR for page 33
 EVALUATION OF EXTERNAL RESEARCH It is not possible to systematically assess all the small-scale work sup- ported by HUD over the last decade, or even a representative sample, because HUD does not keep track of all the products (particularly aca- demic journal articles resulting from the research). However, because these projects are small and exploratory, the success of individual projects is less important than the overall effect of the efforts. What is important is that PD&R is using at least a small share of its resources to catalyze research on emerging topics and potentially to draw new researchers into the field of housing and urban development. In recent years, PD&R has made only limited use of these vehicles. For example, since their inception in 1996, the small grants competitions have become more sporadic. Indeed, since 2001 there have been only two such competitions, both in 2004 (see Table 3-3). Moreover, since 2001 there has been steady erosion (in both real and nominal terms) in the amount of funding that PD&R has allocated to support emerging housing and urban scholars in the form of dissertation and postdoctoral grants (see Table 3-4). In 2006, no awards were made, and the planned awards for 2007 represent a decline of 25 percent in nominal dollars when compared with 2005 for both doctoral and early doctoral support combined. TABLE 3-3 PD&R Small Grant Competitions, by Year, Amount, and Focus Year Amount Research Area 1996 $450,000 Fair lending small grants 1997 $350,000 Studies of mortgage purchases 1997 $263,000 NSF small grants 1997 $500,000 Spatial patterns of assisted housing 1998 1999 $750,000 PATH grants 2000 2001 $750,000 NSF-PATH academic grants 2002 2003 2004 $438,000 Changes in urban areas 2004 $507,000 Home ownership grants 2005 2006 2007 NOTE: NSF = National Science Foundation, PATH = Partnership for Advancing Technology in Housing.

OCR for page 33
 REBUILDING THE RESEARCH CAPACITY AT HUD TABLE 3-4 Dissertation Grant Awards by Type, Number, and Amount (in thousands of dollars) Postdoctorala Doctoral Early Doctorate Amount Amount Amount Year (000s) Number (000s) Number (000s) Number 1994 225 15 1995 218 15 1996 224 15 1997 253 17 0 0 0 0 1998 225 15 0 0 0 0 1999 240 16 0 0 0 0 2000 450 30 0 0 0 0 2001 588 21 150 10 1,515 ? 2002 385 17 144 10 0 0 2003 400 17 147 10 795 ? 2004 393 16 120 8 400 ? 2005 396 17 143 10 0 0 2006 0 0 0 0 0 0 2007 296 12 103 7 0 0 NOTE: Doctoral and early doctoral grants were made directly by PD&R; postdoctoral grants were made through a contractor. aAmount obligated to the contractor; it does not necessarily indicate the years of award to the ultimate recipient. Postdoctoral awards were for approximately $55,000 each. SOURCE: Unpublished data from HUD, Office of Policy Development and Research. THE AGENDA-SETTING PROCESS AND OVERALL AGENDA The processes used to develop HUD’s funded research agenda limit input from outside the department and constrain PD&R’s access to creative and innovative thinking about both research issues and methodologies. Each year, PD&R staff engage in a structured process for establishing the agenda of external research to be funded. This process includes both formal (from the assistant secretary) and informal (staff level) outreach to HUD’s program offices, inviting ideas for needed studies. The staff then assembles a list of potential studies, including preliminary descriptions of approach, scale, and cost. These “candidate” projects are then assigned priority rank- ings by the PD&R assistant secretary, based in part on meetings with assistant secretaries for the department’s major programs. Finally, the list of studies to be funded is determined on the basis of the priority rankings and available funding. Because funds for external research are limited, the final list may exclude some high-priority studies that are expensive in favor of lower-priority studies that are more affordable.

OCR for page 33
 EVALUATION OF EXTERNAL RESEARCH Although this process provides ample opportunities for input from the program offices within HUD about needed research, it does not include any systematic outreach to congressional staff, the Office of Management and Budget, other federal agencies, advocacy and industry groups, philan- thropic foundations, or academics and other researchers. Broader outreach of this kind would certainly pose some challenges. The process could be very time consuming; it might raise expectations among external audiences that PD&R could not satisfy; some of the research topics identified might be irrelevant to or inconsistent with priorities of the department; and there might be a risk of inappropriately disclosing information to potential bid- ders on competitive procurements. Despite these difficulties, however, a process of broader and more open outreach could potentially broaden the range of research and policy issues addressed, identifying high-priority policy questions relevant to HUD’s mission that go beyond the immediate concerns of the program offices. In addition, outreach of this type might also yield new funding partnerships or increased resources. For example, consultations with congressional staff might result in a supplemental appro- priation to support a high-cost, high-impact study, like the HDS-2000. Consultations with other federal agencies might identify opportunities to jointly fund a project with cross-cutting policy implications, like the Jobs- Plus demonstration. And consultations with major foundations might iden- tify opportunities to leverage PD&R’s resources in support of innovative demonstrations or surveys, like the MTO demonstration. PD&R’s research agenda could also be strengthened through more stra- tegic engagement in relevant academic conferences. Although some staff do attend these conferences, PD&R is not systematically represented, either to present the results of its research or to learn more about evolving research methods or emerging findings. If PD&R had a policy of encouraging, finan- cially supporting, and perhaps assigning staff to attend selected conferences on a regular basis, it could help PD&R staff stay up to date on evolving research and methods, find out about promising scholars, gain insight on emerging policy questions, and generate fresh ideas about potential research that HUD should be supporting. PD&R’s shrinking budget constrains staff from conceptualizing a more ambitious, high-impact research agenda. It is entirely understandable that PD&R staff decide not to spend time conceptualizing or planning research projects that the office lacks the resources to support. Paradoxically, how- ever, it is just this kind of research that has the potential to attract addi- tional support from Congress and from foundation partners. To illustrate, the MTO demonstration was mandated by Congress in 1992, with a special appropriation of $70 million to cover the cost of vouchers and accompany- ing mobility counseling. The cost of ealuating MTO is estimated at $22.7 million, of which PD&R has contributed about half ($11.3 million) and the

OCR for page 33
0 REBUILDING THE RESEARCH CAPACITY AT HUD remainder has been leveraged from foundations and other outside funders. Similarly, the Jobs-Plus demonstration involved about $4.7 million in fund- ing to participating housing agencies for rent incentives, and $25.3 million in research and technical assistance costs. PD&R contributed substantially less than half of the research and technical assistance funding ($9.5 mil- lion), with the remainder coming from the Department of Labor ($0.5 mil- lion), the Department of Health and Human Services ($1.2 million), and foundations ($14.2 million). Like MTO, HDS-2000 was congressionally mandated, and a special allocation of fair housing enforcement funding was earmarked to cover the research costs, which totaled $16.5 million. Although HDS did not leverage funding from foundations or other agen- cies, it did elicit sufficient congressional interest to generate the needed funds for a very ambitious project. In sum, very substantial resources can potentially be mobilized from Congress, other federal agencies, and philan- thropic foundations when PD&R conceptualizes and launches high-impact research initiatives that address fundamental policy issues of importance to the nation. More broadly, HUD’s research agenda over this period has failed to produce rigorous analyses of the effects or cost-effectiveness for many important programs. Although intermediate-scale studies like those that dominate PD&R’s external research portfolio can provide useful informa- tion, this level of funding will almost always be inadequate to answer the core policy questions of a program’s effects and its cost-effectiveness. Throughout its history, HUD has lacked a tradition or expectation that its major programs would be rigorously evaluated on a routine basis. For each of the department’s primary program areas, Table 3-5 lists studies that produced what independent scholars would consider to be reliable estimates of program effects, involving some form of counterfactual (control or com- parison groups) or other statistical controls. Although PD&R has conducted useful studies in all of HUD’s program areas, only a few have used method- ologies that yield rigorous impact or cost-effectiveness estimates, and most of those were focused quite narrowly on a single site or a single outcome of a multifaceted program. For example, the only rigorous evaluation of HUD’s supportive housing programs addressed the effects of supportive housing developments on neighborhood property values and crime in Denver. The only program for which rigorous evaluations have systematically (and repeatedly) produced impact estimates is the housing voucher pro- gram. Devoting substantial evaluation resources to the voucher program is reasonable; it is currently the largest program in the HUD budget.12 How- ever, a number of other important, though smaller, programs and initiatives 12 The second and third largest programs, public housing and Section 8 new construction, are no longer producing additional housing units on an annual basis.

OCR for page 33
 EVALUATION OF EXTERNAL RESEARCH TABLE 3-5 History of Rigorous Evaluation for HUD’s Major Program Areas Rigorous Evaluations of Program Impacts or Cost Program Area Effectiveness (Year) Public housing Jobs-Plus demonstration—controlled experimental design evaluation of a public housing employment initiative (2005) Statistically controlled estimates of HOPE VI impacts on neighborhood property values (2003) Statistically controlled estimates of impacts of scattered- site public housing on neighborhood property values and crime in Denver (1999) Subsidized rental production Comparative cost study of alternative housing subsidy programs programs, controlling for unit quality (1980) Statistically controlled estimates of impacts of supportive housing developments on neighborhood property values in Denver (2000) Housing vouchers MTO demonstration—random assignment evaluation of relocation to low-poverty neighborhoods (ongoing) Welfare voucher study—random assignment evaluation of impacts for welfare families (2006) Statistically controlled estimates of impacts of voucher families on neighborhood property values in Baltimore County (1999) Experimental Housing Allowance Program—random assignment evaluation of demand-side subsidies (1983) Single-family mortgage insurance programs Homeless assistance programs Community Development Block Statistically controlled estimates of CDBG impacts on Grant (CDBG) and HOME property values (2003) Investment Partnerships Program Empowerment Zone and Enterprise Community Program Fair housing grant programs Low-Income Housing Tax Statistically controlled estimates of LIHTC project Credits (LIHTC) impacts on property values (2002) NOTE: This table includes only studies that produced what independent scholars would consider to be reliable estimates of program impacts, involving some form of control or com- parison groups or other statistical controls.

OCR for page 33
 REBUILDING THE RESEARCH CAPACITY AT HUD have not been the subject of rigorous evaluations, among them such key programs as empowerment zones and enterprise communities, family self- sufficiency, and home ownership vouchers. PD&R’s ability to implement rigorous studies of program effects and cost-effectiveness is constrained by two important factors. The first is PD&R’s limited funding for external research; rigorous impact evaluations are generally expensive. In the absence of sufficient funding, PD&R staff may have opted for descriptive implementation assessments instead of rig- orous evaluations. In addition, because evaluation mandates are not built into HUD programs, PD&R may have been constrained—at least in some cases—to be able to implement the necessary data collection protocols. CONCLUSIONS AND RECOMMENDATIONS PD&R’s funded research includes many high-quality studies, includ- ing excellent examples in three key categories: (1) large-scale, high-impact studies; (2) intermediate-scale policy and program studies; and (3) small- scale exploratory studies. Many studies reviewed by the committee in all three categories meet high standards of relevance, methodological quality, and understandability. PD&R’s external research provides key insights about the demographic, social, and market challenges that HUD programs seek to address as well as about the implementation and operation of these programs. However, for most of the office’s history, the mix of PD&R-funded research has included too few of the ambitious, large-scale studies that answer fundamental questions about impact and effectiveness. Also, too few have produced rigorous estimates of impacts or cost-effectiveness for HUD’s major programs. As a consequence, PD&R has missed opportuni- ties to inform HUD, Congress, and the public about emerging housing and urban development challenges or about the impacts and cost-effectiveness of alternative strategies for addressing these problems. PD&R’s portfolio of funded research is profoundly constrained by both limited appropriations and limited expectations. And due to persistent bud- get constraints, PD&R too often opts for descriptive process evaluations or qualitative assessments of program implementation instead of conducting significant, high-impact studies and evaluations. In part because of its shrinking financial and staff resources, PD&R’s processes for establishing its funded research agenda limit access to creative and innovative thinking about emerging policy challenges, research issues, and methodologies. One way that PD&R has reached out to capture this kind of input has been to invite proposals for small-scale, exploratory studies involving a wider diversity of researchers. This has proven to be

OCR for page 33
 EVALUATION OF EXTERNAL RESEARCH a very effective strategy in the past. However, the use of this approach in recent years has been very limited, closing off an important avenue for PD&R to engage with the policy and research community. Major Recommendation 1: PD&R should regularly conduct rigorous evalu- ations of all HUD’s major programs. Recommendation 3-1: Congress and the secretary should assign PD&R responsibility for conducting rigorous, independent evaluations of all major programs and demonstrations and should ensure that the necessary data collection protocols and controls are built into the early stages of program implementation. Recommendation 3-2: Congress should allocate a small fraction of HUD program appropriations to support rigorous evaluations designed and con- ducted by PD&R. Recommendation 3-3: PD&R should design and fund more ambitious, large-scale studies that answer fundamental questions about housing and mortgage markets and about the impact and effectiveness of alternative programs and strategies. As part of this effort, PD&R should launch at least two new large-scale studies annually, partnering with other federal agencies and philanthropic foundations when appropriate. Recommendation 3-4: PD&R should ensure that its research reports adhere to established research standards before the research begins and greater accuracy and precision by everyone who participates in the writing, review- ing, and editing of its reports. Recommendation 3-5: When PD&R designs intermediate-scale studies that do not involve large-scale data collection from a statistically representative sample of agencies or individuals, it should make more effective use of administrative data and limit its use of small (nonrepresentative) samples of site visits and interviews. Recommendation 3-6: PD&R should conduct more small grant compe- titions that invite new research ideas and methods and should increase funding to support emerging housing and urban scholars in the form of dissertation and postdoctoral grants.

OCR for page 33