Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume I, Review and Recommendations (1991)

Since the inception of the U.S. federal system in 1789, decision makers have sought social welfare policy information of the type we describe, and such information has helped shape many important political debates over the decades.¹ James Madison, in 1789, suggested that the content of the decennial census be expanded to provide information for decision making in addition to fulfilling its constitutionally mandated role in congressional reapportionment. He observed that Congress

Congress added questions on age and sex to the 1790 census schedule, and, by the time of the Civil War, the questionnaire included items on citizenship, education, disability, marital status, place of birth, industry, occupation, and value of real estate owned (Bureau of the Census, 1970:4).

Information about industrial practices and their effects on the population that was gathered by the great "muckraking" journalists at the turn of the century, including Ray Stannard Baker, Upton Sinclair, Lincoln Steffens, and Ida Tarbell,

played a role in many Progressive Era legislative initiatives, such as the Pure Food and Drug Act of 1906, the Meat Inspection Act of 1906, and the 1906 Hepburn Act that strengthened the regulatory powers of the Interstate Commerce Commission (Morris, 1953:267-268). Yet, throughout most of U.S. history, the limitations of available data, research knowledge, and computational methods and machinery greatly constrained the extent of useful policy information, and the demand for such information remained largely ad hoc in nature.

Beginning in the 1960s, quantum improvements in data sources, socioeconomic research, and computing technology made it possible to supply information of much greater depth and breadth for the policy process. In turn, the activist posture of the federal government with regard to social welfare policies—Medicare, the expansion of social security coverage and benefits, and the Supplemental Security Income program for the low-income elderly and disabled are only a few examples of the legislative initiatives of that period—both stimulated the production of policy research and analysis and drew on its results.

Today, policy information plays a much greater role at every stage of the political process than ever before in U.S. history. At one end of the process, the results of research studies documenting social needs typically play a part in identifying problems and moving them onto the federal agenda. At the other end, sophisticated evaluations of government programs often lead to further policy initiatives. In the middle stage of the process, the policy community in Washington takes for granted that neither the administration nor the Congress will consider legislation to alter any of the nation's social welfare programs or the tax code without looking closely at "the numbers," and these numbers, in many instances, are the product of team efforts to apply formal modeling techniques to the task of estimating the effects of alternative policies.

By formal, we mean models that are based on a coherent modeling strategy and set of assumptions, developed for repeated application, and designed to produce consistent estimates for a range of policy proposals within a common framework that is, or can be, well documented and evaluated. By their nature, formal models circumscribe, although they do not eliminate, the role of individual analysts' judgments. Such models—macroeconomic and microsimulation models as well as several other major types—vary in size, scope, and the types of data and modeling strategies they use, but they share the attributes we have listed.

Formal models, as we have defined them, are at one extreme of a continuum of policy analysis tools. At the other extreme are very simplistic and idiosyncratic "back-of-the-envelope" calculations, which an individual analyst might develop at the outset of a policy debate as a very rough guide to the likely effects of a proposed policy. In between are models that are developed by an analyst on an ad hoc basis—often using personal computer spreadsheets—to respond to a specific policy debate. Such models, which vary greatly in complexity and approach, will reflect the analyst's best efforts to use all available

data to develop the estimates needed for the particular debate, but they are not generally designed with any future application in mind.

We believe that formal models bring strengths to the policy estimation process. Through their facility for repeated application within a consistent, documented framework, they can earn a return on investment in their development, serve as a vehicle for identifying important data and research gaps, and contribute to a cumulative body of knowledge relevant to social welfare policy formulation, as well as the development of improved models. However, in practice, formal models may be inadequate to the task if, for instance, they do not reflect relevant data or research findings or cannot readily be altered in a timely manner to respond to the changing course of a policy debate.

Today, for some social welfare policy topics, notably taxation, the use of formal models has largely displaced more ad hoc estimation methods. For other topics, formal models play an important role, although ad hoc models developed for specific applications by individual analysts very often remain the estimating tool of choice.

What we term the first information revolution—the institutionalization of the role of quantitative data in shaping legislation—originated with agencies in the executive branch, which led the way in developing modeling tools for analyzing alternative legislative proposals. Executive branch agencies first pioneered the use of large computerized macroeconomic models, such as those of Data Resources, Inc. (DRI) and Wharton Econometrics. These models are used to generate forecasts of economic aggregates, such as the gross national product and the rate of inflation, and also to evaluate alternative government fiscal policies. Macroeconomic models existed in the 1940s, but their use in the policy process took off in the 1960s when the advent of more powerful computing technology made it possible to develop highly complex models that more accurately reflected the operation of the economy. The forecasts produced by macroeconomic models proved useful for providing the economic assumptions used in developing ''current services" estimates of federal spending—that is, projections of spending levels over the next 2-5 years that assume no future legislative changes—against which to evaluate the President's proposed budget and legislative proposals of Congress.

Executive branch agencies also pioneered the use of microsimulation models for more detailed cost and distributional analysis. In the early 1960s, a few far-seeing analysts at the Treasury Department encouraged work by the Brookings Institution to conduct the first detailed analysis of federal tax burdens using a large microdatabase and a model to calculate taxes for each individual filing unit (e.g., a married couple or a single adult) in the sample (Atrostic and Nunns, 1988:43-47). At the same time, analysts at the Department of

Health, Education, and Welfare and the Office of Economic Opportunity were exploring the use of the microsimulation modeling techniques pioneered by Guy Orcutt for analysis of social welfare program policy alternatives. In 1968, the President's Commission on Income Maintenance developed the first operational social welfare policy model—Reforms in Income Maintenance (RIM)—which was used extensively over the next few years to model alternative welfare reform proposals (Orcutt et al., 1980:84-85).²

Beginning in the mid-1970s, not only the executive branch but also the Congress became accustomed to requesting and receiving detailed estimates of the budgetary impact and also the anticipated social impact of legislation. In particular, Congress sought information on which groups—the elderly, children, the middle class—would benefit and which would be adversely affected by a program change. The Food Stamp Reform Act of 1977 was a milestone in this regard. Over a 2-year span from 1975 to 1977, the Food and Nutrition Service in the Department of Agriculture used a microsimulation model—the Micro Analysis of Transfers to Households (MATH) model—to produce cost and distributional estimates for at least 200 variations of the proposed legislation (Beebout, 1980). As the congressional debate progressed, tables generated by the model on participants whose benefits were increased by a particular proposal as well as those who lost benefits—"gainers and losers"—became more and more elaborate, with more and more detailed categories. Congress, instead of simply accepting the administration's proposal, used the information generated by the model to shape the legislation to meet its own priorities on geographic effects, income cutoff levels, work incentives, and other factors (Shipp, 1980).

Congress set the capstone to the first information revolution when it passed the Budget Act in 1974. That act specified a formal process for setting budget targets and authorized the establishment of a Congressional Budget Office (CBO), charging it to provide Congress with analyses of the federal budgetary cost impact of every piece of legislation reported by a congressional committee. No longer would Congress rely solely on the expertise of executive agency staff; rather, it would obtain independent estimates of legislative proposals originating from the administration or Congress itself. CBO first opened its doors in 1975; today its staff numbers 225, and it prepares more than 1,200 formal cost estimates annually, as well as many informal estimates of costs (e.g.,

for proposals being considered by committee members and staff) and estimates of the social impact of proposed legislation.

Throughout the 1970s and into the 1980s, both congressional and executive branch agencies made use of a variety of models and analytical techniques for social welfare policy analysis. For example, in the early 1980s both the Social Security Actuary's cell-based model and the Dynamic Simulation of Income 2 (DYNASIM2) microsimulation model played a role in evaluating alternative proposals for altering various aspects of the social security system.

The 1980s ushered in a period of severe budget constraints that impaired the ability of policy analysts to respond to new policy issues and to maintain, let alone improve, the quality of the information provided to the policy debate. All key elements of the policy analysis infrastructure—data collection, policy research, and model development—were affected by budget reductions. The operating budgets of nine major federal statistical agencies that produce key economic statistics—such as the gross national product, consumer price index, unemployment rate, and poverty rate—declined by nearly 13 percent (in constant dollars) from 1980 to 1988 (Wallman, 1988:13). The number of professional statistical staff also declined, as did the number of staff in the Office of Management and Budget who were charged with coordinating federal statistical programs. Some policy analysis agencies experienced even more severe cutbacks. The HHS headquarters policy analysis group—the Office of the Assistant Secretary for Planning and Evaluation (ASPE)—from 1980 to 1988 experienced sizable staff losses and a staggering reduction of 86 percent (in constant dollars) in the budget for research and modeling (Citro, 1989:Table 2).

We discuss in Chapter 3 the adverse effects of budget constraints on the data inputs to policy analysis. High-quality, relevant data from surveys and administrative records are essential for all types of policy analysis, regardless of the type of analytical approach that is used or whether the analysis tool is the largest, most complex model or the simplest, most ad hoc calculation. Also, in our review of microsimulation models in Part II, we note many ways in which budget constraints limited model development and enhancement. These models have generally stayed bound to old computer technology (although their logic was rewritten to optimize the use of this technology); key relationships in the models have been only infrequently updated with results from new data; and, in some cases, more complicated but potentially important model components have been put aside. Institutional mechanisms for coordinating and fertilizing model development, such as a microsimulation users' group sponsored by ASPE in the late 1970s, languished.

Yet even as budgets remained tight, the pressure to produce estimates for legislative initiatives remained strong. Indeed, the pressure increased in the mid-1980s with the introduction on the legislative agenda of such explosive and complex issues as tax and welfare reform and the passage of the Gramm-Rudman-Hollings Act, calling for strict deficit reduction measures and

necessitating intense scrutiny of the cost and revenue effects of every piece of legislation. Recently, both Congress and the executive branch have become cognizant of the deterioration in the government's capability for policy research and analysis that supplies critical information for the policy debate and the need for investment to restore and enhance that capability (see, e.g., Boskin, 1990; Council of Economic Advisers, 1990, 1991; Darby, 1990).

In the next chapter, we consider three key areas for improving the tools that support policy analysis for social welfare programs. But first we review some aspects of the relationship of policy analysis to both social science research and politics that are important in framing an effective strategy for improving the policy analysis function. We also provide a case study of the advantages and disadvantages of microsimulation models in comparison with other analysis tools for shaping social welfare legislation that identifies some of the problems most in need of attention.

Hanushek (1990:147-148) defines policy analysis as social science research "that is directly linked to the policy process…[and that] responds to specific and detailed questions such as those that arise over a bill before Congress or a policy proposal in the Office of Management and Budget." He distinguishes policy analysis from both "disciplinary research" and "policy research.'' The former is basic research designed to advance the frontiers of knowledge, which only fortuitously may have near-term policy implications; the latter is applied research designed to produce broad guidance for policy makers. Policy research "is similar to disciplinary research in that it gives heavy weight to hypothesis formulation, to rigorous analysis, and to agreed-upon statistical standards of evidence. It differs, however, in that its objective is to produce policy implications that have some hope or expectation of being taken seriously."

Policy analysis, which is our concern, in turn differs from policy research in several important respects (Hanushek, 1990:147-148):

From the perspective of policy analysis, an important concern is how to maximize the flow of relevant and useful knowledge from basic and applied

social science research to the people who have to evaluate specific alternative legislative proposals. The current information flow is not necessarily optimal. Funding constraints for research, particularly research that is targeted to improving the policy analysis function, obviously play a role. In addition, characteristics of the basic and applied research worlds in the social sciences—such as disciplinary fragmentation and the imperatives of academic publishing, which emphasize path-breaking work over replication and consolidation—often militate against providing the kinds of information that policy analysts most need.

Turning to the other end of the link, we do not naively assume that there is a direct connection between the information generated by policy analysis and the outcome of the legislative process. The information may not be of sufficient quality or relevance to the issue at hand. Or the information may be reasonably good but lead to the conclusion that there is no clearly dominant policy choice—one option may be attractive because of low costs but unattractive for other reasons, and vice versa. More important, policy information is only one input to the decision process. Other inputs include political information and the decision makers' own beliefs and values, which act as information filters. Indeed, policy information may often be consulted simply to justify a decision that has already been made.

Other fundamental characteristics of the modern political process shape and constrain the role of policy information. First, because the process involves multiple issues and multiple actors with different perspectives, it inevitably engenders compromise and, indeed, it is designed to do so. Although such compromises may often represent the fruit of careful consideration of the available information and reconciliation of competing objectives, they may also involve blatant horse-trading or logrolling with little concern for the merits of the various provisions.

Second, because the process confronts decision makers with both multiple issues and heavy time pressures, it inevitably reduces the attention that they can devote to navigating the ocean of information that often engulfs them. This situation strengthens the role of intermediaries. One experienced observer (Hoagland, 1989) has concluded that members of Congress rely on the following sources of information in making legislative decisions, in order of decreasing importance: the member's personal staff and committee staff, constituents, fellow Senators and Representatives, newspapers, newsletters, studies and research, and congressional hearings. Of course, staff members, constituents, and colleagues may rely more heavily on information sources such as studies and research; however, the congressional member must take on trust that a staff member or other intermediary has properly interpreted study and research results.

Heavy time pressures also constrain the analysts who are involved in producing policy information. They often lack the time to do as thorough

a job as they might prefer or to consult all relevant outside experts. Once a policy debate is under way, they often cannot wait for development of improved modeling tools. If existing formal models are not suited to the task at hand, the analysts must work on a more ad hoc basis to develop the best estimates possible with other methods.

Third, the process often constrains policy estimates to be consistent with other information in a manner that may, in some cases, distort their meaning. For example, administration (and CBO) estimates of proposed legislation are constrained to the administration's economic forecasts, which may be based in part on political considerations. Or estimates may be constrained to be consistent with prior estimates for a similar proposal.

Finally, the process places a premium on certainty. Legislators, operating in an era in which legislative initiatives that add to budgeted expenditures must be offset by reductions elsewhere, want to have numbers that "add up." They do not want to be informed that a cost estimate is shaky or has a likely range of plus or minus several billion dollars. In fact, to our knowledge, information about costs and population effects of proposed legislation is hardly ever transmitted to the Congress in a form other than as a set of specific estimates—"point estimates" in the parlance of statistics. At most, policy analysts may indicate that a particular number is more conjectural than they would like.

One effect of the emphasis on certainty is that legislators frequently engage in fine-tuning policy alternatives in a way that cannot be supported by the available data. They will often request more and more model runs to determine the effects of this or that minor change on this or that specific group in order to develop a legislative package that meets a variety of objectives. They will see no need to limit their demands for information, at least until an external deadline intervenes, such as the end of a fiscal year. Yet the existing information base may be totally inadequate to distinguish reliably among policy alternatives that differ only marginally or that affect small groups.

For all these reasons, actual legislative outcomes may, to the outside observer, bear little relationship to the available information about their merits. In many cases, the legislation may be based on a greater degree of certainty in the numbers than is warranted. In other cases, the legislation may ignore the best—or any and all—information that is available.

Yet we believe that it is completely unfounded to reach the cynical conclusion that policy information is only window dressing for the political process. There is ample evidence that decision makers seek information to help them make good policy choices in the presence of uncertainty; indeed, the role of information in shaping policy has increased enormously in the recent past. Staff members close to the legislative process have testified that members of Congress want both political and policy-oriented information, about more and

more aspects of proposed legislation.³ On the political side, they want information about how a bill will affect them in the next election and how it will affect relationships between the executive and legislative branches. On the policy side, they want to know how much a bill will cost—including who will pay and who will benefit, the effect on vulnerable groups, the effect on recipients, and the impact on basic societal behaviors and values. They also want to know how one piece of legislation will interact with another, how a bill will affect U.S. international competitiveness, and how a bill will affect state and local governments and the private sector.

What we have sought to keep in mind in our work is that policy information generally bears an indirect rather than a direct relationship to legislative outcomes. Recommendations must take into account the political, social, and personal realities that channel and filter the flow of information and must seek as much as possible to build on the strengths, and work around the weaknesses, of the process whereby information is generated and used in shaping legislation.

The policy analysts at work today, both congressional or executive agency staff and employees of contract research firms, have a variety of data sources and estimating tools to call on in responding to the needs of decision makers for information about alternative legislative proposals. In this section we illustrate some of the capabilities and limitations of microsimulation models vis-à-vis other approaches for policy analysis through considering how they were and might have been used in the legislative process that culminated in the Family Support Act of 1988.⁴ The Family Support Act (FSA) includes several major provisions that alter the nation's support for poor children and their families:

a program to increase the amount of child support that is obtained from noncustodial parents; a program to provide job opportunities and basic skills training to recipients of Aid to Families with Dependent Children (AFDC); a program of supportive services for families who move off welfare, including extended child care and Medicaid eligibility; and amendments to the AFDC program itself, to expand coverage of two-parent families in which the primary earner is unemployed and to increase the incentives for AFDC families to seek jobs by increasing the amount of earnings that is disregarded in computing benefits and raising the allowable deduction for child care expenses. The latter provisions were restricted by the 1981 Omnibus Budget Reconciliation Act (OBRA) that embodied the Reagan administration's program to cut back government spending and to reduce taxes. (Another provision that was originally considered for the FSA but did not survive the legislative bargaining process was mandating a minimum AFDC benefit standard in all states.)

The analysts at CBO and ASPE faced a difficult job in developing estimates for the proposed legislation. Because of the agreement in advance that the costs of a reform package could not exceed $3.0-$3.5 billion, the analysts knew that cost estimates would be particularly important to the decision makers. They also knew that the effects of the proposed legislation on particular population groups (often referred to as distributional effects) could be important for some provisions. For example, decision makers were interested in the impact on key states of the proposal to mandate a minimum AFDC benefit standard.

It was immediately apparent that no single formal model existed with the capability to handle all of the proposed provisions. Moreover, for some provisions, no formal model existed that could do the job. The analysts developed the needed estimates using several different formal models together with their own specially tailored calculations. (It is not unusual for policy analysts to use different techniques, for example, to produce estimates of costs as distinct from distributional effects or to handle different provisions of a legislative package; the very disparate provisions of the FSA virtually dictated such an eclectic approach.)

We consider first one program feature that the analysts could evaluate with existing microsimulation models: extending program coverage to poor, unemployed two-parent families across the nation. Prior to the FSA, all states provided coverage to single-parent families who satisfied the income and assets criteria and to two-parent families who could not support themselves because of disability, but 22 states did not provide coverage to two-parent families who could not support themselves due to unemployment of the principal earner. The

original FSA proposal (and the final bill) mandated coverage of the latter group of families in all jurisdictions.

A key issue is the advantages and disadvantages of using microsimulation modeling rather than more aggregate techniques to estimate the total added costs of this proposal and the number of added beneficiaries. Considering the roughest, most global approach first, an analyst could elect to prepare overall estimates on a calculator or personal computer as follows: from current published administrative data for the AFDC program, compute the ratio of the unemployed-parent caseload (benefits) to the total caseload in the 28 states with the program; use this ratio to calculate the added caseload in the remaining states. The analyst could go a step further to provide more detailed information: from the Integrated Quality Control System (IQCS) database,⁵ run tabulations, using a statistical analysis package such as SAS, of characteristics of current unemployed-parent beneficiaries (such as race of head, size of family, and prebenefit income), and use the resulting distributions to estimate the characteristics of the added caseload.

Estimates produced in this manner are relatively easy and inexpensive to obtain, yet an analyst is likely to be dissatisfied. First, the estimates require the strong assumption that the ratio of the unemployed-parent component to the total caseload in states currently covering unemployed-parent families is a reasonable proxy for the ratio that would obtain in the remaining states. Second, the set of characteristics for which the analyst can provide detail is limited to the variables contained in the IQCS, some of which are of doubtful quality (see Chapter 5) and all of which are compromised by the same assumption that the caseload in the states currently providing unemployed-parent coverage provides all the needed information about the likely caseload in the other states. Also, the estimates do not take into account interactions with other proposed changes in AFDC or with other programs, such as food stamps and Medicaid. For example, the new AFDC beneficiaries would virtually all be eligible for food stamps, which would add costs; however, some of them would already be receiving food stamps, so their receipt of AFDC benefits would lower their food stamp allotment. Most important, this approach does not give the analyst a tool with which to respond readily to requests for estimates for alternative versions of the proposal to expand the unemployed-parent program.

A more sophisticated approach to producing the estimates would rely on cell-based modeling techniques. Depending on the complexity of the analysis desired, an analyst could implement a cell-based model using a spreadsheet program or call upon a programmer to develop the model. A simple cell-based model for our example might include a cell for each state, with aggregate state

administrative data on caseload, benefits, and characteristics for unemployed-parent recipients and others, together with such relevant information as the state unemployment rate. The analyst could pair each state not currently covering unemployed-parent families with a state that does cover such families, has comparable characteristics for the basic caseload, and faces a comparable unemployment situation. Then the model could use the appropriate ratio to calculate the expected unemployed-parent caseload for each currently uncovered state.

Estimates produced from such a model would be made more realistic by treating each state individually. However, such estimates would still require the assumption that the experience of states previously covering unemployed-parent families is directly applicable to other states. Moreover, by basing the model only on administrative data for current recipients, the estimates would ignore the distinction between program eligibility and program participation. Participation rates for many income support programs are well below 100 percent: they are currently estimated at about 77 percent of eligible families for the basic AFDC program (Ruggles and Michel, 1987:34), even lower for the AFDC unemployed-parent program, and at about 60 percent of eligible households for food stamps (Doyle, 1990:12). In addition, they vary significantly by state and family characteristics such as prebenefit income. (States show wide variations, which are not well understood, in estimated participation rates for the unemployed-parent component of AFDC.) Experience in the 1970s demonstrated the pitfalls of relying solely on administrative caseload data to develop projections of future caseloads. Projections of continuing sizable increases in AFDC, based on the explosive growth in the program in the 1960s, were off the mark, because the eligible population was growing at a much slower rate.

Determining program eligibility in a model requires a database, such as the Current Population Survey (CPS) March income supplement, that covers the entire population and includes relevant variables about family composition, income, and other characteristics. An analyst could expand the cell-based model in our example to include cells for groups of families within each state (such as unemployed-parent families in several income classes), develop eligibility estimates for each cell by tabulating the March CPS, apply participation rates based on regression analysis of the CPS data, and finally apply estimates of average benefits by cell.

However, the cell-based approach would quickly lose appeal. The analyst would want the model to produce estimates for current recipients that approximate the administrative data and, hence, would find it necessary to experiment with the specification of the cells and would most likely want to expand their number. The analyst would also find it very difficult to determine eligibility for cells, given the complex nature of the eligibility provisions for the AFDC program, which take into account in a very detailed way family composition, income sources and amounts, expenses, and asset holdings. (Also, because the

March CPS does not contain all of the needed information, the analyst would have to find a way to use other data sources.) In addition, with a cell-based model, the analyst would be hard-pressed to model alternative versions of the unemployed-parent program, to model interactions with other provisions or other income support programs, or to produce additional information about the effects on particular groups in response to decision makers' queries.

Microsimulation modeling conceptually solves many of an analyst's problems in developing detailed cost and distributional estimates for expansion of AFDC coverage of unemployed-parent families. Instead of working with aggregated or tabular data, microsimulation models of income support programs operate directly on large databases, such as the March CPS (augmented with other data sources), that contain information for many individual families and people chosen to represent the entire population. The models attempt to mimic the operation of programs such as AFDC, subjecting each sample record in the database to a set of tests to determine which family members comprise a potential assistance unit, whether the unit is in fact eligible, and the amount of benefits to which the unit is entitled. The models then simulate the decision by each unit as to whether or not it will participate in the program. The models go through this process for current program provisions and for each alternative that is specified, retaining the simulated values on each record. Hence, an analyst can readily specify tabulations that compare alternative proposals and assumptions on a wide range of dimensions, limited only by the content and sample size of the March CPS.

In addition, the models can simulate many different kinds of program changes separately or concurrently: for example, they can model the interaction between expanding the unemployed-parent program and increasing the deductions for child care expenses and a fraction of earnings. Moreover, because the major models for AFDC also include modules for other related income support programs, such as supplemental security income and food stamps, they are well equipped to trace through program interactions in a consistent framework.

In fact, both CBO and ASPE staff relied largely on the TRIM2 microsimulation model (Transfer Income Model 2; see Appendix to Part II) to produce estimates for the unemployed-parent provisions of the FSA. They also produced rough estimates to compare with the TRIM2 results: TRIM2 projected a larger increase in the unemployed-parent caseload than did the rough calculation.

However, state-of-the-art microsimulation modeling arguably added to the cost of developing policy impact estimates for this and other provisions of the Family Support Act. Added costs stem from the complexity of the models and their implementation in mainframe, batch-oriented computing technology. Microsimulation models have long had a reputation for high computer costs relative to other types of models. Today the cost for a computer run with a model like TRIM2 is quite reasonable, because the models have been optimized for sequential batch processing and use other strategies, such as operating with

a subset of CPS households below a specified income level, to reduce computer costs. Nevertheless, total costs can mount up because the use of these models inevitably requires highly skilled programming staff as well as multiple runs.

Current models, because they include many parameters, can readily simulate many proposed program changes—such as raising or lowering the value of the child care expenses deduction—simply by resetting a switch. However, other simulations would involve changing the computer code, which can quickly add time and costs to the project. Indeed, the FSA version finally adopted for the unemployed-parent program—namely, that the states not previously covering these families could limit coverage to as few as 6 months of the year—could not be modeled in TRIM2 during debate on the FSA. Both CBO and ASPE analysts adjusted the TRIM2 estimates outside the model to develop the final estimates for this provision, using monthly information provided by TRIM2. (After the FSA was enacted, ASPE invested the time and money needed for the TRIM2 programmers to revise the code appropriately to handle this provision inside the model.)

The TRIM2 model could readily handle some other changes to the basic AFDC program—specifically, increasing the child care expenses deduction and earnings allowance—and the ASPE analysts used TRIM2 to estimate the effects of these changes. However, the CBO analysts used TRIM2 to analyze these changes only for families made newly eligible for the AFDC program; for families already receiving benefits they used a "benefit-calculator" model running with an IQCS database.

Benefit calculators are a simplified type of microsimulation model. Like the TRIM2 class of income-support program models, benefit calculators for programs such as AFDC operate at the level of the individual assistance unit, mimic the program rules for determining benefit amounts, and take account of some joint program interactions, such as those between AFDC and food stamps. Because they look only at administrative data for current beneficiaries, they are smaller, less expensive, and much easier to use than the TRIM2-type models that simulate the entire population. Moreover, by definition, benefit calculators faithfully reflect the characteristics of actual beneficiaries (at least as reported to caseworkers and transcribed for the IQCS). In contrast, the CPS-based simulations for the entire population from TRIM2-type models have historically produced higher estimates of the proportion of the AFDC caseload with earnings and in other ways painted a different picture of the caseload from that drawn from the administrative records. (The reasons for these differences are not well understood. They may result from such phenomena as undercoverage of low-income population groups in household surveys like the CPS, as well as differences between the IQCS and the CPS in definitions of variables, question wording, etc.)

On the disadvantage side, however, benefit-calculator models include only current program recipients. Hence, although they may provide good estimates

for proposed program contractions, they cannot simulate the full impact of proposed program expansions, such as a liberalization of allowable deductions, that may bring new participants into the program as well as raise benefits for current participants. Moreover, in the case of program expansions, the strategy of using a benefit-calculator model for current beneficiaries and a model such as TRIM2 for newly eligible cases may introduce a measure of inconsistency into the estimates.

Finally, TRIM2 could be used to model the provision to mandate a federal minimum benefit standard for AFDC. The modeling task was very difficult because of the need to anticipate the behavior of the states with regard to raising benefits in the absence of a mandate, as well as the need to work out the expected time path of state actions to raise their benefits to meet a mandated standard. The ASPE analysts built very large and complicated spreadsheets to develop appropriate estimates of benefit levels by state and household size for TRIM2 to use in its baseline simulation of current law and its simulation of proposed law, including a mandated minimum benefit. CBO analysts used their benefit-calculator model to develop estimates of benefits for current recipients and TRIM2 to develop estimates of benefits for newly eligible units under a minimum benefit standard. The proposed provision was dropped early in the FSA debate because of its projected large costs.

Other major provisions in the FSA, which covered transitional support services, child support, and jobs and training, were arguably the heart of the legislation, but no existing microsimulation model could readily analyze them. The available income-support microsimulation models are cross-sectional in nature, designed to produce a snapshot of the caseload in a particular month or on an average monthly basis for a given year. They are not oriented to the type of longitudinal analysis required to assess the effects of providing extended child care and Medicaid benefits to AFDC families who increase their earnings sufficiently to become ineligible for basic benefits.⁶ ASPE analysts initially used TRIM2 to produce estimates of the characteristics of those eligible for transitional benefits to feed into their spreadsheet models. However, they discontinued this approach because it proved too difficult to produce estimates that could interact in a consistent way with the spreadsheet estimates for the jobs program.

Some microsimulation models in the 1970s—such as the MATH and KGB (Kasten-Greenberg-Betson) models—included modules for jobs programs.

The TRIM2 staff are currently developing a module that simulates a labor supply response to changes in income support programs. However, in the mid-1980s, no model had a usable module that could readily be adapted to prepare cost estimates for the jobs and training component of the FSA. Similarly, no microsimulation model could readily simulate the child support provisions. Moreover, even if suitable microsimulation models for the impact of these provisions on individual assistance units had been available, those models would not have been able to simulate the response of the state governments to various administrative changes—such as higher federal matching rates for state expenditures—that represented major elements of the child support and jobs programs in the FSA.⁷

The agencies could have decided to invest in model capabilities and relevant data sources in anticipation of the welfare reform debate. For example, information about both parents of children in single-parent families—not available in any data source today—could permit the development of microsimulation models to estimate the likely offsets to AFDC costs of mandatory withholding of child support payments from the wages of the noncustodial parent. However, resources were exceptionally scarce for such investment in the first half of the 1980s, and the likelihood that welfare reform would be a live topic with serious chances for enactment seemed almost nil as late as 1986. Once the debate was under way in early 1987, there was no time to invest in making substantial modifications to the current income-support microsimulation models to handle the child support, jobs program, or transitional assistance provisions of the FSA. The CBO and ASPE analysts developed their own specially tailored models to accomplish the task of producing policy impact estimates for those important components of the FSA.⁸

In briefly reviewing the FSA cost-estimation process, we noted policy areas for which microsimulation modeling could have been appropriate but for which suitable models and databases were not available. There were also several

problems that confronted the analysts in producing the estimates, whether they were using a microsimulation model or another approach.

One problem concerns methods for projecting policy estimates into a future period. Congress requires that cost estimates be made for the current fiscal year and the five succeeding years (projections for changes to such programs as social security must be made over a much longer period). Yet, at best, many of the data available for use in models are 1-2 years out of date and, of course, do not represent future conditions. Techniques exist, and have been used in policy contexts, to ''age" microsimulation model databases to future years, prior to conducting policy simulations. One commonly used approach, called "static aging," is to reweight the March CPS (or other) database on several dimensions, such as demographic characteristics and household composition, for which outside projections are available. Another approach, called "dynamic aging," is to simulate processes such as birth, marriage, death, educational attainment, and labor force participation, and generate a new database for each year of the projection. The population projections that lie at the heart of static aging and typically serve to calibrate the output from dynamic aging derive from cell-based models long maintained by agencies such as the Census Bureau and the Actuary's Office of the Social Security Administration (SSA).⁹

None of the existing income-support program models uses dynamic aging techniques. Models such as TRIM2 include a static aging capability, but its use to develop aged databases has historically been a complex, expensive, and time-consuming task, and it may distort relationships among key variables, given the inevitable limitations of the available projections. The usual practice of CBO and ASPE analysts—and the FSA estimates were no exception—has been to use TRIM2 to develop estimates of percentage differences in the costs and caseloads of current versus proposed provisions for income support programs using the latest available database (typically the March CPS for the preceding year). They then apply these percentage differences to 5-year projections of costs and caseloads for the current programs developed by using other methods.¹⁰

The preparation of current services projections for AFDC (which, for the administration, are developed in HHS by the Family Support Administration—now part of the Administration for Children and Families) is a complicated process that makes use of a variety of information, including forecasts of unemployment and inflation (which, for the administration, are developed by the Office of Management and Budget and take account of forecasts of leading macroeconomic models); information about likely actions by the states to

modify benefits; and other relevant information, such as projected growth in female-headed families. Analysts have often made use of time-series regression models in developing current services projections. ¹¹ For example, in a time-series model developed for HSS, the equation for projecting caseloads under the AFDC basic program includes variables for the number of female-headed households, current and lagged unemployment rates, seasonal movements in the caseload, weighted average standard of need across states, and a variable for the effect of the 1981 OBRA changes. The equation for the unemployed-parent caseload includes the same variables except that the size of the labor force in states with the unemployed-parent program substitutes for the number of female-headed households.¹² CBO has also developed and used similar time-series regression models for more than 10 years.

The strategy of applying TRIM2 estimates of percentage differences between alternative and current programs to independently developed 5-year projections of costs and caseloads for current programs can provide the analysts with a number of advantages. For example, the current services projections can reflect the latest information on state actions to change AFDC benefits and, generally, it is probably easier to revise them than to redo the database aging process inside of TRIM2. However, developing the projections for current as well as alternative programs inside TRIM2 could provide much more detailed distributional information within a consistent framework. Whatever method is used, there are many difficult problems in projecting future costs and caseloads that have never been fully addressed, nor have the results of alternative methods ever been subjected to rigorous comparative evaluations.

Another major problem in policy estimation concerns the possible effects of policy changes on behavior that, in turn, might have short-term or long-term feedback effects on program costs and caseloads. Such effects could be profound. For example, the transitional Medicaid and child care provisions of the FSA could well reduce the rate of reenrollment in the AFDC program. The child support enforcement provisions over the long term could well alter such basic behaviors as fertility and divorce.

Although Congress does not generally require that such behavioral effects be estimated, policy analysts will try to do so when they believe such effects may be important. However, current models and data sources provide little help for analysts in this regard. Although microsimulation models are conceptually well designed to model the effects of programs on individual behavior, income-support program models do not today (and rarely did in the past) provide a

capability for simulating behavioral responses other than the basic participation decision.¹³ The cost and complexity of implementing behavioral response capabilities have been contributing factors; however, the main impediment has been the absence of an adequate base of research knowledge providing estimates of sufficient reliability in a form suitable for modeling. The analysts who worked on the FSA were able to draw on several research studies to estimate the effects of the jobs program on earnings of AFDC recipients and the consequences for AFDC program costs. However, there were no reliable research findings to permit assessing the behavioral effects of the FSA in any comprehensive manner, whether inside or outside a microsimulation modeling framework.

A third problem area concerns assessment of the quality of the point estimates that are typically provided to decision makers. Congress does not require that analysts accompany policy estimates with information about their quality, and, indeed, as discussed earlier, the political process places a premium on certainty in the numbers. The FSA analysts informed decision makers that some of the estimates were more problematic than others. In order to gauge problems, they compared estimates produced by different agencies and used alternative estimating methods when possible (e.g., comparing the TRIM2 estimate for the expansion of the unemployed-parent program with a rough calculation based on the ratio of the unemployed-parent component to the total AFDC program in states already covering unemployed parents). However, the analysts made no systematic assessment of the variability in the estimates or of their sensitivity to the various assumptions, procedures, and data inputs used to develop them, nor did the analysts present "error bounds" to accompany their final "best" estimates. Validation admittedly is a daunting task, particularly for large, complex models such as the current income-support program microsimulation models. However, as we stress throughout the report, validation is one of the most important tasks to undertake if there is to be adequate information on which to base improvements to modeling tools and data sources that can lead to improved information for future policy debates.