Assessing Progress Toward Democracy and Good Governance: Summary of a Workshop (1992)

rules that are in force; (3) the existence of mechanisms to ensure application of the rules; (4) the resolution of conflicts through an independent judiciary or through arbitration, and (5) known procedures for amending the rules when they no longer serve their purpose. These elements contribute to predictability in the sense that rules and regulations are clear and applied even-handedly, and lines of authority are clear. Adequate and reliable information are preconditions for both accountability and the rule of law: they refer to the availability of accurate information about the economy, about market conditions, and about the intentions of government that critically affects the efficiency and competitiveness of the private sector. They also refer to transparency of decision-making processes in government.

A number of examples, from the Philippines, Poland and Romania, Mauritania, and Ghana, illustrate the way these elements are manifested in particular country settings (World Bank, 1991). Having defined the elements, it would seem that indicators could be developed for assessing change along those dimensions; the indicators can be used for both research and policy purposes. However, the World Bank has not, to date, developed indicators, nor have they initiated a discussion of methodological issues concerning assessment at either the micro or macro levels of performance. The concepts are used to provide criteria for decision making, but they are also contributions to theory.

The discussion of measurement issues was initiated by Michael Coppedge’s presentation. He outlined a five-step procedure for measuring such complex, multidimensional concepts as democracy or governance:

• Unpack the concept by dividing it into simple, measurable components;

• Measure each component as though it was a separate dimension;

• Assess relationships among different measures to ascertain whether they are indicating the same or different dimensions;

• Disregard dimensions that now seem largely irrelevant, and

• Combine the components that indicate the same dimension into a single index.

His research with Wolfgang Reinicke (Coppedge and Reinicke, 1990) uncovered two dimensions that seem to capture the Dahl’s conception of democracy as polyarchy: broad participation in elections and an index consisting of elected rulers, freedom of expression, and freedom to organize; and a pluralistic media. From these indicators, they derived a Guttman scale ranging from full polyarchy to nonpolyarchic countries, emphasizing the dimension of public contestation.¹

Coppedge recommended that a similar procedure be used to measure governance. Drawing on a wide variety of source information, it should be possible to rate countries on

five to ten components of governability. That effort would reveal several dimensions, such as effective leadership, technical policy competence, administrative efficiency, and institutionalization of the political system.

Another approach to measurement is illustrated by the work of Ian Budge who is interested primarily in the way that policies are transformed into actions. He has developed refined measures of democratic functioning focusing primarily on developed democracies (see, e.g., Budge and Hofferbert, 1990). In one project, he codes political documents in terms of saliency or the frequency with which alternative policies are mentioned. The codes are then used to test competing theories of democratic functioning—for example, a confrontational versus a saliency versus a party mandate theory-as well as theories of coalition formation that emphasize the importance of policy closeness in decisions to align. Budge suggested that the same approach could be used to measure governance, addressing such questions as how parties address problems, how the electorate defines problems, and how government expenditures work to create solutions.

Axel Hadenius presented his ongoing research (Hadenius, 1992). He measured the level of democracy in 132 countries representing about 80 percent of the number of independent states in the world. His cross-sectional, comparative approach is designed to address the question; What accounts for differences in levels of democracy achieved by 1988? The two equally weighted components of his democracy index are elections and political freedoms. More democratic countries are characterized by universal suffrage and regularly held open and honest elections. Political freedoms include organizational freedoms, freedom of opinion, and a lack of political violence or oppression perpetrated by the government’s police and military forces. Each of these elements was measured on a scale, usually ranging from 0 (no freedoms) to 8 (all freedoms) with intermediate points such as “considerable freedoms, great restrictions” (4). A wide range of sources provided information for coding, including compilations of data on most countries of the world (e.g., Keesing’s record of world events), journal articles and newspapers (e.g., Le Monde, Financial Times), and specialized regional publications, such as the Asian Yearbook and the Far Eastern Economic Review.

A subset of variables associated with the coded components of democracy was then identified by step wise multiple regression analysis. Sixty percent of the variation in levels of democracy was explained by the combined impact of seven factors (including a capitalist economy and the level of literacy). These results support other research that argues against single-factor explanations for democracy. Hadenius cautioned, however, that the correlational findings obtained in this study do not suggest a direction of causation. Beyond this, his research to date does not provide insights into the way countries may change over time, toward or away from democratic institutions. A next step for this project is to perform a comparative longitudinal analysis of democratic performance on a few of the countries in his sample.

A number of issues were raised in the ensuing discussion. Dahl noted the distinction between empirically grounded theory and policy analysis. Coded variables are not to be construed as detailed knowledge about a country. Rather, they contribute to a perspective that calls attention to some factors that may move a country up or down the “democracy ladder.” It is a labor-intensive enterprise that can be facilitated by a carefully designed codebook and precise conceptual definitions. Coppedge and Hadenius noted

considerable progress in their projects toward achieving their goals: Coppedge is developing a codebook in which assumptions underlying measurement will be made explicit; Hadenius’ study can be reproduced or updated in a fraction of the time it took to complete the initial work.

Three conceptual distinctions were the subject of further discussion. First, it seemed useful to distinguish between stable and unstable democracies, asking what are the conditions that contribute to the one or the other. Second, a distinction should be made between progress toward democracy and relative “amounts” of democracy; the former is a longitudinal issue defined in country-specific terms; the latter is a cross-sectional or comparative issue (illustrated by Hadenius’ research). Third, it is necessary to decide whether democracy and good governance are fundamentally similar or different concepts. Each of these issues are empirical questions that should not be predefined by a particular framework. Each can be elucidated through systematic research on relationships among various indicators of the concepts.

A discussion of data quality issues was initiated by Michael Stohl. An overview of the basics of social science measurement proceeded in five steps: delineating the concept, specifying the relevant dimensions into disaggregated components, defining and calibrating the dimensions in operational terms, measuring the dimensions with available sources of information, and, if appropriate, reaggregating the dimensions into a composite index. His criteria of scope (range of coverage), precision (accuracy within and between indicators), parsimony (fewest and simplest set of components), and generalizability (reproducing the indicators on many cases) can be used to judge the adequacy of any data set. The goal is to develop the smallest set of indicators that will result in the most accurate and reproducible measurement of a concept (see Lopez and Stohl, 1992).

Reliability refers to consistent results in successive measurements of the same case and comparability across cases. Threats to reliability include changes in the reporting periods, changes in reporting criteria, changes in the proficiency of the data collector, errors of omission (overlooking events you do not want to know about for a variety of possible reasons), and errors of commission (misnaming an event or including it in the wrong category). Validity refers to the adequacy with which a measure represents the concept. Two types of validity are face validity and construct validity: face validity is judged on the basis of appearing to satisfy theoretical and substantive arguments about the definition of a concept; construct validity is satisfied by the accumulation of evidence showing that an indicator is related to and consistent with other measures also deemed to indicate the concept.

These methodological concerns apply to high-quality and relatively unbiased data. Using a series of political cartoons, Stohl illustrated several reasons that data in such politically sensitive areas as human rights might be distorted (Carleton and Stohl, 1985; Stohl et al., 1984). One reason is that collectors of information may have few incentives for discovering new knowledge. A second is that data can be unintentionally misrepresented due to a lack of understanding of subtleties in a situation. A third is that subjects or

respondents may alter their behavior when they are aware of being monitored. Another reason is that one’s own political ideology can unwittingly influence measures of violations of human rights. The fifth reason is the intentional misrepresentation (or nonreporting) of information to serve special interests. Being aware of these potential problems is a safeguard against uncritical acceptance of political data but it does not ensure attaining quality data. Nor does it prevent mistakes made by an investigator during the process of generating his or her own data for analysis.

Daniel Druckman discussed the connection between measurement scales and statistical analysis. More powerful analyses, in the sense of avoiding Type I errors (to reject an hypothesis when it is true) or Type II errors (to accept an hypothesis when it is false), are possible with more refined measurement scales, for example, interval rather than ordinal or nominal scales. With regard to measurement precision, however, the questions are how precise does one want to be and how precise the measurement can be. The former question is driven by theory, the latter by technique. It is apparent that, unlike some other areas of political analysis, the theory of democracy has not advanced to a point at which precise measurement and, therefore, powerful statistical analysis, is feasible.

Raymond Gastil note that it is important to distinguish between hypothesis testing or correlational analysis and the needs of A.I.D. Developing a scale that is valid and reliable for the needs of academic social science is a very different enterprise from improving a scale so that it measures the range of changes in a polity in which A.I.D. practitioners actually appear to be interested. Druckman countered that it may well be that statistics used to test hypotheses, such as the analysis of variance, may be inappropriate. However, he noted that statistics play an important role in sampling designs since inferences about countries are made on the basis of analyses of data collected from samples of respondents or events. Useful illustrations of applications of statistical analysis are presented in the sourcebook by Daniel Frei and Dieter Ruloff (1989).

Hadenius’ research highlights some trade-offs between quantitative and qualitative analysis. The testing of well-formulated hypotheses by statistical methods has the advantage of allowing an analyst to control for spurious causation in the computed relationships between variables. Qualitative analysis of a few cases allows for a penetrating and broad picture of the particular situation under study. One or the other approach may be better suited to particular research purposes, such as the difference between gaining theoretical knowledge or giving practical advice. Together, the approaches can contribute to the goal of comparative analysis by illuminating broad patterns and trends as well as the way that these trends are manifest in particular regions and countries.

The discussion of these issues concluded with observations made by George Lopez and Ian Budge. Lopez noted that conceptual work lags behind methodological advances: some concepts lend themselves to measurement more readily than others. Budge commented that there may well be more agreement among researchers on indicators of some concepts than on the viability of the concepts themselves.

The issue of human rights measurement was introduced in the previous workshop (see National Research Council, 1991a). Although human rights are not synonymous with democracy, they are generally regarded as conditions for enhancing democracy as well as a more basic consideration in deciding on performance-related aid. George Lopez proposed a way of classifying human rights abuses associated with regimes. Building from the work of Jack Donnelly and Rhoda Howard (1988) Lopez outlined four clusters of rights: protection rights (e.g., freedom from arbitrary arrests), membership rights (e.g., guarantees of citizenship, religion, minority cultures), empowerment rights (e.g., freedom to associate, a free press), and survival rights (right to food, housing, health care). He noted that there is no ordering of these rights in terms of their inherent importance. As a minimal condition, we expect democracies to protect these rights; one of the desired behaviors in a democracy is enhancing these rights.

Lopez proposed a measurement procedure that would diagnose gradations within each of the rights clusters. Regimes can be monitored in terms of severity of the rights violations (varying from slight to brutal) and the frequency of violations at discrete levels of severity. In addition, Lopez proposed examining the range of rights violations (which groups are under abuse) and who are the offenders. Identifying the latter provides information on the level of commission or omission in the rights strategy of a government. Thus, even if government agents are not offenders, the less willing or able a government is to deal with rights offenders, the less likely it will be able to sustain democracy over time (see Lopez, 1992).

Lopez illustrated the utility of these indicators by examining four different cases of rights abuses: Chile in three distinct time periods under the Pinochet regime (1973–1979, 1980–1984, and 1985–1987); El Salvador and Nicaragua since 1979); and Chicago, Illinois, where various cases of torture and brutal treatment in city jails has led to investigation of cases by Amnesty International in London. During the discussion, Lopez noted that among the advantages of using such a multiple system of rights violations, two are most important. First, it permits external monitors to see variation from one emerging democracy to another in the types of rights that may be in jeopardy. Second, the scheme encourages the investigator to use a large variety of sources of information for coding the violations.

A graded scale for measuring rights violations was developed by Raymond Gastil (see Gastil, 1980). Michael Stohl observed that this scale has practical value in determining limits on aid for particular countries. It requires only ordinal judgments—distances between levels are not equal but a country at level 1 is doing better than a country judged to be at level 2—and is easy to use. Like all scales, it is only as good as the quality of the information gathered for making the judgments. Although it does not deal with issues of how to intervene to deal with a country’s problems, it does provide cutoff points that can be used to determine aid levels. The levels of violations are as follows:

• Level 1: Citizens live under a secure rule of law, people are not imprisoned for their views, and torture is rare or exceptional.

• Level 2: There is a limited amount of imprisonment for nonviolent political activity, but few people are affected; torture and beating are exceptional.