-
Level 3: There is extensive political imprisonment or a recent history of such imprisonment. Executions or other political murders and brutality may be common. Unlimited detention, with or without trials, is accepted. (This is a threshold level for defining gross human rights violations.)
-
Level 4: The practices of level 3 are expanded to larger numbers. Murders, disappearances, and torture are a common part of life. In spite of its generality, terror at this levels affects primarily those involved in politics or ideas.
-
Level 5: The terrors of level 4 are extended to the whole population. The leaders of these countries place no limits on the means with which they pursue personal or ideological gains.
Although this scale may miss certain nuances in regime practices, it has the advantage of wide applicability and comparison among countries and, within countries, for different time periods. It calls for judgments that are relatively easy to make, using available information for dealing quickly with immediate problems.
FROM THEORY TO PRACTICE
On the second day of the workshop the discussion focused on the interface between academic researchers and practitioners. An emphasis was placed on the practicality of indicators for making allocation decisions or evaluating aid programs. This issue primarily confronts the mission-oriented agencies. Several agencies are involved in projects that develop or make use of indicators of democracy. Three of the larger projects are those being carried out by the Africa Bureau of A.I.D., the United Nations Development Programme (UNDP), and Freedom House. A representative from each of these agencies summarized the progress made to date followed by a general discussion of the issues.
Jennifer Windsor described the Africa Bureau project. The bureau’s purpose is to devise a system that would serve as a decision aid in the allocation process. It would also be used to facilitate a dialogue among those African countries whose performance in certain areas is a source of concern. Five criteria for measuring progress toward democracy and governance have been developed: freedom of information; freedom of association; human rights with regard to the integrity of the person; public accountability, which includes an independent judiciary; and public-sector efficiency, which includes budget transparency and levels of corruption. Using several sources of information—for example, State Department country reports, Amnesty International, Freedom House, Africa Watch—experts evaluate each country independently on the five criteria. Discussions among the evaluators are intended to illuminate reasons for their decisions with the goal of obtaining a consensus judgment. Although this is a subjective process, the bureau attempts to make it as transparent as possible, explaining clearly how the judgments are made. Yet it is also the case that whenever a system is to be used for making practical decisions, several sources of distortion are inevitable. Politics inevitably plays a role in aid allocations, no matter what measurement system is used. In addition, the measurement process must cope with problems of instability and armed conflict in some African countries and the difficulties involved in attempting to capture changes that often occur quite rapidly.
Bernard Hausner described the work being done at UNDP. Based on its earlier efforts to construct a “human freedom index” (United Nations Development Programme, 1991), the agency is attempting to develop a wider perspective and revised methodology for measuring freedom. For that index, the agency examined 13 human rights instruments, covering both international and regional covenants, and identified 124 different types of rights, including social and economic as well as political and civil rights. It distilled the political and civil rights into five broad clusters: personal security, rule of law, freedom of expression, political participation, and equality of opportunity. The checklist for personal security, for example, consisted of indicators pertaining to arbitrary arrest and detention, torture or cruel treatment, extrajudicial killing and disappearances. Although U.N. documents served as the main sources of data, a variety of other types of documents were also included among the 116 sources, for example, reports from nongovernmental organizations, such as Amnesty International, Human Rights Watch. Each of the 104 countries for which data was obtained was rated on each cluster by three sets of experts in the human rights field, culminating in a “grade,” on which there was considerable agreement among the experts.
Hausner commented on the methodology, noting, in particular, problems of reporting, aggregation of separate measurements into a combined scale, and a lack of agreement on a weighing system for indicators within a cluster as well as for the clusters. With regard to aggregation, for example, it is not clear whether to average the values of the five clusters, to take the product of these values, to use the minimum of the five values, or to leave the cluster grades in disaggregated form.
During the discussion, Raymond Gastil noted that changes may be captured best by computing 5-year running averages. He also observed that the meaning of high agreement among judges needs to be clarified and that measurement should capture changes rather than states. It might be more important to develop indicators for relative opportunities for assistance to democracy than for either changes or states. Along these lines, a representative from A.I.D. commented that the agency is moving away from concepts that depict “levels” or “states,” toward concepts that capture trends over time.
George Zarycky reported on the work being done by Freedom House. The purpose of their comparative survey of freedom, since its inception in the 1970s, is to provide an annual evaluation of political rights and civil liberties around the world (McColm, 1992). In an overview of the survey methodology, Joseph Ryan (1992:13) states: “The Survey attempts to judge all places by a single standard and to point out the importance of democracy and freedom.” Political rights are defined as enabling people to participate freely in the political process; civil liberties are the freedom to develop views, institutions, and personal autonomy apart from the state. Zarycky summarized the items that are used to assess political rights (for example, fair electoral laws, fair polling, fair campaigning opportunities) and civil liberties (for example, free and independent media, cultural expression, and literature). The items are arranged in checklists that guide the country ratings. Each country is awarded from 0 to 2 points per checklist item, depending on the degree of compliance with the standard. The highest possible score for political rights is 18 points, based on up to 2 points for each of nine questions. The highest possible score for civil liberties is 26 points, based on up to 2 points for each of 13 questions. By asking the same questions of all countries, it is the intention of the survey to enable users to make
comparative judgments about the situation that exists at a given time in several countries. The ratings are situation assessments, not forecasts or prescriptions for action. However, Ryan (1992:15) notes that “we wish to use the survey as a prod to improve the condition of all countries.”
Although there are many similarities among the three projects, they also differ in important ways. The criteria developed by the Africa Bureau include both indicators of democracy (e.g., freedom of information) and governance (e.g., public accountability). This is the only project that is designed for purposes of practical decision making; therefore, it is sensitive to the political problems that could lead to distorted estimates. The UNDP project is an ambitious attempt to cover many varieties of political freedoms. It has recognized the complex methodological issues involved in making and aggregating judgments, and as a consequence they have been cautious in asserting that the system is ready for application. The Freedom House project has the longest history. Its indicators of political rights and civil liberties have been widely used—and is being used by A.I.D.’s Latin American Bureau— and they have invested considerable resources in refining the methodology. Like the UNDP project, it does not use the system for policy purposes. All of the investigators agreed that the indicators should be used in conjunction with country-specific narratives to get a more complete picture of a country’s progress toward democracy.
The discussion was energized by a spirited exchange between the researchers and the practitioners, which highlighted the issue of what general coding systems can and cannot do to help aid donors in making resource allocation decisions. The researchers cautioned that their projects were designed to assess change and to analyze relationships among variables that are of interest in trying to build theories about democracy. The indicator systems developed for such research can provide the means to measure overall country performance, to compare one country’s performance against another’s, and to identify at least broad changes in country performance over time. But the kinds of indicators discussed in the workshop in both the May 1991 workshop and here are generally not well suited to making aid allocation decisions among projects for a given country nor to evaluate the impact or effectiveness of projects.
Michael Coppedge discussed some of the reasons that one should not expect indicators of democracy to be suited to evaluating programs. Current indicators can detect only rather large changes in democracy, which are not likely to be produced by small foreign aid programs, and say little about whether the aid program per se influenced the change. More broadly, although a good deal has been learned about the processes of transition away from authoritarian rule, political science has not produced an accepted consensus about how democracies develop nor about their relationship to economic development (see National Research Council, 1991b; 1992). Moreover, only some of the changes that democratization programs could foster, such as increased voter turnout or reductions in the level of human rights abuses, can be readily quantified. This is frustrating for aid agencies accustomed to working in such areas as health and agriculture, for which changes are far easier to track and measure.
In the absence of a theory about how democracies can be consolidated, seasoned practitioners and country specialists with in-depth knowledge may be the best source of advice on project allocations and for reliable qualitative assessments of impact and effectiveness. Reinforcing this point, Raymond Gastil commented that a full understanding
