Appendix A
Validation Exercise
This appendix discusses and compares two of the most comprehensive indices of the rule of law (ROL): (a) the World Justice Project (WJP)1 and (b) the Varieties of Democracies (V-DEM) Project.2 Both projects provide a composite ROL Index through a multidimensional set of outcome indicators, each of which reflects a particular aspect of this complex concept. The two measures are highly correlated.
The evidence reviewed by the committee suggests that research can, in principle, measure whether police reforms can support basic tenets of and potentially improve the ROL. The committee identified candidate measures that satisfy requirements of basic face validity and are regularly produced for multiple countries over multiple years.
Many of the indicators in the V-DEM survey reflect specific components of indicators in the WJP indices. However, there are no explicit questions in the V-DEM survey distinguishing civil from criminal justice, or order and security. In contrast, the WJP survey does not explicitly ask about gendered rights or about white-collar crime by government and private officials, nor does it distinguish between higher and lower courts.
In order to validate the measures, we run two simple analyses. First, we correlate V-DEM and WJP indices; the idea is that a high correlation would offer face validity. Second, we explore whether the ROL Index captures which countries are more likely to violate human rights.
___________________
1 See https://worldjusticeproject.org/our-work/research-and-data/wjp-rule-law-index-2020.
Using the last year for which the two leading sources (V-DEM and WJP) have data, Figure A-1 below shows a rather high correlation (0.81) between the two composite indices, which is reassuring.
In further analyses, we compare V-DEM’s index, since it has a broader country coverage than the WJP index, to a measure of human rights violations. We use the Latent Human Rights Protection Scores constructed by Fariss, Kenwick, and Reuning (2020).3 As Figure A-2 makes clear, there is a strong positive and significant relationship between V-DEM’s Rule of Law Index and a country’s human rights score.
___________________
3 The Latent Human Rights Protection Scores can be found at https://dataverse.harvard.edu/dataverse/HumanRightsScores.
Correlates of the Rule of Law
In this section, the correlates of the ROL are explored with no causality claims made: the goal is more modest, and limited to correlational analysis.
While in theory the ROL and a country’s level of democracy are two distinct concepts, in practice both are highly correlated, as Figure A-3 makes clear. Here, V-DEM’s index of Electoral Democracy is used; it has a 0.82 correlation with the ROL Index.
The strong correlation between regime type and the rule of law is not simply a function of how V-DEM constructs both measures. Replacing V-DEM’s continuous measure of democracy with Freedom House’s
three-category measure of democracy (Free; Partially free; Not free),4 a strong correlation is again found between regime type and the rule of law; see Figure A-4.
Interestingly, the strong positive correlation between a country’s level of democracy and its commitment to the rule of law is observed at all income levels. Using the World Bank’s income designation,5 we disaggregate Figure A-3 to three categories: high-income, middle-income, and low-income countries. As Figure A-5 shows, a positive correlation between a country’s
___________________
4 See https://freedomhouse.org/report/freedom-world.
5 See https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-countryand-lending-groups.
level of democracy and the rule of law can be observed at all income levels (though it is admittedly weakest for low-income countries).
Figure A-5 strongly suggests that a country’s wealth is correlated with the rule of law independent of a country’s level of democracy. We explore this correlation in Figure A-6, which examines wealth per capita. As the figure illustrates, with few exceptions, on average, richer countries exhibit stronger commitment to the rule of law.
This page intentionally left blank.