Skip Navigation

Ambitions and Realities in OGP Commitments: Analysis of Commitment Completion Across Countries using Hierarchical Models

Daniel Berliner|

This is the fourth in a six blog series written by winners of the IDRC grant for research on OGP.

The Open Government Partnership (OGP) has 69 participating countries around the world, with over 2,000 commitments made and countless citizens and civil society members involved. Learning from past experiences is essential to moving the OGP forward and helping to better craft and implement commitments. The first round of Independent Reporting Mechanism (IRM) reports has shown that, while some countries completed most of their commitments, others did not. Many commitments have seen only limited progress, or even none at all. In this short research paper, I sought to answer the question: Why are some commitments more likely to be completed than others?

To answer this question, I used data from the OGP Explorer based on IRM evaluations of each country’s commitments, on the 33 second-cohort countries covered with adequate data by the first round of IRM reports. IRM researches coded commitments both on their characteristics, such as their potential impact and their level of specificity, and on the extent to which they were actually completed (I considered commitments that were coded as either “complete” or with “substantial” progress). However, these commitments also need to be considered in context of the countries that made them. Thus I also considered country-level information such as democratic institutions, economic development, and factors associated with the OGP process itself. I used hierarchical models for multilevel data in order to simultaneously combine commitment-level and country-level information.

What did I find? First of all, my results showed that comparing implementation across countries is a complicated task. It is not sufficient to simply compare the average rate of commitment completion from country to country, since not every commitment is identical. Consider two hypothetical countries that both completed half of their commitments, and so would appear to have made identical progress. However, suppose that the first country’s commitments were primarily “easy” to complete – low potential impact, small in scope, and already underway – while the second country’s commitments were “hard” – high potential impact, large in scope, and brand new policy goals. Should these two countries’ equal completion rates be judged as truly identical?

As this example makes clear, the composition of National Action Plans should matter for how we assess countries’ progress towards implementing them. The method used in my study allows for comparison of national completion rates after adjusting for the composition of each country’s commitments. The figure below compares these new adjusted rankings to unadjusted rankings based solely on the average completion rate for each country.

The lines connecting each country’s position in the two columns are colored to show their change in relative position. Green lines indicate countries for which the adjusted measure reflects better performance than the original measure. For these countries, basic average completion rates do not reflect the extent to which their commitments were harder-to-complete on average than those of other countries. Red lines indicate countries for which the adjusted measure reflects worse performance than the original measure. For these countries, basic average completion rates give an overly positive impression, and do not reflect the extent to which their commitments were easier-to-complete on average. Interestingly, the top three positions in terms of country performance remain identical under both measures: Uruguay, Chile, and Latvia. (Please note that this figure only includes the 33 second-cohort countries for which sufficient OGP Explorer data was available).

My study also arrived at findings as to which types of commitments were more or less likely to be completed, and in which types of countries. The results showed that commitments coded as having higher “potential impact” by IRM researchers were less likely to be completed than those with lower potential impact. This “impact gap” suggests that countries were less likely to complete their more ambitious commitments, and more likely to complete their less ambitious ones. Other findings showed that “new” commitments were less likely to be completed than commitments reflecting already-existing policy goals, while more “specific” commitments were more likely to be completed than less specific ones.

Turning to country-level findings, the results highlighted two factors external to the OGP process: democratic institutions (measured using data from Freedom House) and the level of corruption (measured with Transparency International’s Corruption Perceptions Index). More democratic countries and less corrupt countries were more likely to complete their commitments. This suggests that countries where government is already more open to begin with tended to make greater implementation progress. On the other hand, there were no statistically significant results for any factors internal to the OGP process, such as public consultation or civil society involvement.

Finally, I returned to the “impact gap” discussed above, and sought to identify whether it was a feature of all countries, or only certain types of countries. In fact, the “impact gap” – the difference in completion rates between high potential impact commitments and low potential impact commitments – varied in extent from country to country. The results indicated that this variation could be explained by the extent of the civil society network involved in the OGP process in each country: In countries with small or absent civil society networks, the “impact gap” was largest. However, in countries with large civil society networks, the “impact gap” actually disappeared, and the most ambitious commitments were just as likely to be completed as less ambitious ones. This suggests a nuanced yet important role for civil society in the OGP process. While civil society may not have boosted national completion rates directly, these results indicated that they helped to prevent high potential impact commitments from being “left behind.”

More details can be found in the study itself, available here. 

Filed Under: Anti Corruption and Integrity Anti-Corruption Institutions Research
Open Government Partnership