Towards the Monitoring of Goal 16 of the United Nations' Sustainable Development Goals (SDGs): A Study of the Selection, Rationale and Validity of Indicators with Suggestions for Further Improvements

Abstract

Over the past half-decade, the United Nations has engaged in an unprecedented global, participatory and inclusive consultation process that lead to the definition and adoption of a new set of Sustainable Development Goals (SDGs) as a successor to the earlier Millennium Development Goals (MDGs). After literally hundreds of global, regional and national consultation meetings and countless written submissions and online consultations with inputs from national institutions, civil society organizations, academia, regional and international organizations and the departments and agencies of the UN system debating the merits of a multitude of competing objectives, representatives from UN Member States in a series of intergovernmental negotiations agreed on a common and universal set of goals and targets for the period 2015-2030. This study looks into the process and substance of the debate, based on a review of written sources, interviews with a number of experts involved in the development of the goals, targets and indicators, and inputs from written questionnaires, with a view to learn more about the strengths and weaknesses of the process and its outcomes.

The focus of this study is on the development of the global indicator framework for SDG 16, and in particular on indicators relating to rule of law, access to justice and corruption. While there are a number of critical points to be raised about the content and formulation of SDG 16 and its targets, many of which are reviewed in this study, there is a broad consensus among many commentators, as well as the experts interviewed, that the inclusion of a goal on peace, justice and institutions, with a broad number of targets, is a historic achievement by itself and that, generally, the targets within SDG 16 are comprehensive, meaningful and well balanced.

This study also shows that both the strengths and many of the shortcomings of the selection and formulation of the targets are linked to the nature of the SDG adoption process as such. For instance, the sheer number of stakeholders involved in the process, while insuring comprehensive coverage of worthy objectives for a global agenda, also contributed to the inflation of the number of goals and targets, whereas the original intention had been to keep them fairly limited. And because the number of targets had to be limited at some point in the process, the push and shove to have additional concerns reflected in the targets that were already agreed to has led to some convoluted formulations that packed multiple objectives and different concepts into one single target. A prime example of enrolling multidimensional concepts into one single target is 16.3, which now covers a range of worthwhile objectives related to the rule of law and access to justice but was originally conceived of as a goal in itself with several associated targets.

Once it became clearer that the SDGs will have a goal on peace, justice and security and the outlines of the related targets became visible, work on the indicators for SDG 16 intensified. A large part of this study details the proposal, discussion and selection of alternative indicators for the targets under SDG 16 and discusses the strengths and weaknesses of the resulting indicator framework that was provisionally adopted by the UN Statistical Commission in March 2016. Again, there is a broad consensus among the experts interviewed for this study that the indicators for SDG 16 generally do a reasonably good job of covering the targets they are supposed to measure. While there are good reasons to agree with this assessment, three areas of concern should be highlighted.

First, the adoption of some broad, multi-dimensional targets under SDG 16 creates a dilemma for the monitoring of the target when the number of indicators is too limited. For instance, there is no possible combination of only one or two indicators that can cover all aspects of target 16.3 on rule of law and access to justice. Second, issues of data availability and the concerns of many (often smaller or developing) countries about their capacity to measure complex indicators through large-scale and expensive population surveys have already led to the restriction of survey-based sources in favour of administrative sources. However, many issues related to peace, justice and institutions can be appropriately captured only through survey-based measurement. During implementation of the indicator framework, issues of capacity-building and data availability in developing countries should therefore be given high priority in order to avoid large data gaps. Third, in order to monitor progress towards the high aspirations of the Agenda 2030, and in particular on the central theme that "no one will be left behind", special attention should be paid to the capacity to collect data that are disaggregated by various relevant dimensions such as sex, age, income, ethnicity or other relevant disadvantages, both in survey-based data and administrative data sources.

Despite these and other concerns raised in the study, the quantitative analysis of available data on selected targets under SDG 16 provides some evidence that the chosen indicators for targets 16.3 (rule of law and access to justice) and 16.5 (corruption) do, in fact, contribute to the measurement of the underlying concepts of the targets. A further refinement of the indicators, along the lines suggested in this study, as well as an improvement of the data sources for the indicators will further enhance that ability to measure progress towards the targets at the national, regional and global levels.

In addition to providing an in-depth analysis of the long and complex process of developing an appropriate set of goals, targets and indicators for SDG 16, this study also provides two sets of proposals for the short- and medium-term future: 1. A set of proposals for the refinement of indicators on targets 16.3 and 16.5 that can be adopted already in the near future by the Inter-Agency and Expert Group on SDG indicators. These proposals include the splitting of indicator 16.3.1 into two survey-based indicators relating to criminal justice and civil justice, respectively; further disaggregation of indicator 16.3.2 by length of unsentenced detention; and refinements of the formulation of the indicators on the prevalence of bribery by the population (16.5.1) and businesses (16.5.2). 2. A set of proposals for the elaboration of additional indicators used for monitoring SDG targets at the regional level. Regional indicators are currently under discussion by various intergovernmental bodies and regional processes in Africa, the Americas and Europe. This study has compiled a list of potential indicators for regional monitoring of targets 16.3 and 16.5 and has further collected inputs and regional priorities from the research institutes of the United Nations Crime Prevention and Criminal Justice Programme Network that are reproduced here. It is hoped that the study can be of wide use for practitioners and scholars interested in the development of appropriate metrics for the monitoring of the SDGs.

In parallel with the negotiations on the goals and targets, there was a broad-based consultative process on the development of the indicators to monitor the targets. This is no trivial task for any of the goals and targets, but the development of an appropriate indicator framework was particularly important for SDG 16 in order to demonstrate early on in the debate on the post-2015 development agenda that issues of peace, justice and institutions are an integral part of the development agenda and that they are, in fact, measurable. A broad-based effort by UN agencies, selected Member States, civil society organizations, academia and other stakeholders drove the point home. Similar to the development process of the SDGs themselves, a broad participatory approach with multiple stakeholders was adopted for the elaboration of the indicator framework, resulting in a process that often complicated the difficult technical task of selecting suitable indicators by confounding them with non-technical considerations of a political nature.