The difficulties in adequately measuring the impact of open data initiatives are widely known, as it is a global debate that has been with us practically since the beginning of the first open data initiatives, more than ten years ago. The problem is that it is relatively easy to find isolated examples of how various benefits and improvements have been achieved through open data, but in general - and unless we are talking about very specific areas such as economic impact - it is not common to find complete and structured measurements of the impact that has been achieved through open data initiatives in order to demonstrate their full value.

A review of the various initiatives and methods that exist to measure the impact of open data also shows that there is no clear consensus among researchers on how best to capture the results and impact of these initiatives. 

In view of this scenario, we share here the approach to impact measurement that we have been following from the Spanish Government's open data and public information reuse initiative and that is inspired by the recommendations provided by several guides that we consider to be reference in this aspect: the UNE 178301:2015 standard, the framework of common methods for the evaluation of open data, the guide for the empirical analysis of open government data initiatives and the taxonomy of the impact of open data.

Within our personal approach to the problem we must first clarify what we understand “impact” as – “any positive effect or benefit obtained directly or indirectly for individuals, communities or society as a whole, which occurs over a certain period of time and which results from the development of different activities in a given field characterized by the use of open data as a means to an end”. 

Thus, the method we use to measure such impact is based on two main components: a quantitative analysis through indicators on data publication and its characteristics, and a qualitative one through the collection of data use cases.

Quantitative analysis

The objective of quantitative analysis is to offer a series of indicators in a quick and simple way that provide us with an overview of the data publication activity. In this way, we can track its evolution over time and easily detect trends and possible areas for improvement. This will help us to distribute available efforts and resources more effectively. 

The datos.gob.es initiative has its own public control panel that provides quantitative indicators on the following aspects:

Qualitative analysis

Through the qualitative approach, use cases are identified from different sources that can include personalized interviews, content available online, information provided by different media or academic publications. 

These use cases help us to understand the extent to which open data can be considered to have led to positive changes in three main areas described below:

Government level

Including the possible impact on the transparency and accountability of governments, as well as on improving their efficiency and effectiveness. Some of the issues that can be raised in this area are:

• How does openness of data help to improve government efficiency?
• How is data used to examine government resource use and improve existing public services?
• How is open data used to create new public services?
• How does open data contribute to increased transparency and accountability through public scrutiny?

Examples of measurements that are also useful in measuring the impact of open data actions in this area are:

•     Increased collaboration between different government departments and agencies.
•     Creation of platforms and applications that allow citizens to report on their experience in government services.
•     Greater participation of citizens in the formulation of public policies.
•     Improved public resource planning.
•     Reduction in citizens' perception of corruption.

Social area

Including the environmental aspects and the improvements obtained in the inclusion of minority groups in society. Some of the issues that can be raised in this area are:

•     What are the social benefits obtained through the use of open data?
•     How is open data used to improve equality and direct public resources to the citizens who need them most at any given time?
•     What are the benefits of open data in the area of the environment, climate change, pollution, or sustainability?

Examples of indicators that are also useful when measuring the impact of open data actions in this area are:

•     Evidence of greater equality in terms of age, gender, race, social class, disability, geographical location or poverty level.
•     Evidence of improvement in social policies.
•     Number and effectiveness of sustainability programs implemented as a result of the open data
•     Increased attention to environmental factors in project planning.
•     Awareness and sensitization of citizens about their own environmental impacts.

Economic area

Through the influence of open data in supporting existing businesses or creating new business models. Some of the issues we address in this area are:

•     What is the impact of open data on economic growth and innovation?
•     How is open data used to reduce costs in companies and help them become more efficient?
•     What new business models are being developed around open data?

Examples of indicators that are also useful when measuring the impact of data openness actions in this area are:

•     Companies and/or jobs created from the new economy of data openness.
•     New products or services using the open data.
•     Contribution of open data to the growth of the economy in terms of better macroeconomic planning.

 Future areas of improvement

While the method applied may not be perfect - as it may have limitations in terms of the ability to adequately capture desired changes -, in the absence of other reference methodologies, it allows us to explore the emerging impact of ongoing open data initiatives and provide some substantial evidence regarding their impact, at least.

Thus, it would also be desirable to explore new methods in the future that allow for evidence of impact obtained in a more systematic way through the measurement of specific goals and objectives with respect to the initial starting point, and that could also be replicated among the different initiatives, also considering the entire data value chain in the measurements carried out. Some interesting options to be able to put these objectives into practice could be the methods of outcome mapping or social return on investment (SROI) -both already proposed previously in the field of open data but with very limited or no acceptance so far, due most probably to the higher added cost that their implementation would imply.

Content prepared by Carlos Iglesias, Open data Researcher and consultan, World Wide Web Foundation.

Contents and points of view expressed in this publication are the exclusive responsibility of its author.