Measuring the impact of open data is not always easy. As we saw a few weeks ago, there are several theoretical models that are not easy to implement, so we have to look for different approaches. In the Aporta Initiative we use a mixed approach, as explained here: a quantitative analysis through indicators on data publication and its characteristics, and a qualitative one through the collection of cases of data use.

This approach is also used by various local, regional and state initiatives in our country. In today's article, we will focus on concrete examples of mechanisms implemented by Spanish open data initiatives to monitor and measure the impact of the use of their data.

Quantitative analysis

One of the first steps in monitoring impact is to know quantitatively if users are accessing the published data. To do this we can use different tools.

Dashboards

Thanks to the incorporation of web analytics tools in open data platforms, such as Google Analytics or Motomo (which until 2018 was called PIWIK), a series of indicators can be set around data consumption variables, such as how many users visit the web, what is their origin, which data sets are most in demand or in what format they are downloaded. All of this information is of great value when it comes to making decisions that imply improvements to continue promoting the reuse of public information.

With this data, dashboards can be created so that users can also know this information. This is the case of the Castellón Provincial Council, the Madrid City Council, the Catalan Government, Renfe, the Basque Government or the datos.gob.es itself.

Conducting surveys and periodic studies

In addition, it is advisable to carry out frequent public consultations and studies that allow us to know directly the impact of our data. The ONTSI periodically carries out a characterization study of the sector, and an analysis of the Public Administrations as reusers of their own data and that of third parties. Another example is the report on the Infomediary Sector of ASEDIE, now in its 8th edition. This report measures the products and services based on open data that have been generated. Both reports use a stable methodology that allows comparisons between different years. 

Qualitative analysis

It consists of the identification of use cases through different mechanisms, such as

Application and enterprise tracking

Thanks to the mapping of open data use cases, we can know what the impact of a certain data set is. In many open data platforms, whether local, regional or state, we can find a section of applications or companies with examples and reuse success stories that in turn serve as reference and inspiration for the creation of new value services. In the case of datos.gob.es, we have a form for companies or applications that wish to register their information, but we also carry out a proactive search, through contact with the main actors in the ecosystem and media alerts.

Other examples of portals that have applications sections are Andalusia, Castilla y Leon, Navarra, Barcelona, Santander, Malaga, Zaragoza, Valencia, Vitoria or Bilbao, although there are many more.

Implementation of data communities

In order to be aware of new developments in the field of reuse and to exchange knowledge and experiences to align the data publication strategy with the needs of reusers, some initiatives have opted for the implementation of communities. This is the case of the Basque Country, which has created a space to centralize everything that happens around the reuse of its data. This community has been especially useful to collect and measure the work that has been developed on COVID-19 using open data. Under its umbrella there are also activities and competitions that encourage reuse.

For its part, the Castellón Provincial Council has created a Provincial Council of Reusers, a mixed public-private body made up of technicians from the provincial institution itself, and people with recognized professional backgrounds in different economic sectors. These professionals meet once a quarter to hold a conversation to monitor use cases and which favours constant feedback and the enrichment of the Provincial Council's open data strategy. 

The National Library of Spain is working along the same lines and has launched a collaborative work platform so that those citizens who wish to do so can participate in specific projects to enrich the Library's data, making it more accessible and easier to reuse.

In short, all these activities allow monitoring the activity of an open data initiative and its impact on society. They help us to know what challenges we are solving in fields as important for humanity as the environment, health or education. In this way, we will be able to know its evolution over time and easily detect trends and possible areas of improvement, which will lead us to distribute the efforts and resources available in a more effective way.  

Content elaborated by datos.gob.es team.

The difficulties in adequately measuring the impact of open data initiatives are widely known, as it is a global debate that has been with us practically since the beginning of the first open data initiatives, more than ten years ago. The problem is that it is relatively easy to find isolated examples of how various benefits and improvements have been achieved through open data, but in general - and unless we are talking about very specific areas such as economic impact - it is not common to find complete and structured measurements of the impact that has been achieved through open data initiatives in order to demonstrate their full value.

A review of the various initiatives and methods that exist to measure the impact of open data also shows that there is no clear consensus among researchers on how best to capture the results and impact of these initiatives. 

In view of this scenario, we share here the approach to impact measurement that we have been following from the Spanish Government's open data and public information reuse initiative and that is inspired by the recommendations provided by several guides that we consider to be reference in this aspect: the UNE 178301:2015 standard, the framework of common methods for the evaluation of open data, the guide for the empirical analysis of open government data initiatives and the taxonomy of the impact of open data.

Within our personal approach to the problem we must first clarify what we understand “impact” as – “any positive effect or benefit obtained directly or indirectly for individuals, communities or society as a whole, which occurs over a certain period of time and which results from the development of different activities in a given field characterized by the use of open data as a means to an end”. 

Thus, the method we use to measure such impact is based on two main components: a quantitative analysis through indicators on data publication and its characteristics, and a qualitative one through the collection of data use cases.

Quantitative analysis

The objective of quantitative analysis is to offer a series of indicators in a quick and simple way that provide us with an overview of the data publication activity. In this way, we can track its evolution over time and easily detect trends and possible areas for improvement. This will help us to distribute available efforts and resources more effectively. 

The datos.gob.es initiative has its own public control panel that provides quantitative indicators on the following aspects:

Qualitative analysis

Through the qualitative approach, use cases are identified from different sources that can include personalized interviews, content available online, information provided by different media or academic publications. 

These use cases help us to understand the extent to which open data can be considered to have led to positive changes in three main areas described below:

Government level

Including the possible impact on the transparency and accountability of governments, as well as on improving their efficiency and effectiveness. Some of the issues that can be raised in this area are:

• How does openness of data help to improve government efficiency?
• How is data used to examine government resource use and improve existing public services?
• How is open data used to create new public services?
• How does open data contribute to increased transparency and accountability through public scrutiny?

Examples of measurements that are also useful in measuring the impact of open data actions in this area are:

•     Increased collaboration between different government departments and agencies.
•     Creation of platforms and applications that allow citizens to report on their experience in government services.
•     Greater participation of citizens in the formulation of public policies.
•     Improved public resource planning.
•     Reduction in citizens' perception of corruption.

Social area

Including the environmental aspects and the improvements obtained in the inclusion of minority groups in society. Some of the issues that can be raised in this area are:

•     What are the social benefits obtained through the use of open data?
•     How is open data used to improve equality and direct public resources to the citizens who need them most at any given time?
•     What are the benefits of open data in the area of the environment, climate change, pollution, or sustainability?

Examples of indicators that are also useful when measuring the impact of open data actions in this area are:

•     Evidence of greater equality in terms of age, gender, race, social class, disability, geographical location or poverty level.
•     Evidence of improvement in social policies.
•     Number and effectiveness of sustainability programs implemented as a result of the open data
•     Increased attention to environmental factors in project planning.
•     Awareness and sensitization of citizens about their own environmental impacts.

Economic area

Through the influence of open data in supporting existing businesses or creating new business models. Some of the issues we address in this area are:

•     What is the impact of open data on economic growth and innovation?
•     How is open data used to reduce costs in companies and help them become more efficient?
•     What new business models are being developed around open data?

Examples of indicators that are also useful when measuring the impact of data openness actions in this area are:

•     Companies and/or jobs created from the new economy of data openness.
•     New products or services using the open data.
•     Contribution of open data to the growth of the economy in terms of better macroeconomic planning.

 Future areas of improvement

While the method applied may not be perfect - as it may have limitations in terms of the ability to adequately capture desired changes -, in the absence of other reference methodologies, it allows us to explore the emerging impact of ongoing open data initiatives and provide some substantial evidence regarding their impact, at least.

Thus, it would also be desirable to explore new methods in the future that allow for evidence of impact obtained in a more systematic way through the measurement of specific goals and objectives with respect to the initial starting point, and that could also be replicated among the different initiatives, also considering the entire data value chain in the measurements carried out. Some interesting options to be able to put these objectives into practice could be the methods of outcome mapping or social return on investment (SROI) -both already proposed previously in the field of open data but with very limited or no acceptance so far, due most probably to the higher added cost that their implementation would imply.

Content prepared by Carlos Iglesias, Open data Researcher and consultan, World Wide Web Foundation.

Contents and points of view expressed in this publication are the exclusive responsibility of its author.