Portal Europeo de Datos

Measuring the impact of open data is one of the challenges facing open data initiatives. Ther are a variety of methods, most of which combine quantitative and qualitative analysis in order to understand the value of specific datasets.

In this context, data.europa.eu, the European Open Data Portal, has launched a Use Case Observatory. This is a research project on the economic, governmental, social and environmental impact of open data.

What is the Use Case Observatory?

For three years, from 2022 to 2025, the European Data Portal will monitor 30 cases of re-use of open data. The aim is to:

1. Assess how the impact of open data is created.
2. Share the challenges and achievements of the analysed re-use cases
3. Contribute to the debate on the methodology to be used to measure such impact.

The analysed use cases refer to four areas of impact:

• Economic impact: includes reuse cases related to business creation and (re)training of workers, among others. For example, solutions that help identify public tenders or apply for jobs are included.
• Governmental impact: This refers to reuse cases that drive e-government, transparency and accountability.
• Social impact: includes cases of re-use in the fields of healthcare, welfare and tackling inequality.
• Environmental impact: This is limited to cases of re-use that promote sustainability and energy reduction, including solutions related to air quality control or forest preservation.

To select the use cases, an inventory was made based on three sources: the examples collected in the maturity studies carried out each year by the European portal, the solutions participating in the EU Datathon and the examples of reuse available in the repository of use cases on data.europa.eu. Only projects developed in Europe were taken into account, trying to maintain a balance between the different countries. In addition, projects that had won an award or were aligned with the European Commission's priorities for 2019 to 2024 were highlighted. To finalise the selection process, data.europa.eu conducted interviews with representatives of the use cases that met the requirements and were interested in participating in the project.

Three Spanish projects among the use cases analysed

The selected use cases are shown in the following image:

Among them, there are three Spaniards:

• In the Social Impact category is UniversiDATA-Lab, a public portal for the advanced and automatic analysis of datasets published by universities. This project, which won the first prize in the III Desafío Aporta, was conceived by the team that created UniversiData, a collaborative initiative oriented and driven by public universities with the aim of promoting open data in the higher education sector in Spain in a harmonised way. You can learn more about these projects in this interview.
• In the same category we also find Tangible data, a project focused on the creation of sculptures based on data, to bring them closer to non-technical people. Among other data sources, it uses datasets from NASA or Our World in Data.
• In the environment category is Planttes. This is a citizen science project designed to report on the presence of allergenic plants in our environment and the level of allergy risk depending on their condition. This project is promoted by the Aerobiological Information Point (PIA) of the Institute of Environmental Science and Technology (ICTA-UAB) and the Department of Animal Biology, Plant Biology and Ecology (BABVE), in collaboration with the Computer Vision Centre (CVC) and the Library Living Lab, all of them at the Autonomous University of Barcelona (UAB).

First report now available

As a result of the analysis carried out, three reports will be developed. The first report, which has just been published, presents the methodology and the 30 selected cases of re-use. It includes information on the services they offer, the (open) data they use and their impact at the time of writing. The report ends with a summary of the general conclusions and lessons learned from this first part of the research project, giving an overview of the next steps of the observatory.

The second and third reports, to be released in 2024 and 2025, will assess the progress of the same use cases and expand on the findings of this first volume. The reports will focus on identifying achievements and challenges over a three-year period, allowing concrete ideas to be extrapolated to improve methodologies for assessing the impact of open data.

The project was presented in a webinar on 7 October, a recording of which is available, together with the presentation used. Representatives from 4 of the use cases were invited to participate in the webinar: Openpolis, Integreat, ANP, and OpenFoodFacts.

This report published by the European Data Portal explores the so-called Citizen Generated Data (CGD). This category of data refers to those generated by citizens. There is a lack of this type of data within European open data portals, mainly due to the lack of publication and management of CGDs by public administrations. 

The document analyzes various open data portals, whose main objective is to provide a vision of the CGDs that can be part of these portals and how to include them by public administrations. It should be noted that during the analysis, a framework is established for the description, reference, and characterization of the CGDs. 

Finally, based on the conclusions of the previous analysis, the document offers a series of recommendations and guidelines for data publishers. The objective is to increase and improve the presence of CGDs in the publication of open data, involving citizens in the design of policy, processes and governance. 

This report is available at the following link: "Data.europa.eu and citizen-generated data" 

The European Directive 2019/1024 on open data and re-use of public sector information emphasises, among many other aspects, the importance of publishing data in real time. In fact, the document talks about dynamic data, which it defines as "documents in digital format, subject to frequent or real-time updates due to their volatility or rapid obsolescence". According to the Directive, public bodies must make this data available for re-use by citizens immediately after collection, through appropriate APIs and, where possible, as a bulk download.

To explore this further, the European Data Portal, Data.europa.eu, has published the report “Real-time data 2022: Approaches to integrating real-time data sources in data.europa.eu” which analyses the potential of real-time data. It draws on the results of a webinar held by data.europa.eu on 5 April 2022, a recording of which is available on its website.

In addition to detailing the conclusions of the event, the report provides a brief summary of the information and technologies presented at the event, which are useful for real-time data sharing.

The importance of real-time data

The report begins by explaining what real-time data are: data that are frequently updated and delivered immediately after collection, as mentioned above. These data can be of a very heterogeneous nature. The following table gives some examples:

This type of data is widely used to shape applications that report traffic, energy prices, weather forecasts or flows of people in certain spaces. You can find out more about the value of real-time data in this other article.

Real-time data sharing standards

La interoperabilidad es uno de los factores más importantes a tener en cuenta a la hora de seleccionar la tecnología más adecuada para el intercambio de datos en tiempo real. Se precisa un lenguaje común, es decir, formatos de datos comunes e interfaces de acceso a datos que permitan el flujo de datos en tiempo real. Dos estándares que ya son muy utilizados en el ámbito del Internet de las cosas (IoT en sus siglas en inglés) y que pueden ayudar en este sentido son:

SensorThings API (STA)

SensorThings API, from the Open Geospatial Consortium, emerged in 2016 and has been considered a best practice for data sharing in compliance with the INSPIRE Directive.

This standard provides an open and unified framework for encoding and providing access to sensor-generated data streams. It is based on REST and JSON specifications and follows the principles of the OData (OASIS Open Data Protocol) standard.

STA provides common functionalities for creating, reading, updating and deleting sensor resources. It enables the formulation of complex queries tailored to the underlying data model, allowing more direct access to the specific data the user needs. Query options include filtering by time period, observed parameters or resource properties to reduce the volume of data downloaded. It also allows sorting the content of a result by user-specified criteria and provides direct integration with the MQTT standard, which is explained below.

Message Queuing Telemetry Transport (MQTT)

MQTT was invented by Dr. Andy Stanford-Clark of IBM and Arlen Nipper of Arcom (now Eurotech) in 1999. Like STA, it is also an OASIS standard.

The MQTT protocol allows the exchange of messages according to the publish/subscribe principle. The central element of MQTT is the use of brokers, which take incoming messages from publishers and distribute them to all users who have a subscription for that type of data. In this type of environment, data is organised by topics, which are freely defined and allow messages to be grouped into thematic channels to which users subscribe.

The advantages of this system include reduced latency, simplicity and agility, which facilitates its implementation and use in constrained environments (e.g. with limited bandwidth or connectivity).

 In the case of the European portal, users can already find real-time datasets based on MQTT. However, there is not yet a common approach to providing metadata on brokers and the topics they offer, and work is still ongoing.

Other conclusions of the report

As mentioned at the beginning, the webinar on 5 April also served to gather participants' views on the use of real-time data, current challenges in data availability and needs for future improvements. These views are also reflected in this report.

Among the most valued categories of real-time data, users highlighted traffic information and weather data. Data on air pollution, allergens, flood monitoring and stock market information were also mentioned. In this respect, more and more detailed data were requested, especially in the field of mobility and energy in order to be able to compare commodity prices.  Users also highlighted some drawbacks in locating real-time data on the European portal, including the heterogeneity of the information, which requires the use of common standards and formats across countries.

Finally, the report provides a set of recommendations on how to improve the ability to locate real-time data sources through data.europa.eu. To this end, a series of short and medium-term actions have been established, including the collection of use cases, support for data providers and the development of best practices to unify metadata.

You can read the full report here.

This report published by the European Data Portal (EDP) aims to help open data users in harnessing the potential of the data generated by the Copernicus program. 

The Copernicus project generates high-value satellite data, generating a large amount of Earth observation data, this is in line with the European Data Portal's objective of increasing the accessibility and value of open data. 

The report addresses the following questions, What can I do with Copernicus data? How can I access the data?, and What tools do I need to use the data? using the information found in the European Data Portal, specialized catalogues and examining practical examples of applications using Copernicus data.  

This report is available at this link: "Copernicus data for the open data community"

This report published by the European Data Portal (EDP) covers the following topics.  

What are the needs of open data re-users of public sector institutions in Europe? This question is key to facilitate the publication of open data and support for re-users of EU institutions and Member States' public authorities based on their needs for policy-making, service delivery and organizational management. 

 This discussion paper provides an overview of the state of existing methods and indicators in the European open data landscape to assess the needs of public institutions as data re-users. 

 This overview serves as a basis for a discussion with public sector stakeholders on appropriate methods and indicators for measuring the demand for data from public institutions to encourage the publication of demand-driven data. 

The report is available at this link: "Measuring Data Demand Within the Public Sector"

This report published by the European Data Portal (EDP) covers the following topics. 

Making data available as open data in all EU Member States is vital to harnessing its potential for European society and economy. In order to increase impact effectively, efforts must target the datasets that have the greatest potential in society and the economy. 

 In the regulation on open data and re-use of public sector information, the European Commission is mandated to adopt an implementing regulation specifying high-value datasets. 

 The line of argument developed in this report parallels what the Commission has done during the first quarter of 2021, to prepare the implementing regulation that includes a list of high-value datasets. This report reviews relevant literature, policy decisions and national initiatives to enable a deeper understanding of the situation around assessing the value of datasets. 

The report is available at this link: "High-Value Datasets: Understanding the Data Providers' Perspective" 

This report published by the European Data Portal (EDP) explores existing and emerging developments and initiatives around data sharing using data spaces. 

The objective is twofold: to identify the owners of open data involved in the implementation of data spaces and to reflect on the role that open data portals (with special attention to data.europa.eu) could play in this implementation. 

After documentary research and interviews with the promoters of data spaces, it is analyzed how data.europa.eu could be positioned in the common European data spaces that are emerging. 

The report is available at this link: "Data.europa.eu y los espacios comunes de datos europeos: un informe sobre retos y oportunidades"

This report published by the European Data Portal (EDP) aims to advance the debate on the medium and long-term sustainability of open data portal infrastructures. 

It provides recommendations to open data publishers and data publishers on how to make open data available and how to promote its reuse. It is based on the previous work done by the data.europa.eu team, on research on open data management, and on the interaction between humans and data. 

Considering the conclusions, 10 recommendations are proposed for increasing the reuse of data. 

The report is available at this link: " Principles and recommendations to make data.europa.eu data more reusable: A strategy mapping report "

For the seventh consecutive year, the European Data Portal has presented its Open Data Maturity assessment. This is a report that analyses a series of indicators to measure the maturity of open data across Europe, in order to understand the progress of each country, make comparisons and find areas for improvement.

This year, Spain ranks third with a score of 95%, 14 points above the European average (81%). First place goes to France (98%), followed by Ireland (also with 95%).

Spain, above the European average in all the analyzed categories

As every year, the report measures 4 categories:

• Policy, focusing on the existence of specific policies and strategies to promote open data at the national level.
• Impact, which analyzes activities to monitor and measure the reuse of open data.
• Portal, which evaluates the existing functions in the national platform (in the case of our country, datos.gob.es) to access open data and interact with the community.
• Quality, which studies the mechanisms that guarantee the quality of (meta)data.

Spain is ahead of Europe in all categories:

In total, Spain's score has increased by one percentage point compared to 2020, when it stood at 94%. It stands out especially in the impact category, where it is in first position with 100% of the score. The report highlights how the Aporta Initiative conducts an annual national survey to ascertain the control over the use of data by the various national, regional and local public bodies. It also highlights the collaborations that the institutions carry out with universities, such as the agreement adopted by the Valencian Government with all public universities in the Community to work together in training and research activities related to transparency and open data.

Our country continues in the group of open data prescribers

As part of the analysis, the European Portal groups all participating countries into four groups based on the assessment of open data maturity during 2021. With its score, Spain continues to place itself in the leading group of open data in Europe, called trend-setters. The group with the best results is completed by France, Ireland, Poland, Estonia and Ukraine.

These countries are characterized by an advanced open data policy and a platform with a wide range of functions to meet the needs of advanced users and publishers. They are also countries where there are initiatives to ensure the publication of high-quality data and compliance with the DCAT-AP European open data cataloging profile, and where communities around data reuse are in place. Finally, it is highlighted that there are established methodologies in place to assess the impact of data in the different domains.

Europe's average score rises

In 2021, the average score relating to open data maturity of EU27 countries is 81%, an increase of three percentage points compared to 2020.

The report highlights three trends observed in all the countries analysed:

• Member States are updating their policies to transpose the Open Data Directive into their national legislation. More and more countries are carrying out the transposition process. In the case of Spain, it was included in the Royal Decree-Law 24/2021, approved last December 2 by the Congress. It is worth mentioning that the Open Data Directive still requires the adoption of a future implementing regulation by the European Commission, where the list of high-value datasets to be provided free of charge through APIs is included.
• Understanding, monitoring and measuring the impact of open data is becoming more important. The impact dimension has seen the most growth in the last year, while the quality indicator is the least mature. In recent years, impact has been measured mainly through activities such as maintaining records of data downloads or creating lists of use cases. This year there has been a clear trend towards in-depth studies, such as desk research or surveys, to quantify and verify the impact of open data.
• The COVID-19 pandemic continues to highlight the value and impact of open data.  Last year, the health situation led to highlight the importance of systematically collecting data and making it available to the public in an up-to-date manner. This year, the available data is complemented by statistics related to vaccination.

Future recommendations

The report ends with a series of recommendations for each group of countries. In the case of prescribers, a group that includes Spain, some issues are highlighted, such as continuing to develop thematic communities of providers and reusers, giving priority to the high-value data categories highlighted in the Open Data Directive. In this sense, Spain is moving forward with the creation of specific hubs for various sectors such as tourism or health.

Other examples of recommendations are to collaborate with other national and international teams to develop an experimental impact assessment framework and with academic institutions to provide advanced courses and training on open data.

You can see the full report here and the Spain report here.

The European Data Portal (EDP) has presented its report "Copernicus data for the open data community", prepared by con.terra as part of the EDP consortium. As we have reported before, Copernicus is the European Union's Earth Observation program that provides accurate, timely and easily accessible information to improve environmental management, understand and mitigate the effects of climate change and ensure civil security.

The report aims to help users harness the potential of Copernicus data to create Earth observation applications by answering three basic questions:

• What can I do with Copernicus data?
• How can I access the data?
• What tools do I need to use the data?

After an introduction reviewing the main activities and services available from the program, the report is divided into two parts: a first part where examples of Copernicus data applications are examined and a second, more practical part, where a particular use case is replicated in depth.

Copernicus use cases

The first part covers a series of possible use cases at a general level to answer the first of the questions posed above: what can be done with Copernicus data?

The use cases discussed are linked to the thematic areas addressed by the Copernicus program (emergency, security, marine, land, climate change and atmospheric monitoring), as well as to its services and tools. These examples cover the observation of plastic pollution of the oceans, land change due to mining activities, the impact of volcanic activities, ice loss, the creation of artificial islands, deforestation, forest fires, storms or pests.

The report highlights the importance of knowing what data are appropriate for each specific use case. For example, SENTINEL 2 MSI data is suitable for land monitoring, emergency management and security services, while SENTINEL 3 Altimetry data is linked to the areas of marine monitoring and climate change. To assist in this identification task, the guide includes references to various user guides with specifications on the missions, the instruments used to collect the data and the data products generated.

Case study on the use of Copernicus data 

The second part of the report focuses on a particular use case that it addresses in depth, including how to download the appropriate data, process it and build applications with it. Specifically, it addresses the mapping of the lava flow of the Etna volcano using data from the Copernicus emergency management service. The aim is to track the impact of volcanic activities on nature and urban areas.

First, the report shows how to search and download data for this area of interest.  In this case, Sentinel-2 products are used from the Copernicus Open Access Hub. The entry point for accessing the Copernicus data is their own website, which provides an overview of the data access points. Through different images, the report shows search and filter options to locate the appropriate data.

To visualize and process the data, it is proposed to use commercial software such as ArcGIS Pro,  free GIS tools such as QGIS, open source processing tools such as SNAP or programming libraries such as GDAL. In the case of the example, SNAP (Sentinel Application Platform), the European Space Agency (ESA) platform, is used to view the lava flow.

Some explanations on workflow automation with the Open Access Hub API and the SNAPgraph tool are given at the end of the chapter.

Final conclusions

The report ends with several conclusions, among which the following stand out:

1. Users can extract great value from Copernicus data but to do so they need to be familiar with the platforms involved and the necessary tools.
2. For most use cases, it is necessary to combine Copernicus data with in situ data. The Copernicus program itself uses data from ground-based sensors, for example, for calibration and validation of its products.

Spain's role in Copernicus

The Ministry of Transport, Mobility and Urban Agenda, through the National Geographic Institute, and the Ministry for Ecological Transition (MITECO) represent Spain in the Copernicus User Forum, for the monitoring and evolution of the program. In this interview Nuria Valcárcel, Deputy Assistant Director (Observation of the Territory) of the General Directorate of Geodesy and Cartography, in the D.G. National Geographic Institute (IGN) delves into the services of Copernicus and its usefulness in the economic and social field.

In datos.gob.es you can also find this other interview with Stéphane Ourevitch, founder of SpaceTec, who participated as a speaker at the Encuentro Aporta 2019, where he tells us about the usefulness of data for space observation and how the Copernicus program promotes entrepreneurship through actions such as hackathons.

Copernicus data are very useful all over the world. In our country, we also find multiple services and applications developed based on Copernicus data, some of which are collected in this article.