HVD

The cross-cutting nature of open data on weather and climate data has favoured its use in areas as diverse as precision agriculture, fire prevention or the precision forestry. But the relevance of these datasets lies not only in their direct applicability across multiple industries, but also in their contribution to the challenges related to climate change and environmental sustainability challenges related to climate change and environmental sustainability, which the different action lines of the which the different action lines of the European Green Pact seek to address.

Meteorological data are considered by the European Commission, high value data in accordance with the annex to Regulation 2023/138. In this post we explain which specific datasets are considered to be of high value and the level of availability of this type of data in Spain.

The State Meteorological Agency

In Spain, it corresponds to the State Agency for Meteorology (AEMET) the mission of providing meteorological and climatological services at national level. As part of the Ministry for Ecological Transition and the Demographic Challenge. AEMET leads the related activities of observation, prediction and study of meteorological and climatic conditions, as well as research related to these fields. Its mission includes the provision and dissemination of essential information and forecasts of general interest. This information can also support relevant areas such as civil protection, air navigation, national defence and other sectors of activity.

In order to fulfil this mission, AEMET manages an open data portal that enables the reuse by natural or legal persons, for commercial or non-commercial purposes, of part of the data it generates, prepares and safeguards in the performance of its functions. This portal, known as AEMET OpenData currently offers two modalities for accessing and downloading data in reusable formats:

• General access, which consists of graphical access for the general public through human-friendly interfaces.
• AEMET OpenData API, designed for periodic or scheduled interactions in any programming language, which allows developers to include AEMET data in their own information systems and applications.

In addition, in accordance with Regulation 2023/138, it is envisaged to enable a third access route that would allow re-users to obtain packaged datasets for mass downloading where possible.

In order to access any of the datasets, an access key (API Key) which can be obtained through a simple request in which only an e-mail address is required, without any additional data from the applicant, for the sending of the access key. This is a control measure to ensure that the service is provided with adequate quality and in a non-discriminatory manner for all users.

AEMET OpenData also pioneered the availability of open meteorological data in Europe, reflecting AEMET''s commitment to the continuous improvement of meteorological services, support to the scientific and technological community, and the promotion of a more informed and resilient society in the face of climate challenges.

High-value meteorological datasets

The Annex to Regulation (EU) 2023/138 details five high-value meteorological data sets: weather station observations, validated weather data observations, weather warnings, radar data and numerical prediction model (NMP) data. For each of the sets, the regulation specifies the granularity and the main attributes to be published.

If we analyse the correspondence of the datasets that are currently available grouped in 14 categories in the portal AEMET OpenData portal, with the five datasets that will become mandatory in the coming months, we obtain the conclusions summarised in the following table:

Figure 1: Table showing the equivalence between high value datasets and AEMET OpenData datasets.

The regulation also sets out a number of requirements for publication in terms of format, licence granted, frequency of updating and timeliness, means of access and metadata provided.

In the case of metadata, AEMET publishes, in machine-readable format, the main characteristics of the downloaded file: who prepares it, how often it is prepared, what it contains and its format, as well as information on the data fields (meteorological variable, unit of measurement, etc.). The copyright and terms of use are also specified by means of the legal notice. In this regard, it is foreseen that the current licences will be reviewed to make the datasets available under a licensing scheme compliant with the Regulation, possibly following the recommendation by adopting the license CC BY-SA 4.0.

All in all, it seems that the long track record of the State Meteorological Agency (AEMET) in providing quality open data has put it in a good position to comply with the requirements of the new regulation, making some adjustments to the datasets it already offers through AEMET OpenData to align them with the new obligations. AEMET plans to include in this service the datasets required by the Regulation and which are currently not available, as it adapts its regulations on public prices, as well as the infrastructure and systems that make this possible. Additional datasets that will be available will be ten-minute observation data, hourly climatologies and some data parameters from regional radars and numerical wave and forecast models.

Spain, as part of the European Union, is committed to the implementation of the European directives on open data and re-use of public sector information. This includes the adoption of initiatives such as the Implementing Regulation (EU) 2023/138 issued by the European Commission, which defines specific guidelines for government entities with regard to the availability of High value datasets (HVD). These data are categorised into themes previously detailed in earlier discussions: Geospatial, Earth Observation and Environment, Meteorology, Statistics, Societies and Societal Properties, and Mobility. In this article we will focus on the last group mentioned.

The Mobility category encompasses data collections falling under the domain of "Transport Networks", as demarcated in Annex I of the Directive 2007/2/EC of the European Parliament and of the Council of 14 March 2007 establishing an Infrastructure for Spatial Information in the European Community (INSPIRE). In particular, this Directive refers to the requirement to make available to users datasets relating to road, rail, air and inland waterway networks, with their associated infrastructure, connections between different networks and the trans-European transport network, as defined by Decision No 1692/96/EC of the European Parliament and of the Council of 23 July 1996 on Community guidelines for the development of the trans-European transport network.

In addition, it includes the datasets as described in the Directive 2005/44/EC of the European Parliament and of the Council of 7 September 2005 on harmonised River Information Services (RIS) on inland waterways in the Community. The main objective of the Directive is to improve inland waterway traffic and transport, and it applies to canals, rivers, lakes and ports capable of accommodating vessels of between 1,000 and 1,500 tonnes. These datasets include:

Figure 1: Table with the high value datasets related to Directive 2005/44/EC for the creation of a trans-European river information network.

In order for all of us to make the most of the information available, the Regulation defines some basic rules on how this data is shared:

• Free and easy to use. The data must be ready to be used and shared with everyone for any purpose by acknowledging and citing the source of the data, as prescribed by the Creative Commons BY 4.0 licence.
• Easy to read and use. Data will be presented in a way that both people and computers can easily understand them and everything will be explained in public.
• Direct and easy access. There will be special ways (called APIs) that allow programs to access data automatically. In addition, the user can alternatively download a lot of information at once.
• Always up to date. It is important that data is up to date, so there will be access to the most recent version. But if the user needs to access previous data, it will also be possible to view previous versions.
• Detailed and precise. Data will be shared in as much detail as possible, to a very fine level of accuracy, so that the whole territory is covered when combined.
• Information on information. There will be "information about the information" (metadata) that will tell everything about the data. The metadata shall contain at least the elements listed in the Annex to Commission Regulation (EC) No 1205/2008 of 3 December 2008.
• Understandable and orderly: It will explain well how the data are organised and what all means, in a way that is easy for everyone to understand (structure and semantics).
• Common language. Data shall use vocabularies, code lists and categories that are recognised and accepted at European or global level.

in Spain, who is responsible for the creation and maintenance of mobility data?

In Spain, the responsibility for the creation and maintenance of mobility data generally lies with different governmental entities, depending on the type of mobility and the territorial scope:

• Level national level. The Ministry of Transport and Sustainable Mobility is the main body in charge of mobility in terms of infrastructure and transport at national level. This would include data on roads, railways, air and maritime transport.
• Regional and local level. Autonomous communities and municipalities also play an important role in urban and regional mobility. They are responsible for urban mobility, public transport and public roads, within their respective jurisdictions.
• Public business entities. There are entities such as ADIF (acronym for Administrador de Infraestructuras Ferroviarias, that is Railway Infrastructure Administrator), AENA (acronym for Aeropuertos Españoles y Navegación Aérea, that is Spanish Airports and Air Navigation), Puertos del Estado (State Ports) and others tentities hat manage specific data related to their field of action in rail, air and maritime transport, respectively.

In Spain, the Ministry of Transport and Sustainable Mobility, in collaboration with the autonomous communities, plays a key role in providing access to a wide range of mobility data. In compliance with INSPIRE and LISIGE (Law 14/2010 of 5 July 2010 on geographic information infrastructures and services in Spain, which transposes the INSPIRE Directive), it offers resources such as the Geoportal of the Spatial Data Infrastructure of Spain (IDEE in Spanish acronyms) where citizens and professionals can access geographic data and services, especially with regard to mobility.

Does Spain comply with the HVD Mobility Regulation?

To solve this question we have to go to the INSPIRE Geoportal  where official information classified as high value datasets in Europe is available. Specifically in the mobility category.

Figure 2: Screenshot of the Inspire Geoportal.

As of April 2024 Spain has published the following information in the INSPIRE Geoportal:

• Port service areas in Spain. The port service areas include the cartographic and alphanumeric information of the land service area and water areas I and II. The Spanish State-owned Port System is made up of 46 ports of general interest, managed by 28 Port Authorities.
• Spanish Transport Networks. The Transport Network of the Geographic Reference Information of the National Cartographic System of Spain is a three-dimensional network of national coverage, defined and published in accordance with the INSPIRE Directive, which contemplates five modes of transport: road, rail, inland waterways, air and cable, together with their respective intermodal connections and the infrastructures associated with each mode. This information has the linear geometry of the roads and the punctual geometry of the portals and kilometre points.
• ADIF''s Spanish Rail Transport Network. Public geographic dataset on the adaptation of the Spanish ADIF Common Traamification to the INSPIRE regulations (Transport Networks Annex I).

The publication of these high-value datasets responds positively to the question of Spain''s compliance with the HVD regulation, and is an achievement that reflects Spain''s continued commitment to transparency and access to mobility data.

The joint effort between the Ministry of Transport, Mobility and Urban Agenda, the National Cartographic System and the Autonomous Communities and Public Business Entities underlines the importance of a collaborative approach to mobility information management.

The availability of this data highlights Spain''s commitment to publishing high-value datasets and underlines the importance of continuously improving access to information to optimise inland navigation and mobility data.

Content prepared by Mayte Toscano, Senior Consultant in Data Economy Technologies. The contents and points of view reflected in this publication are the sole responsibility of its author.

The European Commission Implementing Regulation (EU) 2023/138 sets clear guidelines for public bodies on the availability of high-value datasets within 16 months from 20 January 2023. These high-value high value datasets (High value datasets or HVD) are grouped into the following themes, which were already described in this post: geospatial Earth observation and environment, Meteorology, Statistics, Societies and corporate ownership, and Mobility.

This article focuses on the category of Earth observation and environment, referred to as High Value Datasets for Earth Observation and Environment (HVDOM ).

Earth observation and environmental data in the regulation of high-value arrays

The HVDOM include data obtained in the following areas:

• Spatial or remotely sensed data.
• Ground or in situ data.

In particular, public bodies are faced with the obligation to make available to citizens high-value datasets that appear in various pieces of legislation:

1. The environmental and climate datasets listed in the annexes of the Directive 2007/2/EC of the European Parliament and of the Council of 14 March 2007 establishing an Infrastructure for Spatial Information in Europe ( INSPIRE) indicated in the following visual:

2. Data sets produced in the context of legal acts and known as priority data, defined in the table below:

At this URL of the INSPIRE register, the list of datasets related to environmental reporting, to be made available by Member States in a step-by-step manner, is published.

To ensure the accessibility and re-use of all these valuable datasets, it is imperative to follow certain provisions to facilitate their publication. Here are the key requirements, which are common to all categories:

• Open licence: All datasets must be made available for reuse under a licence Creative Commons BY 4.0 or any equivalent less restrictive open licence. This encourages the freedom to share and adapt information.
• Open and machine-readable format: Data should be presented in an open, machine-readable format and be publicly documented. This ensures that the information is easily understandable and accessible to any person or automated system.
• Application Programming Interfaces (APIs) and bulk downloading: Application programming interfaces (APIs) should be provided to facilitate programmatic access to data such as direct access download services like Web Feature Services, WFS. In addition, direct bulk downloading of datasets should be possible, allowing flexible options for users according to their needs
• Updated version: The availability of datasets in their most up-to-date version is essential. This ensures that users have access to the latest information, promoting the relevance and accuracy of data. Therefore, it will be necessary to have the most up-to-date datasets as well as historical versions of the available datasets.
• Scale and granularity: Data will need to be published at the available levels of generalisation up to the scale of 1:5000, and covering the whole Member State when combined. If datasets are not available at this scale, but are available at higher spatial resolutions, they shall be provided at the available spatial resolution.
• Metadata: The metadata describing the data concerning the set of themes within the scope of the INSPIRE infrastructure for spatial information shall contain at least the metadata elements set out in the Annex to the Commission Regulation (EC) No 1205/2008 of 3 December 2008of 3 December 2008.
• Semantics: The datasets shall be described in a comprehensive and publicly available online documentation describing at least the data structure and semantics.
• Vocabularies: The datasets shall use Union-controlled or internationally recognised and publicly documented vocabularies and taxonomies. Such as the checklists defined in the INSPIRE register.

In Spain, who is responsible for the creation and maintenance of the Earth Observation and Environment HVDs ?

There are several organisations responsible for these datasets, for example, at the national level, we highlight:

• The National Cartographic System (NCS), which publishes orthoimagery, hydrography, elevations, land use or land cover.
• The Geological and Mining Institute (IGME), responsible for geological data.
• The Spanish Institute of Oceanography  (IEO) of the Spanish National Research Council (CSIC) is responsible for oceanographic features.
• The General Directorate of Cultural Heritage and Fine Arts of the Ministry of Culture publishes the protected sites of historical heritage and culture.
• The Ministry for Ecological Transition and Demographic Challenge (MITECO) and the Ministry of Agriculture, Fisheries and Food (MAPA) are responsible for the datasets subject to other environmental legislation, the priority data.

All these organisations play a crucial role in the implementation and enforcement of the INSPIRE Directives  INSPIRE Directives y LISIGE (Law on Geographic Information Infrastructures and Services in Spain), both of which are fundamental in the management of geographic and environmental information.

• The INSPIRE Directive was developed in cooperation with the Member States and accession countries of the European Union. Its main objective is to make available the geographical information needed for the management of EU environmental policies.
• In Spain, it is transposed by Law 14/2010, known as LISIGE, which was subsequently amended by Law 2/2018 Law 2/2018. This law regulates geographic data and services, applying to those that refer to a geographic area of the national territory and that have been generated or are under the responsibility of public administrations. These data must be in electronic format and be the responsibility of a public sector administration or body.

Both the SCN an instrument regulated by the LISIGE that publishes geographic information through the coordination of national organisations and the Autonomous Communities, has been set up by the LISIGE. The Spanish and Spanish public administrations, such as MITECO and MAPA, in coordination with the autonomous communities, provide access to a wide range of data and information related to biodiversity, climate change, environmental quality and many other environmental issues. This includes the maintenance of the Spatial Data Infrastructure (SDI) of MITECO and MAPA, which is aligned with the INSPIRE, environmental and LISIGE Directives, publishing the datasets through interoperable services and locatable in the Official Catalogue of INSPIRE Data and Services (CODSI)where geographic data and services regulated by the LISIGE on environmental and Earth observation matters are available .

We highlight, for example, that MITECO is making efforts to ensure the adaptation of the main datasets of the Spanish Inventory of Natural Heritage and Biodiversity to the INSPIRE Directive, covering topics such as biogeographic regions, habitats, biotopes and species distribution.

 MITECO plays an essential role in the implementation of the INSPIRE and LISIGE Directives in Spain, providing access to key geographic data and information for the management of environmental policies and the protection of natural heritage, as historical and cultural heritage falls under the Ministry of Culture.

Does Spain comply with the HVD Regulation on Earth observation and environment?
 

Thanks to the continued work of the Consejo Directivo de la Infraestructura de Información Geográfica de España (CODIIGE) since the implementation of LISIGE, Spain has reached an important achievement. Currently, in the INSPIRE Geoportal portal is currently available a wide variety of information, classified as high-value Earth observation and environmental datasets. This information comes from the resources stored in CODSI.

As of May 2024, Spain has published the following information on the CODSI Geoportal:

• INSPIRE Datasets:  The number of datasets published by Spain in each of the mandatory INSPIRE themes is shown below.

See here for the definition of each of the INSPIRE themes.

• Environmental legislation: refers to priority datasets grouped by relevant environmental reporting legislation. The commission called these datasets "Priority Data" to reinforce their importance once the baseline data were already published

https://inspire-geoportal.ec.europa.eu/srv/eng/catalog.search#/qsEnvDom…

All categories are listed below (also indicating those that do not apply to Spain with the acronym N/D):

*N/D= No data, these categories do not apply to Spain

The publication of all these high-value datasets is an achievement that reflects our country''s continued commitment to transparency and access to high-quality environmental and Earth observation data.

Spain has made significant progress in complying with the High Value Sets of Earth Observation and Environment Regulation, with a remarkable amount of datasets available through the INSPIRE Geoportal. This ranges from Directive 2008/50/EC on air quality to Directive 2008/56/EC on Marine Strategy, reflecting a holistic approach to environmental information management.

Compliance with European Directives and regulations demonstrates Spain''s commitment to transparency and access to high quality environmental information. In addition, efforts to ensure the updating and accessibility of data are highlighted, including the adaptation of the main datasets of the Spanish Inventory of Natural Heritage and Biodiversity to the INSPIRE Directive.

The joint effort between MITECO, MAPA and the National Cartographic System and the autonomous communities underlines the importance of a collaborative approach to the management of environmental information and Earth observation, fundamental in the protection and sustainable development of natural heritage.

In short, we can say that Spain currently complies with the HVD Regulation in almost all categories, but does not comply in the following categories:

No API or bulk download

• Orthoimaging
• Oceanographic geographical features

Lack of data related to the following environmental legislation:

• Directive 98/83/EC - Drinking Water Directive
• Directive 2002/49/EC - Noise Directive
• Directive 2012/18/EU - SEVESO III Directive
• National legislation - National bio-geographical regions
• Recommendation 2014/70/EU - Recommendation on hydraulic fracturing
• Regulation (EU) 2017/852 - Regulation of Mercury

In summary, Spain has demonstrated a strong commitment to transparency and accessibility to high quality environmental information, complying with most European directives, although it still needs to improve in specific areas such as the provision of APIs and the updating of certain environmental datasets.

Content prepared by Mayte Toscano, Senior Consultant in Data Economy Technologies. The contents and points of view reflected in this publication are the sole responsibility of the author.

The concept of High-Value data (High-Value datasets) was introduced by the European Parliament and the Council of the European Union 4 years ago, in Directive (EU) 2019/1024. In it, they were defined as a series of datasets with a high potential to generate "benefits for society, the environment and the economy". Therefore, member states were to push for their openness for free, in machine-readable formats, via APIs, in the form of bulk download and comprehensively described by metadata. 

Initially, the directive proposed in its annex six thematic categories to be considered as high value: geospatial, earth observation and environmental, meteorological, statistical, business records and transport network data. These categories were subsequently detailed in an implementing regulation published in December 2022. In addition, to facilitate their openness, a document with guidelines on how to use DCAT-AP for publication was published in June 2023. 

New categories of data to be considered of high value  

These initial categories were always open to extension. In this sense, the European Commission has just published the report "Identification of data themes for the extensions of public sector High-Value Datasets" which includes seven new categories to be considered as high-value data  

•  Climate loss: This refers to data related to approaches and actions needed to avoid, minimize and address damages associated with climate change. Examples of datasets in this category are economic and non-economic losses from extreme weather events or slow-onset changes such as sea level rise or desertification. It also includes data related to early warning systems for natural disasters, the impact of mitigation measures, or research data on the attribution of extreme events to climate change. 

• Energy: This category includes comprehensive statistics on the production, transport, trade and final consumption of primary and secondary energy sources, both renewable and non-renewable. Examples of data sets to consider are price and consumption indicators or information on energy security.   

• Finance: This is information on the situation of private companies and public administrations, which can be used to assess business performance or economic sustainability, as well as to define spending and investment strategies. It includes datasets on company registers, financial statements, mergers and acquisitions, as well as annual financial reports.  

• Government and public administration: This theme includes data that public services and companies collect to inform and improve the governance and administration of a specific territorial unit, be it a state, a region or a municipality. It includes data relating to government (e.g. minutes of meetings), citizens (census or registration in public services) and government infrastructures. These data are then reused to inform policy development, deliver public services, optimize resources and budget allocation, and provide actionable and transparent information to citizens and businesses. 

• Health: This concept identifies data sets covering the physical and mental well-being of the population, referring to both objective and subjective aspects of people's health. It also includes key indicators on the functioning of health care systems and occupational safety. Examples include data relating to Covid-19, health equity or the list of services provided by health centers.  

• Justice and legal affairs: Identifies datasets to strengthen the responsiveness, accountability and interoperability of EU justice systems, covering areas such as the application of justice, the legal system or public security, i.e. that which ensures the protection of citizens. The data sets on justice and legal matters include documentation of national or international jurisprudence, decisions of courts and prosecutors general, as well as legal acts and their content. 

• Linguistic data: Refers to written or spoken expressions that are at the basis of artificial intelligence, natural language processing and the development of related services. The Commission provides a fairly broad definition of this category of data, all of which are grouped under the term "multimodal linguistic data". They may include repositories of text collections, corpora of spoken languages, audio resources, or video recordings.  

To make this selection, the authors of the report conducted desk research as well as consultations with public administrations, data experts and private companies through a series of workshops and surveys. In addition to this assessment, the study team mapped and analyzed the regulatory ecosystem around each category, as well as policy initiatives related to their harmonization and sharing, especially in relation to the creation of European Common Data Spaces. 

Potential for SMEs and digital platforms   

In addition to defining these categories, the study also provides a high-level estimate of the impact of the new categories on small and medium-sized companies, as well as on large digital platforms. One of the conclusions of the study is that the cost-benefit ratio of data openness is similar across all new topics, with those relating to the categories "Finance" and "Government and public administration" standing out in particular. 

Based on the publicly available datasets, an estimate was also made of the current degree of maturity of the data belonging to the new categories, according to their territorial coverage and their degree of openness (taking into account whether they were open in machine-readable formats, with adequate metadata, etc.). To maximize the overall cost-benefit ratio, the study suggests selecting a different approach for each thematic category: based on their level of maturity, it is recommended to indicate a higher or lower number of mandatory criteria for publication, thus ensuring to avoid overlaps between new topics and existing high-value data.  

You can read the full study at this link.