datos.gob.es How to publish open data quickly and easily (with CKAN)

TABLE OF CONTENTS

1. Introduction

2. Selection of the data to be published

2.1. Criteria

2.2. Topics

2.3. Restrictions

3. Licenses to facilitate reuse

3.1. Establishing simple conditions of use

3.2. Adopting a predefined license

4. Data preparation

4.1. Formats

4.2. Level of detail

4.3. Metadata preparation

5. Publishing data (with CKAN)

5.1. Prerequisites

5.2. Basic installation

5.3. Advanced installation

5.4. Administration

5.5. Advanced features

5.5.1. Compatibility with NTI

5.6. Uploading data

6. Evaluation of use

7. Dialogue with the community

8. Maintenance

9. Evolution

10. References

1. INTRODUCTION

The objective of open data is apparently simple: the publication of such data so that they can be reused and everyone can benefit from them. This guide is intended for all those who wish to start sharing their data but have limited resources to do so. The aim of the guide is to aid and guidance so that anyone can easily open their data, even if they do not have much knowledge about strategies for opening information or the technologies associated with them1 The guide has been designed to be useful even when you do not have technical knowledge, however it will be much easier to follow if you already have some basic knowledge. .

In the first part, a series of simple guidelines are established that will serve as a guide for locating and preparing data for publication. Next, we will indicate how to publish the data on the Web as simply as possible using a catalogue. For this, we will use CKAN as an example, which is a free open source data publishing platform. Regardless, the general recommendations set out in this guide would also apply to any other platform that you may wish to use.

2. SELECTION OF THE DATA TO BE PUBLISHED

The ultimate goal of any open data initiative will always be to publish all available data. However, the more limited the time and resources available, the more important it will be to determine what is the most valuable information from the point of view of reuse, in order to give higher priority to the publication of that data.

2.1. Criteria

The prioritization should be adapted to the particularities of each case. Below are some criteria that can serve as a general guide.

In this way, information managers will be the best connoisseurs of the data they handle and their potential audience.

2.2. Topics

Regarding topics in the highest demand, there are several reference criteria promoted by official organisms of relevance in the matter.

The reference criteria promoted by official organisms include regulations, data for society, the economy, news, quantity of data and level of detail.

In a first general analysis, those belonging to the following areas of information could be suggested as topics of high interest.

In any case, each publisher must also complete and adapt their own list of criteria and interests according to their activity.

2.3. Restrictions

Although the general rule is simply the more that is published the better, there are also limitations that must be respected when publishing data.

In any of these cases, restricted information should be filtered and eliminated prior to being published.

3. LICENSES TO FACILITATE REUSE

It is essential that any published data include explicit information about the conditions of use of that information to make it clear to potential users what the possibilities are of reusing that data. To do this simply, we have two alternative options.

3.1. Establishing simple conditions of use

In this case, we will indicate that the data is fully available for reuse under a series of minimum conditions, for which we can be guided, for example, by those indicated in Royal Decree 1495/2011, of October 24.

3.2. Adopting a predefined license

In this other case, we would simply indicate that the data is available for reuse under the conditions set out in any of the licenses compatible with the principles of open data.

These licenses may, at most, establish restrictions related to the attribution of the original source and the obligation to share any work carried out with the data under the same conditions.

We will be able to choose from the following options:

Public domain licenses that allow us to share the data without any type of restriction:

Creative Commons Zero (CC Zero).

Public Domain Dedication and License (PDDL 1.0).

Licenses that only require citingthe original source of the data:

Creative Commons-Recognition (CC BY 4.0).

Open Data Commons Attribution License 1.0 (ODC-By 1.0).

Licenses that require citing the original source and sharing the results under the same conditions:

Creative Commons-Recognition Share Equal (CC BY-SA 4.0).

Open Data Commons Open Database License (ODbL 1.0).

It is also important to point out that, even if you do not want limit the reuse of the data, you have to explicitly indicate it in the terms of use or public domain license, since otherwise we would have two problems.

Potential reusers would not know what the applicable conditions are in practice due to not having any indications about them.

Some default rights (for example, related to rights of ownership or ownership of databases) could continue to apply if they are not explicitly renounced through the declaration of public domain or other conditions.

4. DATA PREPARATION

One of the main requirements for reuse is that automatic data processing be possible, because this is the only way to efficiently work with large amounts of data. To do this, it will be necessary for the information to be published in the suitable formats and with the sufficient level of quality and detail so that the data are usable.

4.1. Formats

Some criteria to follow when selecting the appropriate formats are:

Formats structured that follow standardized data models.

Formats directly readable by machines.

The formats most demanded among our potential reusers.

Open formats that allow use without restrictions to expand the potential audience.

Formats that allow publishing the incorporated metainformation.

Formats compatible with what is established in the Interoperability Technical Standard of the Catalog of Standards.

Below we can see some examples of formats frequently used for the opening of information.

Also, and whenever possible, it is also advisable to simultaneously offer the data in several different formats to try to respond to the needs of the greatest number of reusers possible.

4.2. Level of detail

It is also important to respect the original level of detail of the data, as it is generated in their source, avoiding any modification and alteration prior to their publication so that each reuser can then carry out the information processing that he/she deems fit.

The only exception to this rule will be the filtering that may be necessary due to the previously mentioned issues of Privacy, Confidentiality or security.

4.3. Metadata preparation

The last step prior to data publication will be to prepare the metadata that will accompany them providing additional information to facilitate their location, classification and reuse. It is very important not only to provide as much metadata as possible, but also to do it in the correct way.

On the other hand, metadata will sometimes serve to describe the data sets and others to describe the different resources (or files) that compose them. Below is a table summarizing the metadata we can provide.

For a more detailed description of each of these metadata we can consult Annex III of the NTI of Reuse of information resources and its application guide.

5. PUBLISHING DATA (WITH CKAN)

To optimally manage data for its reuse, it is necessary to establish a data collection in a common access point, which we generally call a data catalogue, which will allow for more efficient organization and easier access by potential reusers.

CKAN is a tool designed to facilitate the publication of data catalogues2 A demo version of CKAN is available at http://demo.ckan.org/es/. . It could be compared with a traditional content management system (CMS), but aimed at publishing data instead of other content. It will allow us to publish and manage collections of data with very few resources and, once published, users will be able to explore them and even preview them through maps, graphs and tables.

5.1. Prerequisites

CKAN can be installed on any normal-capacity machine. As an example of minimum requirements for a basic installation, we could define a machine with a dual core, 4 GB of RAM and 60 GB of available disk space.

5.2. Basic installation

If our technical knowledge is limited, the quickest and easiest way to start working with our data catalogue can be to use a pre-installed version of CKAN in a cloud service such as Amazon Web Services (AWS) with just a few steps. This option is undoubtedly the simplest, but we must also take into account that would entail a cost per use of AWS services of at least $70 per month3 For a custom calculation use http://calculator.s3.amazonaws.com/index.html. (for a use of 24 hours a day/ 30 days a month and the basic configuration that we are going to mention).

For this, once we have created our user account in the AWS4 To create the account it will be necessary to provide a credit card number. , we will access the pre-installed version, we will choose the region where we want to open the application (EU West-Ireland for hosting the data in Europe) [Step 1] and we will select the "Continue" option [Step 2].

Next, we will choose the type of server (m1.medium should be enough to start) [Step 3] and we will have to generate a new key5 In the case that we are already AWS users and have a previously generated key, we can simply select it instead of creating a new one. (key pair) in order to access the server later. We will then use the "Visit the Amazon EC2 Console" option [Step 4] that will open the Management Console in a new window.

Once in the console, we will use the "Create Key Pair" option [Step 5], enter a name for the key [Step 6] and generate it with the "Create" option [Step 7].

A file will be generated with the key name that we have defined and the extension .pem that we must save and keep in a secure location.

We will then return to the previous page and reload it to verify that the key is already assigned [Step 8] and we can finally start our new CKAN using the "Launch with 1-Click" option [Step 9].

Immediately a message will appear indicating that the CKAN instance that we just created is being initialized and in a few minutes, it will be ready to be used.

We can check its status at any time by accessing the control panel [Step 10].

In the control panel, we can see when CKAN is running (the status will become "running") [Step 11] and verify that it works correctly by accessing the public IP address assigned to it on a web browser [Step 12].

If everything has gone correctly, we will see the following in our browser after entering the IP address assigned in the address bar.

We could now directly register ourselves as users and start using CKAN to publish data.

5.3. Advanced installation

If we have broader technical knowledge and want to opt for a more personalized installation, the most comfortable option of those available would be to use the packaged version for 64-bit Ubuntu 12.04 servers. Here we would have two alternatives regarding the hardware to use:

Again we could choose to use a cloud service such as AWS, although this time we would simply start with a pre-installation of Ubuntu and we would need to create a security group with 0.0.0.0 permissions for ports 22, 80 and 5000. We would also have the added advantage of being able to select more affordable servers to perform our tests (m1.small for about 35 dollars per month6 |||UNTRANSLATED_CONTENT_START|||Para un cálculo personalizado utilizar http://calculator.s3.amazonaws.com/index.html|||UNTRANSLATED_CONTENT_END||| ) and, in addition, we could also choose to use a free 1-year trial account that is available for the most basic servers7 Basic servers have very limited features and are not likely to be useful for anything other than testing. (t1.micro).

The other possible alternative in this case would be to perform the installation on a server where we could install or already have the corresponding version of Ubuntu installed.

In any case, once the hardware needs are solved, we would proceed to install CKAN in four easy steps from the command line of our already installed Ubuntu.

In addition, we must ensure that both the PostgreSQL and Solr databases are correctly installed and configured.

If all the steps have been correctly carried out, we can now access our CKAN through http://localhost or the DNS that we have assigned to it.

5.4. Administration

Once we have CKAN installed and running, the next step will be to create an administrator user to be able to perform the basic configuration of our catalog. If we are using AWS, the first thing will be to connect to our remote machine following the steps below from the command line.

After, whether we are using AWS or one of our own servers, we will follow the steps below again using the Ubuntu command line.

With this we can access our CKAN as administrators and change the basic configuration to adapt it to our preferences.

For this we will login using the "Login" option from the top menu and use the option with the "hammer" icon (also at the top) to access the administration menu where we can change aspects such as the name of the site, the presentation style, the motto and the presentation theme using the "Settings" option. Now we have our own catalog prepared and ready to start sharing data.

5.5. Advanced features

For more advanced personalization of the appearance of the catalog we will have to resort to the CKAN personalization guides ' recommendations and to know all the configuration possibilities we can access the advanced administration guide.

In addition, CKAN has a large catalog of extensions that will allow us to considerably expand its functionality in a clean and simple way. Even the most advanced users will be able to create their own extensions to add new functionality.

We also have a complete programming interface (API) that will facilitate the creation of applications and services from the published data.

5.5.1. Advanced features

At this point, it is also important to pay particular attention to the Interoperability Technical Standard for the Reuse of Information Resources, since it is the national reference standard when it comes to guaranteeing the interoperability and ease of use of the data we publish for reuse.

There is the drawback that CKAN has several compatibility problems with NTI8 This comparatison includes an analysis of the level of implementation of the NTI in version 2.2a of CKAN in which it is clear that the compatibility is low. , so it would not be possible to fully comply with this standard with only a default CKAN installation such as the one we have carried out in this guide. To try to alleviate these problems as much as possible and with minimal effort we could make use of the improvements that others have already previously applied, such as:

Installing the CKAN + DCAT extension which considerably improves the DCAT support on the platform (W3C metadata standard to facilitate the interoperability of catalogs on which the NTI is based).

Reuse the code of some of the extensions and improvements made by other previous open data initiatives in our country to improve compatibility with the NTI and facilitate the federation of the data of our platform with the datos.gob.es national data catalog:

Cáceres Open Data Extension for federation with datos.gob.es

Extension of the open data portal of the City of Málaga.

Customization of Aragón Open Data.

Customizing the forms of metadata that are used to enter the information on data sets and thus be able to adapt them to the standards' needs.

5.6. Uploading data

Now that we have everything ready, all we have to do is start adding data to our catalog. The first thing we must do is log in with our username, select the "Data Sets" option from the top menu and then the "Add data set" option.

Next, we can start creating our first data set to share by adding the information corresponding to the "Title"," "Description," "Labels" and "License" metadata.

Once we have finished, we will access the next step using the "Next: Add Data" option in the lower right corner.

Now we can choose between directly uploading a file with the data using the "Upload" option or adding a "Link" directly to the data if they are already available in a web address.

If we choose, for example, to add a "Link," we simply have to include the web address of the site where the data is hosted.

Finally, we would fill out the "Name" and the "Description," as well as the extension of the format in which the data is stored (CSV, JSON, XML).

Once we have finished, we can choose between saving this data and adding more data to this same set using the "Add" option, or finishing adding data and moving on to provide something more using the "Additional information" option.

When we have finished adding all the data, we just have to add some additional information, such as the original "Source" of the data, the "Version" the "Author" or the "email" of the author's contact, among other things.

We can also add other "Personalized Fields" that we can take advantage of for including other metadata that we need (for example, because it is indicated by the regulations) and not included among the previous ones. In this case we must add both the metadata name (Key) and its value.

We will repeat this same process as many times as necessary until we have entered all the desired data sets.

From this moment, the data will also be available through the catalogue for anyone who wants to use them and experiment with the different classification and visualization options available on the platform.

To learn about more possibilities regarding the management of data sets or organizations, you can consult CKAN’s User Guide .

6. EVALUATION OF USE

Once we have published our data, it is also interesting to have certain measures of their use in order to know which could be the most interesting from the point of view of those who are going to reuse them. This will also help us to better define our selection and prioritization criteria in the data publication. The CKAN itself provides us with some tools that may also be useful for this purpose, such as:

The extension for the catalog's statistics , accessible by adding /stats to our catalog's address, will allow us to find out information including:

The number of published data sets.

The number of weekly updates.

The most highly valued data sets.

The data sets with the most updates.

The Google Analytics extension for CKAN, which will help us integrate both systems and have different indicators such as number of visits, downloads, access to the API or different data analytics, among others.

The visitor monitor for pages that, once activated, will help us:

Sort the data sets by popularity.

Highlight data sets and popular resources.

Show visit counters for data sets and resources.

Export the data for analysis.

7. DIALOGUE WITH THE COMMUNITY

In addition to the exclusively quantitative evaluation indicators that we have just seen, we must not forget that in order to be able to understand in depth the demand for data, we should also establish a continuous dialogue with the community of data reusers.

Again, CKAN offers us some interesting tools to help us with dialogue and communication, among which we can highlight:

The functions it provides for sharing data sets in the most popular social networks (Google+, Twitter and Facebook).

The ability to comment on and discuss data sets thanks to extensions like disqus9 Through the external service provided by https://disqus.com/ or comments.

The option to share ideas, applications or any other content related to the data published.

However, it is also important to remember that in order to carry out the proposed objectives, it will be necessary to combine these on-line tools with other offline performances necessary to facilitate continuous communication and collaboration, as well as to know their needs first hand and to get useful feedback through others qualitative indicators for example, questionnaires, interviews and work groups.

8. MAINTENANCE

The initial loading of the data is only the first step in the open publishing the information. The incorporation of new data sets should be a constant from that moment, and special attention should be given to the frequent updating of data sets that have already been incorporated into the platform. For this, we will take into account that we have two types of data sets:

Static: When the data is available through files in any format. This option will require someone to maintain it by updating the data set manually by uploading a new file periodically with the frequency that has been established in each case.

Dynamic: When you can establish a direct connection with the original source from where the data is obtained through a web address (URL) where they have been published. In this case, each time the data is updated, it will be directly reflected in the cataloging platform without the need for any intervention.

9. EVOLUTION

Finally, it should be clear that the purpose of this guide is mainly to help all those who have limited resources and limited knowledge to take their first steps and experiment with the publication of open data. The indications collected here will serve to be able to experiment easily, rapidly and with hardly any need of investment.

However, if you want to bet on a continuous data opening strategy in the medium and long term, it will be necessary to think about carrying out a more complex project that pays special attention to the following points:

10. REFERENCES

-Aragón Open Data, Catalog of open data.

http://opendata.aragon.es/public/documentos/Guia_Gestion_v1.0.pdf

-EU, Measuring European Public Sector Information Resources.

http://ec.europa.eu/information_society/newsroom/cf/document.cfm?doc_id=1198

-Datos.gob.es, Application Guide of Royal Decree 1495/2011.

http://datos.gob.es/sites/default/files/PLANCISP-GRD-07.3.41.pdf

-Datos.gob.es, Guide on cataloging of data sets.

http://datos.gob.es/sites/default/files/sites/default/files/Guia_catalogacion_datosgobes_v3.pdf

-ONTSI, Characterization study of the infomediary sector in Spain.

http://www.ontsi.red.es/ontsi/es/estudios-informes/estudio-de-caracterizaci%C3%B3n-del-sector-infomediario-en-espa%C3%B1-edici%C3%B3n-2012

-UK Cabinet Office, G8 Open Data Charter and Technical Annex.

https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex

-EC, Guidelines on recommended standard licenses, data sets and charging.

http://ec.europa.eu/information_society/newsroom/cf/dae/document.cfm?action=display&doc_id=6421

-Open Data Barometer, OGD Implementation: data availability.

http://opendatabarometer.org/report/analysis/implementation.html

-Datos.gob.es, Taxonomy of the Catalog.

http://datos.gob.es/datos/sites/default/files/files/12_tax_02.pdf

-BOE, Royal Decree 1495/2011, by which Law 37/2007 on the reuse of public sector information is developed.

http://www.boe.es/diario_boe/txt.php?id=BOE-A-2011-17560

-Open Definition, Conformance licenses.

http://opendefinition.org/licenses/

-Creative Commons, About the Licenses.

http://creativecommons.org/licenses/

-LAPSI, License interoperability report.

http://lapsi-project.eu/sites/lapsi-project.eu/files/D5_1__Licence_interoperability_Report_final.pdf

-LAPSI, Licensing guidelines.

http://www.lapsi-project.eu/sites/lapsi-project.eu/files/D5.2LicensingGuidelinesPO%20(1).pdf

-Datos.gob.es, Regulations.

http://datos.gob.es/saber-mas?q=taxonomy/term/28

-Datos.gob.es, Good practices of formats for the publication of data sets.

http://datos.gob.es/sites/default/files/files/SoporteRISP-Buenas%20pr%C3%A1cticas%20formatos%20%20de%20publicaci%C3%B3n.docx

-Datos.gob.es, Methodological guide for sectorial open data initiatives.

http://datos.gob.es/content/saber-mas/guia-metodologica-iniciativas-de-datos-abiertos-sectoriales

-Datos.gob.es, Trends and Open Data Initiatives Report.

http://datos.gob.es/content/informe-de-tendencias-iniciativas-de-datos-abiertos

-Amazon, Amazon Web Services.

http://aws.amazon.com/

-CKAN, User guide.

http://docs.ckan.org/en/latest/user-guide.html

-BOE, Technical Standard of Interoperability of Reuse of information resources.

http://www.boe.es/boe/dias/2013/03/04/pdfs/BOE-A-2013-2380.pdf

-Datos.gob.es, Application Guide of the NTI of Reuse of information resources.

http://datos.gob.es/content/guia-de-aplicacion-de-norma-tecnica-de-interoperabilidad-de-reutilizacion-de-recursos-de

-Aragón Open Data, Report of conformity with the NTI of Reuse of information resources.

http://opendata.aragon.es/public/documentos/Informe_NTI_Aragon_OpenData_v31-01-14.pdf

-Wolters Kluwer, Special Monograph on Open Government and Open Data

https://funkziuni.files.wordpress.com/2014/07/aaj_7-8_2014_br.pdf