Harvesting Metadata

Harvesting Channels

Harvesting is the process of automatically fetching remote metadata. This section describes how B2FIND harvests metadata records from data provider sites. While OAI-PMH, as the de facto standard for metadata harvesting, is preferred, B2FIND also supports other APIs.
Once one of these transfer methods has been successfully implemented, B2FIND first takes up a few test samples to analyse their content, as described in the section Initial Uptake of a New Data Provider.
As soon as the harvesting and mapping has been consolidated and the data provider gives their consent, the metadata are published on the B2FIND discovery portal, and an operational and stable ingestion process is established (see section Synchronous and Operational Ingestion.


OAI-PMH is B2FIND’s preferred metadata harvesting protocol. It can be used to fetch metadata directly from the data providers within research communities. The simplicity of the protocol allows a controlled and easy-to-manage transfer of metadata. Very little information must be provided to enable B2FIND to perform the harvesting process using this protocol:

  • OAI endpoint: This is the URL of the OAI provider server on data provider site, which must be open for OAI-PMH read requests.
  • OAI mdprefix: This is the OAI acronym for the metadata schema in which the provided XML records are coded in.
  • OAI sets (optional): It is recommended to group your records in subsets, because this simplifies the controlled harvesting.


To harvest all Dublin Core (OAI mdprefix is oai_dc) from the subset "ANDS.Centre-1" from DataCite's OAI provider (oai.datacite.org/oai),we submit a HTTP-request with the verb "ListRecords" and the following OAI options set:


Please also check module 02 of the B2FIND training materials, where you will find a step-by-step guide to set up and configure an OAI-server. For a detailed documentation of the OAI-PMH protocol, see http://www.openarchives.org/OAI/openarchivesprotocol.html.


Some data providers offer their metadata encoded as JSON records, which can be retrieved, queried and browsed via a REST API. The API is generally RESTful and returns results in JSON, as the API follows the JSONAPI specification.


The community GBIF (see gbif.org) provides their metadata via the JSON-API at the base URL http://api.gbif.org/v1. By the following request the first 100 JSON records are retrieved from the repository:



Catalog Service for the Web (CSW) is a standard for exposing a catalogue of geospatial records in XML on the Internet (over HTTP). The catalogue is made up of records that describe geospatial data and services. B2FIND uses a CSW implementation to harvest XML records from so-called GeoNetwork portals.


The community Seadatanet (see seadatanet.org) exposes georeferenced metadata via the base GeoNetwork portal with URL endpoint http://sextant.ifremer.fr/geonetwork/srv/fre/csw-SEADATANET. To retrieve the ISO 19139 XML records (namespace specification <gmd:MD_Metadata>), B2FIND submits a "GetRecord" request as follows:


Initial Uptake of a New Data Provider

Once one of the harvesting methods has been deployed successfully and is operational, B2FIND starts with an initial harvesting of a few metadata records. These samples are analysed, metadata elements mapped to correct database indices, and the metadata records are uploaded to a B2FIND test/development server.

When both the harvesting and mapping are at least functional, some of the issues already mentioned in the section on Providing Metadata have to be negotiated with the data provider:

  • Scope and extent: Limit the metadata exposed to B2FIND to those which refer to research data. The best practice would be to gather all records designated to be published in B2FIND in dedicated subsets.
  • Grouping and partitioning: Choose the subsets (e.g. OAI sets) which should be harvested (in some cases whole subsets can be assigned to a ‘Discipline’ or can be grouped to ‘sub-communities’ in the B2FIND portal).
  • Selection, assigning and mapping: Check the quality of the mapping of your specific fields to the B2FIND metadata schema (see section Mapping onto the B2FIND Metadata Schema).

Synchronous and Operational Ingestion

In the long term, not only is it important to have a reliable and sustainable harvesting mechanism established, but also to implement a frequent harvesting schedule. This will guarantee sufficient synchronicity between the provider (community) database and EUDAT service (B2FIND). The OAI-PMH the parameter from can be used to harvest only records which are newly created or changed during a given period. If for instance an update interval of once a week is agreed upon, B2FIND establishes a cron job that is triggered on a weekly basis, with the harvesting option <fromdate> set to a date one week earlier.