Soil observation encodings

Authors: Kathi Schleidt, Max Vercruyssen, Paul van Genuchten

Observation data from field campaigns or analysis on soil samples from laboratories is typically captured according to the Observations, Measurements and Samples (OMS) principles. Various encoding conventions exist to store or exchange such observation data. In this effort we explore a number of these conventions. Including sample code, data and tools.

Background

The Observations Measurements and Samples (previously observations & measurements) working group of the Open Geospatial Consortium has a long history of interoperability of (sensor) observation data. Over time the group has prepared various editions of the OMS UML model. A model to exchange interoperable observation data.

Adopting one of the models for your (research) data has three benefits.

The models assist in identifying what aspects are typically captured on an observation: which property is observed, how can you reference the feature of interest, which unit of measure is used, which procedure is used, when and by who has the observation been made. (For some soil properties the selected procedure effects the result or uncertainty substantially).
When you encode the above information in standardised ways, other users (humans and machines) can easily locate and understand the information
Various software tools are available which support workflows on standardised observation data, such as conversion tools, validation tools, visualisation tools. So you don’t need to write custom software or data models.

Differences between INSPIRE Soil, ISO 28258 and GloSIS data models

The INSPIRE Soil, ISO 28258 and GloSIS data models are each based on the OMS core datamodel, and thus largely interoperable. But there are also notable differences between the models.

The main difference between the INSPIRE and ISO 28258 models pertains to the spatial features defined to represent aspects of soil investigations. While there is clear consensus on the core FeatureTypes required, specifically Site, Plot, Profile and ProfileElements (Layers and Horizons), the 2 models then diverge on various aspects:

Specimen: only provided under ISO 28258
Derived Types: DerivedSoilProfile, SoilDerivedObject, SoilBody only available under INSPIRE
Soil Mapping: SoilMap with the associated classes SoilMappingUnitCategory, SoilTypologicalUnit and SoilMappingUnit are only provided under ISO 28258
Projects and related Analysis Requests are only provided under ISO 28258

Soil observation data encodings

We explore some of the commonly used encodings and how to traverse between them. The encodings explored are:

Annotated tabular data
RO-Crate
OMS as GML
SensorThings API
Relational databases
Semantic web

Annotated tabular data

In the soil science domain it is quite common to share soil observation data in a tabular format (Excel, CSV, DBF). Where samples are listed as rows and observed properties as columns. Column contents are further explained in a readme file, data dictionary or report. Various initiatives exist to standardise the syntax of these readme documents, so also machines can parse this information.

Some existing initiatives for guidance on data dictionaries:

We have not identified a single approach which can describe an observation in its full richness, as well as derive this information from a set of csv’s in a user friendly way. For that reason we propose a dedicated approach to annotate observation data, using a structured data dictionary.

OMS-tab

This novel approach has been drafted to capture data model specific metadata on a dataset consisting of one or more csv files containing (soil) observation data. It structures current data dictionary efforts, to make those efforts more reusable.

Excel templates

We’ve used excel templates to design the approach. We’ve identified typical layouts of observation datasets, and for each of them created an Excel template to identify the relevant metadata elements.

The Simple CSV approach is an interesting starting point to understand the OMS conventions. The template illustrates the aspects involved in capturing data about an observation on a soil profile or measurement on a prepared soil sample.

CSV on the Web

CSV on the Web (CSV-W) used to be a working group at W3C, which has drafted an approach for annotating CSV data on the Web. The initiative provides a valuable framework of documentation and tooling to enrich tabular data on the web.

Our research on CSV-W indicated that it is possible to use the technology to annotate tabular data in such a way, that tooling is able to render rich semantic data from tabular data. The approach is valid for relatively simple data structures such as DCAT records in a catalogue, SKOS concepts in a vocabulary. However observation data is quite complex in nature, and pushes the CSV-W standard to its limits, requiring expert skills for setting up the CSV-W metadata.

R2RML

R2RML is a W3C standard language designed to map relational database content into RDF (Resource Description Framework) datasets. It allows users to define customized mappings that transform relational data into RDF, enabling integration with the Semantic Web. The mappings are expressed as RDF graphs using Turtle syntax.

R2RML is powerfull in transforming observation csv data to rdf. However it is not a technology easily adopted by soil scientists. Our idea is to write tooling which is able to transform OMS-tab to RML, and subsequently use RML tooling to transform the csv data to RDF

RO-Crate

RO-Crate (Research Object Crate) is a lightweight, machine-readable packaging standard for research data, metadata, workflows, and related digital objects. It builds on web standards such as JSON-LD and schema.org to create FAIR (Findable, Accessible, Interoperable, Reusable) research assets. An RO-Crate bundles data files together with rich metadata describing their contents, provenance, creators, and relationships. The approach is domain-agnostic and has been adopted across multiple scientific disciplines. By using linked data principles, RO-Crate enables both human and machine interpretation of research outputs. Community-developed profiles extend the core specification for particular research domains and use cases.

We’ve explored conventions within the RO-Crate community to enable semantics in data dictionaries. RO-Crate does offer an extra layer of data discovery in the FAIR ecosystem, and can be combined with OMS-tab.

OMS as GML

Traditionally data following the UML based models are exchanged via a GML/XML encoding. The OGC Web Feature Service (WFS) protocol is typically used to exchange such GML documents.

On the web various datasets are available in this encoding. For example via the EU data portal.

The Hale Desktop software is an interesting utility to create or consume these GML documents.

Over the years challenges have been identified with this encoding, such as complexity of the GML syntax, large file sizes, limited support in common software tools. Newer encodings are explored below to address these challenges.

Sensor Things API (STA)

The SensorThings API provides a modern mechanism to interact with observation data (advanced select and filter options). STA builds on ODATA protocol and REST principles. An efficient implementation of the API is provided by the Frost Server software. Recent version of QGIS include a STA interface to visualise observation data from STA services.

Beyond the API definition, STA has introduced some conventions on top of OMS, which could be interesting to explore in the scope of using STA model in a file format.

Relational databases

Over time various groups have worked on initiatives to encode observation data following the OMS UML model in a relational database such as SQLite (GeoPackage), MS Access or PostGreSQL.

Based on the INSPIRE good practice for geopackage encoding CREA, in the scope of the EJP Soil project, explored options to encode INSPIRE soil data in a GeoPackage format. GeoPackage is a spatial extension to the common SQLite database format. In the SoilWise project the work is further extended and tested. The SQLite format is focussed on data exchange.

At ISRIC - World Soil Information, a relational database model for soil data based on ISO28258:2013 has been developed. This model is targetting the PostGreSQL database. The model is optimised for use in an operational multi user Soil Information System.

Semantic web

In the project Sino-EU Soil Observatory for intelligent Land Use Management (SIEUSOIL) the group developed a Semantic Web Ontology based on ISO28258, GLOSIS-LD, using existing ontologies, such as SSN/SOSA. This work extends on previous work of the GLOSIS working group of FAO/GSP. The GLOSIS-LD initiative provides mechanisms to encode soil observation data in RDF. The ontology is quite rich in listing Observable properties and observation procedures. These Codelists are also used outside the semantic web context.

These days the Schema.org ontology provides options to capture various aspects of observation data via its Observation class and variableMeasured property.

Other relevant ontologies in this domain are iMash, Sweet, iAdopt and ISO11074.

Alternative models not (yet) explored in this effort

Some alternative/complemetary models used to describe observation data in our domains, but not further explored here, are:

The ISA (Investigation, Study, Assay) framework is a platform designed for managing experimental metadata in life sciences, environmental, and biomedical research.
The Observation class in schema.org aims to capture observation data, including unit and measurement method
GBIF/EML aims to collect information about biological organisms observed in a specific area at a given time
MIAPPE (Minimum Information About Plant Phenotyping Experiments) is a data standard designed to harmonize data from plant phenotyping experiments