Metadata types

Origin

Defines the centre or process that originated the data.

GRIB1

Origin represented with as in GRIB version 1:

  • Centre

  • Subcentre

  • Process

Note

TODO: add references to manual of codes

GRIB2

Origin represented with as in GRIB version 2:

  • Centre

  • Subcentre

  • Process type

  • Background process identifier

  • Process identifier

Note

TODO: add references to manual of codes

BUFR

Origin represented with as in BUFR headers:

  • Centre

  • Subcentre

Note

TODO: add references to manual of codes

ODIMH5

Origin represented with as in ODIM:

  • WMO

  • RAD

  • PLC

Note

TODO: find meanings of ODIMh5 values

Product

Defines what kind of variable is measured or forecast or otherwise represented in the data.

GRIB1

Product defined the same as product definition in GRIB version 1:

  • Origin

  • Table number

  • Product

Note

TODO: add references to manual of codes

GRIB2

Product defined the same as in GRIB version 2:

  • Centre

  • Discipline

  • Category

  • Number

  • Table version

  • Local table version

Note

TODO: add references to manual of codes

BUFR

Product defined the same as in BUFR:

  • Type

  • Subtype

  • Local subtype

Note

TODO: add references to manual of codes

Optionally one can add a number of arbitrary key-value pairs to represent information extracted from the BUFR contents that is significant for archival purposes.

ODIMH5

Product defined as in ODIM:

  • Obj

  • Prod

Note

TODO: find meanings of ODIMh5 values

VM2

Product defined as a VM2 variable ID

Level

Defines the vertical level or layer of the data.

GRIB1

Level represented with as in GRIB version 1:

  • Level type

  • l1 (when applicable, depending on type)

  • l2 (for layer types)

Note

TODO: add references to manual of codes

GRIB2S

Level represented with as in GRIB version 2, surface levels:

  • Level type

  • Scale

  • Value

Note

TODO: add references to manual of codes

GRIB2D

Level represented with as in GRIB version 2, layer levels:

  • Type of first level

  • Scale of first level

  • Value of first level

  • Type of second level

  • Scale of second level

  • Value of second level

Note

TODO: add references to manual of codes

ODIMH5

Level represented with as in ODIM, either as a single floating point value, or as two floating point minimum, maximum values.

Note

TODO: find meanings of ODIM values

Timerange

Defines the forecast step or statistical processing information for the data.

GRIB1

Time range defined as in GRIB version 1:

  • Time range type

  • Time range unit

  • P1

  • P2

Note

TODO: add references to manual of codes

GRIB2

Time range defined as in GRIB version 2:

  • Time range type

  • Time range unit

  • P1

  • P2

Note

TODO: add references to manual of codes

BUFR

Time range representing forecast steps for BUFR generated by forecast processing:

  • Value

  • Unit of time (same as in GRIB version 1 and 2)

GRIB2

Semantic time range definition that can be used to represent most (but not all) time ranges in a unified way.

Time range values that cannot be converted to Timedef are corner cases where the data model of Timedef cannot be applied.

Units of time are defined as in GRIB2 time range time units.

For forecasts without statistical processing:

  • step length (can be 0 for analysys)

  • step unit

For forecasts with statistical processing:

  • step length (can be 0 for analysys)

  • step unit

  • type of statistical processing (same as GRIB time range types)

  • length of period of statistical processing

  • unit for length of period of statistical processing

Arkimet supports converting most time range types to Timedef, and it is possible to use Timedef matching to match metadata in other formats.

Reftime

Date and time for the reference time of one data element.

This is the metadata used for organizing data along the time axis.

What is the reference time is defined by the scanner code.

The maximum time resolution is one second.

Times are assumed to be in UTC, and time zones are not represented.

Note

A timestamped annotation about how arkimet processed this element.

This can be used to track and audit the life of data in the archive.

It can be used for consultation only, and cannot be used in searches.

Source

Reference to where the data is stored.

This is what allows to fetch the actual data given its metadata.

Blob

The data is available in the local file system:

  • filename points to the file that has the data

  • offset is the position in the file where the data is stored

  • size is the size in bytes of the data

basedir is a hint that can be used to resolve relative filename values. It is not stored in the metadata, and when reading it defaults to the path where the metadata is found, so that data can be referenced relative to the metadata.

It is possible that filename points to a directory segment or a .zip file: in that case, the value of the offset is used to reference the data in the directory or zipfile based on the meaning given by the directory segment or zip file segment implementation.

Inline

The data follows the metadata in the same data stream.

This stores the size in bytes of the data to be read after the metadata in the stream.

URL

The data is stored at a remote location.

This is a string containing a URL that points at the data remotely.

At the moment, remotely accessing a single data element is not supported, and this field is usually only filled with the URL of the remote dataset that contains the data.

Fetching remote data is usually done transparently and more efficiently at query time, with data returned inline after the metadata.

Area

Geographical area relative to a data element.

When possible, Area values should be convertible to coordinate polygons, allowing matching using geospatial primitives (contains, intersects, …).

GRIB

Collection of key-value pairs, interpreted in the context of GRIB grid definitions.

ODIMH5

Collection of key-value pairs, interpreted in the context of ODIM area information.

VM2

Area information as an integer VM2 station identifier.

Proddef

Product definition, encoding extra information to represent variations on the defined product.

This is used, for example, to encode the ensemble parameters and distinguish products that have been computed using different ensemble conditions.

GRIB

Collection of key-value pairs, interpreted in the context of GRIB conventions.

Run

Time of day when the model was run that generated this data.

This is generally filled when needed from the data reference time.

Note

TODO: [Enrico] I have a vague memory that this was introduced to distinguish metadata that would otherwise be the same, and only change according to the model run time.

I would like to document it with an example of when this is needed, but I cannot find any at the moment. If no example can be found, it may be time to check if this metadata item is still at all needed.

Minute

Model run time of day, in minutes from midnight

Task

Representation of ODIM Task as a string value.

Note

TODO: find meanings of ODIMh5 values

Quantity

Representation of ODIM quantity as a list of one or more string values.

Note

TODO: find meanings of ODIMh5 values

Value

The value of very short data encoded as part of the metadata

This is currently used to encode the non-metadata part of VM2 data so that it can be extracted from metadata or dataset indices and completed using the rest of metadata values, avoiding disk lookips