############## Metadata types ############## ****** Origin ****** Defines the centre or process that originated the data. GRIB1 ===== Origin represented with as in GRIB version 1: * Centre * Subcentre * Process .. note:: TODO: add references to manual of codes GRIB2 ===== Origin represented with as in GRIB version 2: * Centre * Subcentre * Process type * Background process identifier * Process identifier .. note:: TODO: add references to manual of codes BUFR ==== Origin represented with as in BUFR headers: * Centre * Subcentre .. note:: TODO: add references to manual of codes ODIMH5 ====== Origin represented with as in ODIM: * WMO * RAD * PLC .. note:: TODO: find meanings of ODIMh5 values ******* Product ******* Defines what kind of variable is measured or forecast or otherwise represented in the data. GRIB1 ===== Product defined the same as product definition in GRIB version 1: * Origin * Table number * Product .. note:: TODO: add references to manual of codes GRIB2 ===== Product defined the same as in GRIB version 2: * Centre * Discipline * Category * Number * Table version * Local table version .. note:: TODO: add references to manual of codes BUFR ==== Product defined the same as in BUFR: * Type * Subtype * Local subtype .. note:: TODO: add references to manual of codes Optionally one can add a number of arbitrary key-value pairs to represent information extracted from the BUFR contents that is significant for archival purposes. ODIMH5 ====== Product defined as in ODIM: * Obj * Prod .. note:: TODO: find meanings of ODIMh5 values VM2 === Product defined as a VM2 variable ID ***** Level ***** Defines the vertical level or layer of the data. GRIB1 ===== Level represented with as in GRIB version 1: * Level type * l1 (when applicable, depending on type) * l2 (for layer types) .. note:: TODO: add references to manual of codes GRIB2S ====== Level represented with as in GRIB version 2, surface levels: * Level type * Scale * Value .. note:: TODO: add references to manual of codes GRIB2D ====== Level represented with as in GRIB version 2, layer levels: * Type of first level * Scale of first level * Value of first level * Type of second level * Scale of second level * Value of second level .. note:: TODO: add references to manual of codes ODIMH5 ====== Level represented with as in ODIM, either as a single floating point value, or as two floating point minimum, maximum values. .. note:: TODO: find meanings of ODIM values ********* Timerange ********* Defines the forecast step or statistical processing information for the data. GRIB1 ===== Time range defined as in GRIB version 1: * Time range type * Time range unit * P1 * P2 .. note:: TODO: add references to manual of codes GRIB2 ===== Time range defined as in GRIB version 2: * Time range type * Time range unit * P1 * P2 .. note:: TODO: add references to manual of codes BUFR ==== Time range representing forecast steps for BUFR generated by forecast processing: * Value * Unit of time (same as in GRIB version 1 and 2) GRIB2 ===== Semantic time range definition that can be used to represent most (but not all) time ranges in a unified way. Time range values that cannot be converted to Timedef are corner cases where the data model of Timedef cannot be applied. Units of time are defined as in GRIB2 time range time units. For forecasts without statistical processing: * step length (can be 0 for analysys) * step unit For forecasts with statistical processing: * step length (can be 0 for analysys) * step unit * type of statistical processing (same as GRIB time range types) * length of period of statistical processing * unit for length of period of statistical processing Arkimet supports converting most time range types to Timedef, and it is possible to use Timedef matching to match metadata in other formats. ******* Reftime ******* Date and time for the reference time of one data element. This is the metadata used for organizing data along the time axis. What is the reference time is defined by the scanner code. The maximum time resolution is one second. Times are assumed to be in UTC, and time zones are not represented. **** Note **** A timestamped annotation about how arkimet processed this element. This can be used to track and audit the life of data in the archive. It can be used for consultation only, and cannot be used in searches. ****** Source ****** Reference to where the data is stored. This is what allows to fetch the actual data given its metadata. Blob ==== The data is available in the local file system: * ``filename`` points to the file that has the data * ``offset`` is the position in the file where the data is stored * ``size`` is the size in bytes of the data ``basedir`` is a hint that can be used to resolve relative ``filename`` values. It is not stored in the metadata, and when reading it defaults to the path where the metadata is found, so that data can be referenced relative to the metadata. It is possible that ``filename`` points to a directory segment or a ``.zip`` file: in that case, the value of the offset is used to reference the data in the directory or zipfile based on the meaning given by the directory segment or zip file segment implementation. Inline ====== The data follows the metadata in the same data stream. This stores the size in bytes of the data to be read after the metadata in the stream. URL === The data is stored at a remote location. This is a string containing a URL that points at the data remotely. At the moment, remotely accessing a single data element is not supported, and this field is usually only filled with the URL of the remote dataset that contains the data. Fetching remote data is usually done transparently and more efficiently at query time, with data returned inline after the metadata. **** Area **** Geographical area relative to a data element. When possible, Area values should be convertible to coordinate polygons, allowing matching using geospatial primitives (contains, intersects, ...). GRIB ==== Collection of key-value pairs, interpreted in the context of GRIB grid definitions. ODIMH5 ====== Collection of key-value pairs, interpreted in the context of ODIM area information. VM2 === Area information as an integer VM2 station identifier. ******* Proddef ******* Product definition, encoding extra information to represent variations on the defined product. This is used, for example, to encode the ensemble parameters and distinguish products that have been computed using different ensemble conditions. GRIB ==== Collection of key-value pairs, interpreted in the context of GRIB conventions. *** Run *** Time of day when the model was run that generated this data. This is generally filled when needed from the data reference time. .. note:: TODO: [Enrico] I have a vague memory that this was introduced to distinguish metadata that would otherwise be the same, and only change according to the model run time. I would like to document it with an example of when this is needed, but I cannot find any at the moment. If no example can be found, it may be time to check if this metadata item is still at all needed. Minute ====== Model run time of day, in minutes from midnight **** Task **** Representation of ODIM Task as a string value. .. note:: TODO: find meanings of ODIMh5 values ******** Quantity ******** Representation of ODIM quantity as a list of one or more string values. .. note:: TODO: find meanings of ODIMh5 values ***** Value ***** The value of very short data encoded as part of the metadata This is currently used to encode the non-metadata part of VM2 data so that it can be extracted from metadata or dataset indices and completed using the rest of metadata values, avoiding disk lookips