arkimet documentation

Arkimet is a set of tools to organize, archive and distribute data files. It currently supports data in GRIB, BUFR, HDF5 and VM2 formats.

Arkimet manages a set of datasets (see Datasets), each of which contains omogeneous data stored in segments (see Data segments). It exploits the commonalities between the data in a dataset to implement a fast, and space-efficient indexing system.

When data is ingested into arkimet, it is scanned and annotated with metadata, such as reference time and product information, then it is dispatched to one of the datasets configured in the system. From this point onwards, data are treated like opaque blobs, and only metadata are used for indexing or searching, reducing the need for format-specific operations after ingesting.

Datasets can be queried both locally and over the network using an HTTP-based protocol.

Old data can be archived using an automatic maintenance procedure, and archives can be taken offline and back online at will.

A summary of offline data is kept online, so that arkimet is able to report that more data for a query would be available but is currently offline.

Arkimet is Free Software, licensed under the GNU General Public License version 2 or later.

Contents:

• Metadata types
  • Origin
  • Product
  • Level
  • Timerange
  • Reftime
  • Note
  • Source
  • Area
  • Proddef
  • Run
  • Task
  • Quantity
  • Value
• Match expressions
  • Examples
• Datasets
  • iseg dataset format
  • simple dataset format
  • ondisk2 dataset format
  • dataset archives
  • error dataset
  • duplicates dataset
  • remote dataset
  • outbound dataset
  • discard dataset
  • Dataset configuration
• Data segments
  • concat segments
  • dir segments
  • gz and gzidx segments
  • tar segments
  • zip segments
• arki-server HTTP API
  • Sorting
  • arki-server urls
• Arkimet runtime configuration
  • Environment variables
• Arkimet match-alias.conf database
• Arkimet Python HOWTOs
  • Read the output of arki-query --inline
  • Formatting the contents of metadata
• Indices and tables
• Arkimet Python API
  • arkimet package
  • arkimet.cfg package
  • arkimet.dataset package
  • arkimet.scan package
  • arkimet.formatter package
  • arkimet.bbox package
• Indices and tables

---

• Index

• Module Index

• Search Page