arkimet package

Subpackages

Module contents

Python modules to work with Arkimet archives.

class arkimet.BBox

BBox for arkimet metadata.

compute(type: Union[Dict[str, Any], str]) → str

compute the bounding box for the given area

class arkimet.Formatter

Formatter for arkimet metadata.

format(type: Dict[str, Any]) → str

format the given type to a human understandable string

class arkimet.Matcher

Precompiled matcher for arkimet metadata

expanded

return the query with all aliases expanded

match(md: arki.Metadata) → bool

return the result of trying to match the given metadata

merge(matcher: arki.Matcher) → arki.Matcher

return a matcher that selects at least the data from this and the other matcher

update(matcher: arki.Matcher) → arki.Matcher

return this matcher plus all the rules of the other matcher. When they both have a rule for the same metadata, only the rule of the second matcher is kept

class arkimet.Metadata

Metadata for one data item.

Each single element stored in Arkimet has a number of metadata associated. This class stores all the metadata for one single element.

The constructor takes no arguments and returns an empty Metadata object, which can be populated with item assigment.

Item assigned can take metadata items both in string format or as Python dicts.

For example:

md = arkimet.Metadata()
md["origin"] = 'GRIB1(098, 000, 129)'
md["reftime"] = {"style": "POSITION",  "time": datetime.datetime(2007, 7, 8, 13, 0, 0)}
with open("test.json", "wb") as fd:
    md.write(fd, format="json")
data

get the raw data described by this metadata

data_size

return the size of the data, if known, else returns 0

del_notes()

remove all notes from this Metadata

get_notes() → List[Dict[str, Any]]

get the notes for this metadata

has_source() → bool

check if a source has been set

make_absolute()

Make path in source blob absolute

make_inline()

Read the data and inline them in the metadata

make_url(baseurl: str) → None

Set the data source as URL

Parameters

baseurl – the base URL that identifies the dataset

print_type_documentation(file: Union[TextIO])

Print documentation about all available metadata types to the given file

read_bundle(src: Union[bytes, ByteIO], dest: Callable[[metadata]=None, Optional[bool]], basedir: str=None, pathname: str=None) → Union[bool, List[arkimet.Metadata]

Read all metadata from a given file or memory buffer

Parameters

  • src – source data or binary input file

  • dest – function called for each metadata decoded. If None, a list of arkimet.Metadata is returned instead

  • basedir – base directory used to resolve relative file names inside the metadata

  • pathname – name of the file to use in error messages, when src.name is not available

read_json(src: Union[str, StringIO, bytes, ByteIO]) → arkimet.Metadata

Read a Metadata from a JSON file

read_yaml(src: Union[str, StringIO, bytes, ByteIO]) → Optional[arkimet.Metadata]

Read a Metadata from a YAML file

Return None in case of end of file

to_python(type: str = None) → dict

Return the metadata contents in a python dict

to_string(type: str = None) → Optional[str]

Return the metadata contents as a string

write(file: Union[int, BytesIO], format: str = 'binary', annotate: bool = False, skip_data: bool = False) → None

Write the metadata to a file

Parameters

  • file – the output file. The file can be a normal file-like object or an integer file or socket handle

  • format – “binary”, “yaml”, or “json”. Default: “binary”.

  • annotate – if True, use a arkimet.Formatter to add metadata descriptions to the output

  • skip_data – if True, do not write data after the metadata even if the source type is INLINE

write_bundle(mds: Iterable[arkimet.Metadata], file: BinaryIO)

Write all metadata to a given file

class arkimet.Summary

Arkimet summary

TODO: document

Examples:

TODO: add examples
add(val: Union[arki.Metadata, arki.Summary])

merge a metadata or summary into this summary

count

Return the number of metadata described by this summary

get_convex_hull() → Optional[str]

compute the convex hull for this summary and return it as a WKT string

None is returned if the convex hull could not be computed.

read_binary(src: Union[bytes, ByteIO]) → arkimet.Summary

Read a Summary from a binary file

read_json(src: Union[str, StringIO, bytes, ByteIO]) → arkimet.Summary

Read a Summary from a JSON file

read_yaml(src: Union[str, StringIO, bytes, ByteIO]) → arkimet.Summary

Read a Summary from a YAML file

size

Return the total size of all the metadata described by this summary

to_python() → Dict[str, Any]

return the summary contents in a python dict

write(file: Union[int, BinaryIO], format: str = 'binary', annotate: bool = False) → Optional[arki.cfg.Section]

write the summary to a file

Parameters

  • file – the output file. The file needs to be either an integer file or socket handle, or a file-like object with a fileno() method that returns an integer handle.

  • format – “binary”, “yaml”, or “json”. Default: “binary”.

  • annotate – if True, use a arkimet.Formatter to add metadata descriptions to the output

write_short(file: Union[int, BinaryIO], format: str = 'binary', annotate: bool = False) → Optional[arki.cfg.Section]

write the short summary to a file

Parameters

  • file – the output file. The file needs to be either an integer file or socket handle, or a file-like object with a fileno() method that returns an integer handle.

  • format – “binary”, “yaml”, or “json”. Default: “binary”.

  • annotate – if True, use a arkimet.Formatter to add metadata descriptions to the output

arkimet.config() → Dict[str, Dict[str, str]]

return the arkimet runtime configuration

arkimet.debug_tty(text: str)

write a debug message to /dev/tty

arkimet.expand_query(query: str) → str

expand aliases in an Arkimet query, returning the same query without use of aliases

arkimet.get_alias_database() → arkimet.cfg.Sections

return a the current matcher alias database

arkimet.get_version() → str

get a string with the current Arkimet version

arkimet.make_merged_dataset(cfg: Union[str, arkimet.cfg.Sections]) → arkimet.DatasetReader

create a merged dataset from all the datasets found in the given configuration

arkimet.make_qmacro_dataset(datasets: Union[str, arkimet.cfg.Sections], name: str, query: str) → arkimet.DatasetReader

create a QueryMacro dataset that aggregates the contents of multiple datasets

Arguments:

cfg: base configuration of the resulting dataset datasets: configuration of all the available datasets name: name of the query macro to use

arkimet.set_verbosity(verbose: bool = False, debug: bool = False)

set the arkimet warning verbosity