outputbundle package

Module contents

Read/write arkimaps output bundles.

An output bundle is a zip or tar file with the output products of an arkimaps run, and their associated metadata.

The main entry points for reading an output bundle are ZipReader and TarReader.

Example program to show all products in a zip bundle:

#!/usr/bin/python3

import json
import sys

from arkimapslib.outputbundle import ZipReader


reader = ZipReader(sys.argv[1])
products = reader.products()
for path, info in products.by_path.items():
    georef = info.georef
    recipe_info = products.by_recipe[info.recipe]
    legend_info = recipe_info.legend_info
    print(json.dumps({"path": path, "georef": georef, "legend": legend_info}, indent=1))

class arkimapslib.outputbundle.InputProcessingStats

Statistics collected while processing inputs

add_computation_log(elapsed: int, what: str) → None

Add an entry to the computation log.

Parameters:

• elapsed – elapsed time in nanoseconds

• what – description of the computation

computation_log: List[Tuple[int, str]]

List of strings describing computation steps, and the time they took in nanoseconds

classmethod from_jsonable(data: Dict[str, Any])

Recreate an object serialized by to_jsonable

to_jsonable() → Dict[str, Any]

Produce a JSON-serializable summary about this input

used_by: Set[str]

List of recipes that used this input to generate products

class arkimapslib.outputbundle.InputSummary

Summary about inputs useed in processing

add(inp: Input)

classmethod from_jsonable(data: Dict[str, Any])

Recreate an object serialized by to_jsonable

inputs: Dict[str, InputProcessingStats]

Per-input processing information indexed by input name

to_jsonable() → Dict[str, Any]

Return a version of this object that can be serialized to JSON, and deserialized with from_jsonable()

class arkimapslib.outputbundle.Log

A collection of log entries.

Iterate Log.entries for a list of all log entries generated during processing.

append(*, ts: float, level: int, msg: str, name: str)

Add a log entry

entries: List[LogEntry]

List of all log entries

classmethod from_jsonable(data: Dict[str, Any])

Recreate an object serialized by to_jsonable

to_jsonable() → Dict[str, Any]

Return a version of this object that can be serialized to JSON, and deserialized with from_jsonable()

class arkimapslib.outputbundle.LogEntry(ts: float, level: int, msg: str, name: str)

One serializable log entry

level: int

Log level

msg: str

Log message

name: str

Logger name

ts: float

Timestamp in seconds

class arkimapslib.outputbundle.ProductInfo(*, recipe: Optional[str] = None, reftime: Optional[datetime] = None, step: Optional[ModelStep] = None, georef: Optional[Dict[str, Any]] = None)

Information about a generated product image

add_georef(georef: Dict[str, Any])

add_instant(instant: Instant)

add_recipe(recipe: Recipe)

classmethod from_jsonable(data: Dict[str, Any])

Recreate an object serialized by to_jsonable

georef: Dict[str, Any]

Georeferencing information

recipe: Optional[str]

Name of the recipe used for this product

reftime: Optional[datetime]

Reference time

step: Optional[ModelStep]

Step

to_jsonable() → Dict[str, Any]

Return a version of this object that can be serialized to JSON, and deserialized with from_jsonable()

class arkimapslib.outputbundle.Products

Information about the products present in the bundle

add_order(order: Order) → None

Add information from this order to the products summary

by_path: Dict[str, ProductInfo]

Products generated, indexed by their path

by_recipe: Dict[str, RecipeOrders]

Recipes used, indexed by name

flavour: Optional[Dict[str, Any]]

Information about the flavour used

classmethod from_jsonable(data: Dict[str, Any])

Recreate an object serialized by to_jsonable

to_jsonable() → Dict[str, Any]

Return a version of this object that can be serialized to JSON, and deserialized with from_jsonable()

class arkimapslib.outputbundle.Reader

Read functions for output bundles

abstract find() → List[str]

List all paths in the bundle

input_summary() → InputSummary

Return summary of all inputs used during processing

abstract load_artifact(bundle_path: str) → bytes

Load processing artifact.

Return the raw file data

abstract load_product(bundle_path: str) → bytes

Load a product by its path.

Return the raw PNG image data

log() → Log

Return the log generated during processing

products() → Products

Return metadata for all products

abstract version() → str

Return the bundle version information

class arkimapslib.outputbundle.RecipeOrders

Information and statistics for all orders generated from one recipe

add(order: Order)

by_reftime: Dict[datetime, ReftimeOrders]

Summary of all products produced for this recipe at this reference time

classmethod from_jsonable(data: Dict[str, Any])

Recreate an object serialized by to_jsonable

legend_info: Optional[Dict[str, Any]]

Legend information for products produced from this recipe

to_jsonable() → Dict[str, Any]

Return a version of this object that can be serialized to JSON, and deserialized with from_jsonable()

class arkimapslib.outputbundle.ReftimeOrders(*, inputs: Optional[Sequence[str]] = None, steps: Optional[Dict[ModelStep, int]] = None, render_time_ns: int = 0)

Information and statistics for all orders for a given reftime

add(order: Order)

classmethod from_jsonable(data: Dict[str, Any]) → ReftimeOrders

Recreate an object serialized by to_jsonable

inputs: Set[str]

Names of inputs used

render_time_ns: int

Total processing time in nanoseconds

steps: Dict[ModelStep, int]

Numer of products produced for each step

to_jsonable() → Dict[str, Any]

Return a version of this object that can be serialized to JSON, and deserialized with from_jsonable()

class arkimapslib.outputbundle.Serializable

Base for classes that can be serialized to JSON

abstract classmethod from_jsonable(data: Dict[str, Any])

Recreate an object serialized by to_jsonable

abstract to_jsonable() → Dict[str, Any]

Return a version of this object that can be serialized to JSON, and deserialized with from_jsonable()

class arkimapslib.outputbundle.TarReader(path: Path)

Read an output bundle from a tar file

Usage:

with TarReader(path) as reader:
    for name in reader.find():
        if name.endswith(".png"):
            print(name)

See Reader for the full method list

find() → List[str]

List all paths in the bundle

load_artifact(bundle_path: str) → bytes

Load processing artifact.

Return the raw file data

load_product(bundle_path: str) → bytes

Load a product by its path.

Return the raw PNG image data

version() → str

Return the bundle version information

class arkimapslib.outputbundle.TarWriter(out: IO[bytes])

Write an output bundle as a tar file.

add_artifact(bundle_path: str, data: IO[bytes])

Add a processing artifact

add_product(bundle_path: str, data: IO[bytes])

Add a product

class arkimapslib.outputbundle.Writer

Write functions for output bundles

abstract add_artifact(bundle_path: str, data: IO[bytes])

Add a processing artifact

add_input_summary(input_summary: InputSummary)

Add inputs.json with a summary of inputs used

add_log(entries: Log)

Add log.json with log entries generated during processing.

If no log entries were generated, log.json is not added.

abstract add_product(bundle_path: str, data: IO[bytes])

Add a product

add_products(products: Products)

Add products.json with information about generated products

class arkimapslib.outputbundle.ZipReader(path: Path)

Read an output bundle from a zip file

Usage:

with ZipReader(path) as reader:
    for name in reader.find():
        if name.endswith(".png"):
            print(name)

See Reader for the full method list

find() → List[str]

List all paths in the bundle

load_artifact(bundle_path: str) → bytes

Load processing artifact.

Return the raw file data

load_product(bundle_path: str) → bytes

Load a product by its path.

Return the raw PNG image data

version() → str

Return the bundle version information

class arkimapslib.outputbundle.ZipWriter(out: IO[bytes])

Write an output bundle as a zip file.

add_artifact(bundle_path: str, data: IO[bytes])

Add a processing artifact

add_product(bundle_path: str, data: IO[bytes])

Add a product