outputbundle package¶

Module contents¶

Read/write arkimaps output bundles.

An output bundle is a zip or tar file with the output products of an arkimaps run, and their associated metadata.

The main entry points for reading an output bundle are ZipReader and TarReader.

Example program to show all products in a zip bundle:

#!/usr/bin/python3

import json
import sys

from arkimapslib.outputbundle import ZipReader


reader = ZipReader(sys.argv[1])
products = reader.products()
for path, info in products.by_path.items():
    georef = info.georef
    recipe_info = products.by_recipe[info.recipe]
    legend_info = recipe_info.legend_info
    print(json.dumps({"path": path, "georef": georef, "legend": legend_info}, indent=1))

class arkimapslib.outputbundle.InputProcessingStats(*, computation_log: List[Tuple[int, str]] = None, used_by: Set[str] = None)[source]¶

Statistics collected while processing inputs.

Used in inputs.json.

add_computation_log(elapsed: int, what: str) → None[source]¶

Add an entry to the computation log.

Parameters:

elapsed – elapsed time in nanoseconds
what – description of the computation

computation_log: List[Tuple[int, str]]¶: List of strings describing computation steps, and the time they took in nanoseconds

dict(*args: Any, **kwargs: Any) → Dict[str, Any][source]¶: Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

classmethod fix_layout(values: Any) → Any[source]¶

used_by: Set[str]¶: List of recipes that used this input to generate products

class arkimapslib.outputbundle.InputSummary(*, inputs: Dict[str, InputProcessingStats] = None)[source]¶

Summary about inputs useed in processing.

Used in inputs.json.

add(name: str, stats: InputProcessingStats)[source]¶

dict(*args: Any, **kwargs: Any) → Dict[str, Any][source]¶: Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

classmethod fix_layout(values: Any) → Any[source]¶

inputs: Dict[str, InputProcessingStats]¶: Per-input processing information indexed by input name

class arkimapslib.outputbundle.Log(*, entries: List[LogEntry] = None)[source]¶

A collection of log entries.

Iterate Log.entries for a list of all log entries generated during processing.

Used in log.json.

append(*, ts: float, level: int, msg: str, name: str)[source]¶: Add a log entry

dict(*args: Any, **kwargs: Any) → Any[source]¶: Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

entries: List[LogEntry]¶: List of all log entries

classmethod fix_layout(values: Any) → Any[source]¶

classmethod parse_obj(obj: Any) → Log[source]¶

class arkimapslib.outputbundle.LogEntry(*, ts: float, level: int, msg: str, name: str)[source]¶

One serializable log entry.

Used in log.json.

level: int¶: Log level

msg: str¶: Log message

name: str¶: Logger name

ts: float¶: Timestamp in seconds

class arkimapslib.outputbundle.PathInfo(recipe: str, georef: Dict[str, Any] | None = None)[source]¶

Information about a single generated product image.

georef: Dict[str, Any] | None¶: Georeferencing information

recipe: str¶: Recipe name

class arkimapslib.outputbundle.ProductInfo(*, georef: Dict[str, Any] | None = None)[source]¶

Information about a single product.

georef: Dict[str, Any] | None¶: Georeferencing information

class arkimapslib.outputbundle.ProductKey(flavour: str, recipe: str)[source]¶

Unique identifier for products generated from the same recipe and flavour.

flavour: str¶: Flavour name

recipe: str¶: Recipe name

class arkimapslib.outputbundle.Products(*, products: Dict[ProductKey, RecipeProducts] = None)[source]¶

Information about the products present in the bundle

add_order(order: Order) → None[source]¶: Add information from this order to the products list

property by_path: Dict[str, PathInfo]¶: Return a standard layout of information for each image file in the output.

property by_recipe: Dict[str, RecipeOrders]¶: Return a standard layout of information for each recipe in the output.

dict(*args: Any, **kwargs: Any) → Any[source]¶: Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

classmethod fix_layout(values: Any) → Any[source]¶

classmethod parse_obj(obj: Any) → Products[source]¶

products: Dict[ProductKey, RecipeProducts]¶

class arkimapslib.outputbundle.Reader[source]¶

Read functions for output bundles

abstract find() → List[str][source]¶: List all paths in the bundle

input_summary() → InputSummary[source]¶: Return summary of all inputs used during processing

abstract load_artifact(bundle_path: str) → bytes[source]¶

Load processing artifact.

Return the raw file data

abstract load_product(bundle_path: str) → bytes[source]¶

Load a product by its path.

Return the raw PNG image data

log() → Log[source]¶: Return the log generated during processing

products() → Products[source]¶: Return metadata for all products

abstract version() → str[source]¶: Return the bundle version information

class arkimapslib.outputbundle.RecipeOrders(legend_info: Dict[str, Any] | None)[source]¶

Information and statistics for all orders generated from one recipe

legend_info: Dict[str, Any] | None¶: Legend information for products produced from this recipe

class arkimapslib.outputbundle.RecipeProducts(*, flavour: Dict[str, Any] | None = None, recipe: Dict[str, Any] | None = None, reftimes: Dict[str, ReftimeProducts] = None)[source]¶

Information about all products generated for a recipe

add_order(order: Order) → None[source]¶: Add information from this order

flavour: Dict[str, Any] | None¶: Flavour used for this product

recipe: Dict[str, Any] | None¶: Name of the recipe used for this product

reftimes: Dict[str, ReftimeProducts]¶: Product information by reference time

class arkimapslib.outputbundle.ReftimeProducts(*, inputs: Set[str] = None, steps: Dict[ModelStep, int] = None, legend_info: Dict[str, Any] | None = None, render_stats: RenderStats = None, products: Dict[str, ProductInfo] = None)[source]¶

Information and statistics for all orders for a given reftime

add_order(order: Order)[source]¶

add_product(relpath: str, georef: Dict[str, Any] | None = None) → None[source]¶: Add information for a product.

dict(*args: Any, **kwargs: Any) → Dict[str, Any][source]¶: Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

inputs: Set[str]¶: Names of inputs used

legend_info: Dict[str, Any] | None¶: Legend information for products produced from this recipe

products: Dict[str, ProductInfo]¶: Products indexed by relative path

render_stats: RenderStats¶: Rendering statistics

steps: Dict[ModelStep, int]¶: Numer of products produced for each step

class arkimapslib.outputbundle.RenderStats(*, time_ns: int = 0)[source]¶

Rendering statistics.

time_ns: int¶

class arkimapslib.outputbundle.Serializable[source]¶

Base for classes that can be serialized to JSON

classmethod from_jsonable(data: Dict[str, Any])[source]¶: Recreate an object serialized by to_jsonable

to_jsonable() → Dict[str, Any][source]¶: Return a version of this object that can be serialized to JSON, and deserialized with from_jsonable()

class arkimapslib.outputbundle.TarReader(path: Path | str)[source]¶

Read an output bundle from a tar file

Usage:

with TarReader(path) as reader:
    for name in reader.find():
        if name.endswith(".png"):
            print(name)

See Reader for the full method list

find() → List[str][source]¶: List all paths in the bundle

load_artifact(bundle_path: str) → bytes[source]¶

Load processing artifact.

Return the raw file data

load_product(bundle_path: str) → bytes[source]¶

Load a product by its path.

Return the raw PNG image data

version() → str[source]¶: Return the bundle version information

class arkimapslib.outputbundle.TarWriter(out: IO[bytes])[source]¶

Write an output bundle as a tar file.

add_artifact(bundle_path: str, data: IO[bytes])[source]¶: Add a processing artifact

add_product(bundle_path: str, data: IO[bytes])[source]¶: Add a product

class arkimapslib.outputbundle.Writer[source]¶

Write functions for output bundles

abstract add_artifact(bundle_path: str, data: IO[bytes])[source]¶: Add a processing artifact

add_input_summary(input_summary: InputSummary)[source]¶: Add inputs.json with a summary of inputs used

add_log(entries: Log)[source]¶

Add log.json with log entries generated during processing.

If no log entries were generated, log.json is not added.

abstract add_product(bundle_path: str, data: IO[bytes])[source]¶: Add a product

add_products(products: Products)[source]¶: Add products.json with information about generated products

class arkimapslib.outputbundle.ZipReader(path: Path | str)[source]¶

Read an output bundle from a zip file

Usage:

with ZipReader(path) as reader:
    for name in reader.find():
        if name.endswith(".png"):
            print(name)

See Reader for the full method list

find() → List[str][source]¶: List all paths in the bundle

load_artifact(bundle_path: str) → bytes[source]¶

Load processing artifact.

Return the raw file data

load_product(bundle_path: str) → bytes[source]¶

Load a product by its path.

Return the raw PNG image data

version() → str[source]¶: Return the bundle version information

class arkimapslib.outputbundle.ZipWriter(out: IO[bytes])[source]¶

Write an output bundle as a zip file.

add_artifact(bundle_path: str, data: IO[bytes])[source]¶: Add a processing artifact

add_product(bundle_path: str, data: IO[bytes])[source]¶: Add a product

outputbundle package¶

Module contents¶

arkimaps

Navigation

Related Topics