Python dballe.volnd package

The volnd API

volnd is a way of extracting entire matrixes of data out of a DB-All.e database.

This module allows to extract multidimensional matrixes of data given a list of dimension definitions. Every dimension definition defines what kind of data goes along that dimension.

Dimension definitions can be shared across different extracted matrixes and multiple extractions, allowing to have different matrixes whose indexes have the same meaning.

This example code extracts temperatures in a station by datetime matrix:

from dballe import volnd
query = dict()
query["var"] = "B12101"
query["rep_memo"] = "synop"
query["level"] = (103, 2000)
query["trange"] = (254, 0, 0)
with db.transaction() as tr:
    vars = volnd.read(tr.query_data(query), (volnd.AnaIndex(), volnd.DateTimeIndex()))
data = vars["B12101"]
# Data is now a 2-dimensional Masked Array with the data
#
# Information about what values correspond to an index in the various
# directions can be accessed in data.dims, which contains one list per
# dimension with all the information corresponding to every index.
print("Ana dimension is", len(data.dims[0]), "items long")
print("Datetime dimension is", len(data.dims[1]), "items long")
print("First 10 stations along the Ana dimension:", data.dims[0][:10])
print("First 10 datetimes along the DateTime dimension:", data.dims[1][:10])

class dballe.volnd.AnaIndex(shared=True, frozen=False, start=None)

Index for stations, as they come out of the database.

The constructor syntax is: AnaIndex(shared=True, frozen=False, start=None).

The index saves all stations as AnaIndexEntry tuples, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

class dballe.volnd.AnaIndexEntry

AnaIndex entry, with various data about a single station.

It is a named tuple of 4 values:

• id: station id

• lat: latitude

• lon: longitude

• ident: mobile station identifier, or None

classmethod from_record(rec)

Create an index entry from the contents of a Dict[str, Any]

class dballe.volnd.Data(name, dims, checkConflicts=True)

Container for collecting variable data. It contains the variable data array and the dimension indexes.

If v is a Data object, you can access the tuple with the dimensions as v.dims, and the masked array with the values as v.vals.

append(rec, val)

Collect a new value from the given dballe record.

You need to call finalise() before the values can be used.

appendAttrs(rec, codes=None)

Collect attributes to append to the record.

You need to call finalise() before the values can be used.

finalise()

Stop collecting values and create a masked array with all the values collected so far.

class dballe.volnd.DateTimeIndex(shared=True, frozen=False, start=None)

Index for datetimes, as they come out of the database.

The constructor syntax is: DateTimeIndex(shared=True, frozen=False, start=None).

The index saves all datetime values as datetime.datetime objects, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

class dballe.volnd.Index(shared=True, frozen=False)

Base class for all volnd indices.

An Index describes each entry along one dimension of a volnd volume. There is an entry in the index for each point along that axis, and each entry can be an arbitrary structure with details.

Index objects can be shared between homogeneous volumes.

copy()

Return another version of this index: it can be a reference to the exact same index if shared=True; otherwise it’s a new, empty version.

freeze()

Set the index as frozen: indexing elements not already in the index will raise a SkipDatum exception

class dballe.volnd.IntervalIndex(start, step, tolerance=0, end=None, *args, **kwargs)

Index into equally spaced points in time, starting at start, with a point every step time.

Index points are at fixed time intervals, and data is acquired in one point only if it is within a given tolerance from the interval.

The constructor syntax is: IntervalIndex(start, step, tolerance=0, end=None, shared=True, frozen=False).

start is a datetime.datetime object giving the starting time of the time interval of this index.

step is a datetime.timedelta object with the interval between sampling points.

tolerance is a datetime.timedelta object specifying the maximum allowed interval between a datum datetime and the sampling step. If the interval is bigger than the tolerance, the data is discarded.

end is an optional datetime.datetime object giving the ending time of the time interval of the index. If omitted, the index will end at the latest accepted datum coming out of the database.

approve(rec)

copy()

Return another version of this index: it can be a reference to the exact same index if shared=True; otherwise it’s a new, empty version.

index_record(rec)

short_name()

class dballe.volnd.LevelIndex(shared=True, frozen=False, start=None)

Index for levels, as they come out of the database

The constructor syntax is: LevelIndex(shared=True, frozen=False, start=None).

The index saves all levels as dballe.Level tuples, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

class dballe.volnd.ListIndex(shared=True, frozen=False, start=None)

Indexes records along an axis.

Each index entry is an arbitrary details structure extracted from records. All records at a given index position will have the same details.

approve(rec)

Return true if the record can be placed along this index

details_from_record(rec)

Extract the full data information from the record

index_record(rec)

Return an integer index along this axis for the given record

key_from_record(rec)

Extract the indexing key from the record

class dballe.volnd.NetworkIndex(shared=True, frozen=False, start=None)

Index for networks, as they come out of the database.

The constructor syntax is: NetworkIndex(shared=True, frozen=False, start=None).

The index saves all networks as NetworkIndexEntry tuples, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

exception dballe.volnd.SkipDatum

class dballe.volnd.TimeRangeIndex(shared=True, frozen=False, start=None)

Index for time ranges, as they come out of the database.

The constructor syntax is: TimeRangeIndex(shared=True, frozen=False, start=None).

The index saves all time ranges as dballe.TimeRange tuples, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

dballe.volnd.read(cursor, dims, filter=None, checkConflicts=True, attributes=None)

cursor is a dballe.Cursor resulting from a dballe query

dims is the sequence of indexes to use for shaping the data matrixes

filter is an optional filter function that can be used to discard values from the query: if filter is not None, it will be called for every output record and if it returns False, the record will be discarded

checkConflicts tells if we should raise an exception if two values from the database would fill in the same position in the matrix

attributes tells if we should read attributes as well: if it is None, no attributes will be read; if it is True, all attributes will be read; if it is a sequence, then it is the sequence of attributes that should be read.

dballe.volnd.tddivmod(td1, td2)

dballe.volnd.tddivmod1(td1, td2)

Division and quotient between time deltas

dballe.volnd.tddivmod2(td1, td2)

Division and quotient between time deltas (alternate implementation using longs)

dballe.volnd.tddivmod3(td1, td2)

Dimensions

This is the list of dimensions supported by dballe.volnd:

class dballe.volnd.AnaIndex(shared=True, frozen=False, start=None)

Index for stations, as they come out of the database.

The constructor syntax is: AnaIndex(shared=True, frozen=False, start=None).

The index saves all stations as AnaIndexEntry tuples, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

The data object used by AnaIndex is:

class dballe.volnd.AnaIndexEntry

AnaIndex entry, with various data about a single station.

It is a named tuple of 4 values:

• id: station id

• lat: latitude

• lon: longitude

• ident: mobile station identifier, or None

classmethod from_record(rec)

Create an index entry from the contents of a Dict[str, Any]

class dballe.volnd.NetworkIndex(shared=True, frozen=False, start=None)

Index for networks, as they come out of the database.

The constructor syntax is: NetworkIndex(shared=True, frozen=False, start=None).

The index saves all networks as NetworkIndexEntry tuples, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

class dballe.volnd.LevelIndex(shared=True, frozen=False, start=None)

Index for levels, as they come out of the database

The constructor syntax is: LevelIndex(shared=True, frozen=False, start=None).

The index saves all levels as dballe.Level tuples, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

class dballe.volnd.TimeRangeIndex(shared=True, frozen=False, start=None)

Index for time ranges, as they come out of the database.

The constructor syntax is: TimeRangeIndex(shared=True, frozen=False, start=None).

The index saves all time ranges as dballe.TimeRange tuples, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

class dballe.volnd.DateTimeIndex(shared=True, frozen=False, start=None)

Index for datetimes, as they come out of the database.

The constructor syntax is: DateTimeIndex(shared=True, frozen=False, start=None).

The index saves all datetime values as datetime.datetime objects, in the same order as they come out of the database.

details_from_record(rec)

Extract the full data information from the record

key_from_record(rec)

Extract the indexing key from the record

short_name()

class dballe.volnd.IntervalIndex(start, step, tolerance=0, end=None, *args, **kwargs)

Index into equally spaced points in time, starting at start, with a point every step time.

Index points are at fixed time intervals, and data is acquired in one point only if it is within a given tolerance from the interval.

The constructor syntax is: IntervalIndex(start, step, tolerance=0, end=None, shared=True, frozen=False).

start is a datetime.datetime object giving the starting time of the time interval of this index.

step is a datetime.timedelta object with the interval between sampling points.

tolerance is a datetime.timedelta object specifying the maximum allowed interval between a datum datetime and the sampling step. If the interval is bigger than the tolerance, the data is discarded.

end is an optional datetime.datetime object giving the ending time of the time interval of the index. If omitted, the index will end at the latest accepted datum coming out of the database.

approve(rec)

copy()

Return another version of this index: it can be a reference to the exact same index if shared=True; otherwise it’s a new, empty version.

index_record(rec)

short_name()

Extracting data

The extraction is done using the dballe.volnd.read() function:

dballe.volnd.read(cursor, dims, filter=None, checkConflicts=True, attributes=None)

cursor is a dballe.Cursor resulting from a dballe query

dims is the sequence of indexes to use for shaping the data matrixes

filter is an optional filter function that can be used to discard values from the query: if filter is not None, it will be called for every output record and if it returns False, the record will be discarded

checkConflicts tells if we should raise an exception if two values from the database would fill in the same position in the matrix

attributes tells if we should read attributes as well: if it is None, no attributes will be read; if it is True, all attributes will be read; if it is a sequence, then it is the sequence of attributes that should be read.

The result of dballe.volnd.read() is a dict mapping output variable names to a dballe.volnd.Data object with the results. All the Data objects share their indexes unless the xxx-Index definitions have been created with shared=False.

This is the dballe.volnd.Data class documentation:

class dballe.volnd.Data(name, dims, checkConflicts=True)

Container for collecting variable data. It contains the variable data array and the dimension indexes.

If v is a Data object, you can access the tuple with the dimensions as v.dims, and the masked array with the values as v.vals.

append(rec, val)

Collect a new value from the given dballe record.

You need to call finalise() before the values can be used.

appendAttrs(rec, codes=None)

Collect attributes to append to the record.

You need to call finalise() before the values can be used.

finalise()

Stop collecting values and create a masked array with all the values collected so far.