putarki.sh File Reference

Modules with functions for archiving the model result to a configured Arkimet dataset. More...

Functions
	putarki_archive_and_wait ($1,$*)
	Archive one or more files and wait for the completion of the operation. More...
	putarki_archive ($1,$*)
	Archive one or more files. More...
	putarki_wait_for_deletion ($*)
	Wait until the requested files have been deleted. More...
	putarki_model_output ($1, $2)
	Archive the output of a model run while it is being produced. More...

Detailed Description

Modules with functions for archiving the model result to a configured Arkimet dataset.

This module provides functions for archiving files, tipically model output in GRIB format, into a desired dataset of an Arkimet archive.

It is an optional module and it has to be sourced after the nwptime.sh module.

When sourcing the module, the following default assignments are made:

PUTARKI_WAITSTART=30
PUTARKI_WAITMAX=300
PUTARKI_WAITDEL=5

$PUTARKI_WAITSTART is the initial wait time between checks for the appearance of new output files, $PUTARKI_WAITMAX is the maximum wait time between checks when no events happen, $PUTARKI_WAITDEL is the wait time between checks of file disappearance after archiving, all wait times are in seconds. If $ARKI_USE_INOTIFY has the value of Y, then the sleep's between file checks are replaced by the inotifywait command, so that the reaction to changes in filesystem are immediate, in that case the aforementiond wait times are used as a timeout to inotifywait, to account for malfunctions of the inotify process. The value of these environment variables can be changed after sourcing the module.

Function Documentation

putarki_archive_and_wait	(		$1,
			$*
	)

Archive one or more files and wait for the completion of the operation.

This function archives the files passed as arguments to the configured dataset and waits until the files have been archived. It combines the putarki_archive() function for archiving and the putarki_wait_for_deletion() function for waiting.

Parameters

$1	the type of file being archived, either grib or bufr
$*	the files to be archived

putarki_archive	(		$1,
			$*
	)

Archive one or more files.

This function archives the files passed as arguments to the configured dataset. The files are archived according to the value of $ARKI_SCAN_METHOD:

• arki_importer: assuming that a consumer process is active, the files are copied to the directory $ARKI_IMPDIR configured in the consumer, the function exits suddendly without waiting for the termination of the archiving; this method is advantageous because it allows concurrent processes to simultaneously send data to the same dataset
• remote_arki_importer: it is similar to the previous method, but it assumes that the consumer process is active on a different host, in this case the variable $ARKI_IMPSSH indicates the credentials for accessing the remote import server by ssh/scp in the form user@host and $ARKI_IMPDIR indicate the import directory configured on the remote server; paswordless ssh access to the remote server must be set up; this method allows concurrency as well, but it is less performant due to the access to a remote server through ssh.
• arki-scan: an arki-scan is performed with configuration file $ARKI_CONF, in this case the function exits when the archival has finished; this approach does not require a consumer process, but concurrent attempts to archive in the same dataset may fail because of locking issues.

When using the arki_importer approach, the function prints to stdout the list of temporary files created in $ARKI_IMPDIR that are being imported and whose deletion indicates that the archiving has finished. Regardless of the archiving method, it is safe to remove the original file at function return since a copy (or hard link if possible) is made in case of asynchronous archiving.

Parameters

$1	the type of file being archived, either grib or bufr
$*	the files to be archived

putarki_wait_for_deletion

(

$*

)

Wait until the requested files have been deleted.

This function waits until the files passed as arguments (which should reside in $ARKI_IMPDIR) have been deleted, meaning that their archiving has completed. If configured by the environment assignment ARKI_USE_INOTIFY=Y, the function uses the command inotifywait to speedup the detection of the deletion process, otherwise a series of check and sleep is performed. All the operations, regardless of the use of inotify, have a timeout of $PUTARKI_WAITDEL seconds. It is a function specific to the putarki_archive() function, it should not be used as a generic file deletion checking function.

Parameters

$*	the files to be deleted

putarki_model_output	(		$1,
			$2
	)

Archive the output of a model run while it is being produced.

This function waits for the appearing of model ouput files and archives them as soon as possible in the configured Arkimet dataset with the putarki_archive() function. It relies on the creation, in the current directory, of ready-files with name matching $READYFILE_PATTERN and on the existance of a model-specific function model_readyfiletoname() taking as argument the name of a specific ready-file and printing to stdout the name of all the output files related to that ready-file. If the $PUTARKI_WAITTOTAL variable is defined, the function will exit after that number of seconds. even if the work is not finished. For an example of model-specific setup, see the documentation of the cosmo_module.sh module.

Parameters

$1	the number of ready files to wait for before exiting
$2	optional, if equal to -w tells the function to wait for the archiving to complete before exiting