mentat.stats.rrd module

This module contains object encapsulation of RRDTOOL for easier usage in Mentat project.

Overview

In Mentat RRD databases are used to store quick periodical performance statistics. Each database file is configured to store only single dataset and when generating charts multiple RRD files must be combined when necessary to create more complex charts. This approach has the benefit of very easy way of deleting datasets, that are not required anymore, or obsolete and not appearing in life data anymore.

Every RRD database file is configured in following way:

  • internal dataset name: eventcnt

  • default step: 300 seconds (5 minutes)

  • raw storage size: 3 months

  • aggregated storage size: 1 year (min, max, avg)

The whole RRD handling functionality is implemented in RrdStats, which receives certain parameters (like locations of database and chart files) upon instantination and those can be then omitted from calls to all other methods. See the documentation below for more details.

There is a definition of color palette in COLOR_PALETTE constant value, which contains all possible colors usable in RRD charts.

The implementation is based on Python library rrdtool (PyPI, documentation).

RRD database identifier naming convention

Names of RRD database files are constructed as follows:

type.snake_case_name.rrd

When being processed, each RRD database file name is split at . character, which yields three tokens:

  • type: type, category, or group of the dataset

  • snake_case_name: escaped name of the dataset, anything [^-_a-zA-Z0-9] is replaced with _

  • file suffix

The idea behind the naming scheme is, that statistics are being calculated separatelly for multiple message groups, for example according to the name of the detector, categories, etc. The type token should uniquelly indentify each of these groups. The snake_case_name then contains underscore-escaped name of the detector/category/etc.

The type.snake_case_name pair is called db_id in further documentation and it is unique identifier of the dataset.

Example usage

>>> rrd_stats = mentat.stats.rrd.RrdStats('/var/tmp', '/var/tmp')
>>> stats.find_dbs()
defaultdict(<class 'list'>, {})

>>> rrd_stats.prepare_db('typea.test_a')
('/var/tmp/utest_rrdstats/typea.testa.rrd', True)
>>> rrd_stats.prepare_db('typea.test_b')
('/var/tmp/utest_rrdstats/typea.testb.rrd', True)
>>> rrd_stats.prepare_db('typeb.test_a')
('/var/tmp/utest_rrdstats/typeb.testa.rrd', True)
>>> rrd_stats.prepare_db('typeb.test_b')
('/var/tmp/utest_rrdstats/typeb.testb.rrd', True)
>>> stats.find_dbs()
defaultdict(<class 'list'>, {'typeb': [('typeb.test_a', 'typeb', 'test_a', '/var/tmp/typeb.test_a.rrd', False), ('typeb.test_b', 'typeb', 'test_b', '/var/tmp/typeb.test_b.rrd', False)], 'typea': [('typea.test_a', 'typea', 'test_a', '/var/tmp/typea.test_a.rrd', False), ('typea.test_b', 'typea', 'test_b', '/var/tmp/typea.test_b.rrd', False)]})
class mentat.stats.rrd.RrdStats(rrds_dir, reports_dir, step=300)[source]

Bases: object

Class implementing RRD database file management and chart generation.

static clean(name)[source]

Cleanup given name so it can be used as RRD database file name.

Parameters

name (str) – Name to be cleaned.

Returns

Name with all forbidden characters escaped with _.

Return type

str

export(ds_id)[source]

Export whole RRD database into JSON structure.

Parameters

ds_id (str) – Identifier of the dataset.

Returns

Absolute filesystem path to the prepared RRD database file as string, boolean flag marking creation of new fileand axported dataset as JSON structure.

Return type

tuple

find_dbs(flt_type=None)[source]

Find all currently available RRD database files.

Parameters

flt_type (str) – Dataset type filter to dump part of the full result.

Returns

Structured dictionary containing list of RRD database metadata structures for each type.

Return type

collections.defaultdict

generate(tst, flt_type=None)[source]

Generate all available RRD charts, spark charts and JSON export files.

Parameters

  • tst (int) – Upper time boundary as unix timestamp.

  • flt_type (str) – Dataset type filter to dump part of the full result.

Returns

List of generated files.

Return type

list of str

lookup(flt_type=None)[source]

Lookup all available RRD charts, spark charts and JSON export files.

Parameters

flt_type (str) – Dataset type filter to dump part of the full result.

Returns

List of available charts.

Return type

list of dict

prepare_db(ds_id, time_start=None)[source]

Ensure existence of appropriate RRD database file for storing single data set.

Parameters

  • ds_id (str) – Identifier of the dataset.

  • time_start (int) – Name to be cleaned.

Returns

Absolute filesystem path to the prepared RRD database file as string and boolean flag marking creation of new file.

Return type

tuple

update(ds_id, value, tst=None)[source]

Update given RRD database.

Parameters

  • ds_id (str) – Identifier of the dataset.

  • value (int) – Value to be set.

  • tst (int) – Update timestamp.

Returns

Absolute filesystem path to the prepared RRD database file as string and boolean flag marking creation of new file.

Return type

tuple

update_all(value, tst=None, flt_type=None)[source]

Updade all RRD database files with given value.

Parameters

  • value (int) – Value to be set.

  • tst (int) – Update timestamp.

  • flt_type (str) – Dataset type filter to dump part of the full result.

Returns

List of updated RRD database files.

Return type

list of tuples

exception mentat.stats.rrd.RrdStatsException(description)[source]

Bases: Exception

Base class for all RrdStats specific exceptions.

The rrdtool module does not provide enough granularity with its exceptions and it is necessary to parse the exception string to determine the exact problem. This module provides custom set of exceptions to diferentiate between different problems.

exception mentat.stats.rrd.RrdsCreateException(description)[source]

Bases: mentat.stats.rrd.RrdStatsException

Exceptions occuring during creating of RRD database files.

exception mentat.stats.rrd.RrdsExportException(description)[source]

Bases: mentat.stats.rrd.RrdStatsException

Exceptions occuring during exporting of RRD database files.

exception mentat.stats.rrd.RrdsGraphException(description)[source]

Bases: mentat.stats.rrd.RrdStatsException

Exceptions occuring during generating of RRD charts.

exception mentat.stats.rrd.RrdsUpdateException(description)[source]

Bases: mentat.stats.rrd.RrdStatsException

Exceptions occuring during updating of RRD database files.