mentat.services.eventstorage

Event database storage abstraction layer. The current implementation requires the PostgreSQL database and is based directly on the Psycopg2 library for performance reasons.

Warning

Current implementation is for optimalization purposes using some advanced schema features provided by the PostgreSQL database and thus no other engines are currently supported.

Warning

The PostgreSQL extension ip4r must be installed.

References

exception mentat.services.eventstorage.DataError[source]

Bases: mentat.services.eventstorage.EventStorageException

Class for custom event storage exceptions related to data errors.

class mentat.services.eventstorage.EventStorageCursor(cursor)[source]

Bases: object

Encapsulation of psycopg2.cursor class.

close()[source]

Close current database connection.

count_events(parameters=None)[source]

Count the number of IDEA messages in database.

Parameters:parameters (dict) – Count query parameters.
Returns:Number of IDEA messages in database.
Return type:int
delete_event(eventid)[source]

Delete IDEA message with given primary identifier from database.

Parameters:eventid (str) – Primary identifier of the message to fetch.
delete_events(parameters=None)[source]

Delete IDEA messages in database according to given parameters.

Parameters:parameters (dict) – Delete query parameters.
Returns:Number of deleted events.
Return type:int
fetch_event(eventid)[source]

Fetch IDEA message with given primary identifier from database.

Parameters:eventid (str) – Primary identifier of the message to fetch.
Returns:Instance of IDEA message.
Return type:mentat.idea.internal
insert_event(idea_event)[source]

Insert given IDEA message into database.

Parameters:idea_event (mentat.idea.internal) – Instance of IDEA message.
search_column_with(column, function='min')[source]

Search given column with given aggregation function. This method is intended to produce single min or max values for given column name.

search_events(parameters=None, event_factory=<function record_to_idea>, columns=('id', 'detecttime', 'category', 'description', 'source_ip', 'target_ip', 'source_port', 'target_port', 'source_type', 'target_type', 'protocol', 'node_name', 'node_type', 'cesnet_resolvedabuses', 'cesnet_storagetime', 'cesnet_eventclass', 'cesnet_eventseverity', 'cesnet_inspectionerrors', 'event'))[source]

Search IDEA messages in database according to given parameters. The parameters will be passed down to the mentat.services.eventstorage.build_query() function to generate proper SQL query.

Parameters:
search_relapsed_events(group_name, severity, ttl)[source]

Search for list of relapsed events for given group, severity and TTL. Event is considered to be relapsed, when following conditions are met:

  • there is record in thresholds table with thresholds.ttltime <= $ttl (this means that thresholding window expired)
  • there is record in events_thresholded table with events_thresholded.createtime >= thresholds.relapsetime (this meant that the event was thresholded in relapse period)
Parameters:
  • group_name (str) – Name of the abuse group.
  • severity (str) – Event severity.
  • ttl (datetime.datetime) – Record TTL time.
Returns:

List of relapsed events as mentat.idea.internal.Idea objects.
Return type:

list
table_cleanup(table, column, ttl)[source]

Clean expired table records according to given TTL.

Parameters:
  • table (str) – Name of the table to cleanup.
  • column (str) – Name of the column holding the time information.
  • ttl (datetime.datetime) – Maximal valid TTL.
Returns:

Number of cleaned up records.
Return type:

int
threshold_check(key, ttl)[source]

Check thresholding cache for record with given key.

Parameters:
  • key (str) – Record key to the thresholding cache.
  • ttl (datetime.datetime) – Upper TTL boundary for valid record.
Returns:

Full cache record as tuple.
Return type:

tuple
threshold_save(eventid, keyid, group_name, severity, createtime)[source]

Save given event to the list of thresholded events.

Parameters:
  • eventid (str) – Unique event identifier.
  • keyid (str) – Record key to the thresholding cache.
  • group_name (str) – Name of the abuse group.
  • severity (str) – Event severity.
  • createtime (datetime.datetime) – Record creation time.
threshold_set(key, thresholdtime, relapsetime, ttl)[source]

Insert new threshold record into the thresholding cache.

Parameters:
  • key (str) – Record key to the thresholding cache.
  • thresholdtime (datetime.datetime) – Threshold window start time.
  • relapsetime (datetime.datetime) – Relapse window start time.
  • ttl (datetime.datetime) – Record TTL.
thresholded_events_clean()[source]

Clean no longer valid records from list of thresholded events. Record is no longer valid in following cases:

  • there is no appropriate record in thresholds table (there is no longer active thresholding window)
  • the events_thresholded.createtime < thresholds.relapsetime (there is an active thresholding window, but event does not belong to relapse interval)
Returns:Number of cleaned up records.
Return type:int
thresholded_events_count()[source]

Count number of records in list of thresholded events.

Returns:Number of records in list of thresholded events.
Return type:int
thresholds_clean(ttl)[source]

Clean no longer valid threshold records from thresholding cache.

Parameters:ttl (datetime.datetime) – Maximal valid TTL.
Returns:Number of cleaned up records.
Return type:int
thresholds_count()[source]

Count threshold records in thresholding cache.

Returns:Number of records in thresholding cache.
Return type:int
watchdog_events(interval)[source]

Perform watchdog operation on event database: Check if any new events were added into the database within given time interval.

Parameters:interval (int) – Desired time interval in seconds.
Returns:True in case any events were stored within given interval, False otherwise.
Return type:bool
exception mentat.services.eventstorage.EventStorageException[source]

Bases: Exception

Class for custom event storage exceptions.

class mentat.services.eventstorage.EventStorageService(**conncfg)[source]

Bases: object

Proxy object for working with persistent SQL based event storages. Maintains and provides access to database connection.

close()[source]

Close current database connection.

commit()[source]

Commit currently pending changes into persistent storage.

count_events(parameters=None)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.count_events() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

cursor_new()[source]

Create new database cursor.

database_create()[source]

Create database SQL schema.

database_drop()[source]

Drop database SQL schema.

database_status(brief=False)[source]

Determine status of all tables within current database and general PostgreSQL configuration.

delete_event(eventid)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.delete_event() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

delete_events(parameters=None)[source]

Delete IDEA messages in database according to given optional parameters. These parameters will

Parameters:parameters (dict) – Optional delete query parameters.
Returns:Number of deleted events.
Return type:int
distinct_values(column)[source]

Return distinct values of given table column.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

Parameters:column (str) – Name of the column to query for distinct values.
Returns:List of distinct values.
Return type:list
fetch_event(eventid)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.fetch_event() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

index_create()[source]

Create default set of table indices.

index_drop()[source]

Drop default set of table indices.

insert_event(idea_event)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.insert_event() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

mogrify(query, parameters)[source]

Format given SQL query, replace placeholders with given parameters and return resulting SQL query as string.

rollback()[source]

Rollback currently pending changes into persistent storage.

search_column_with(column, function='min')[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.search_column_with() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

search_events(parameters=None, event_factory=<function record_to_idea>, columns=('id', 'detecttime', 'category', 'description', 'source_ip', 'target_ip', 'source_port', 'target_port', 'source_type', 'target_type', 'protocol', 'node_name', 'node_type', 'cesnet_resolvedabuses', 'cesnet_storagetime', 'cesnet_eventclass', 'cesnet_eventseverity', 'cesnet_inspectionerrors', 'event'))[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.search_events() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

search_relapsed_events(group_name, severity, ttl)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.search_relapsed_events() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

table_cleanup(table, column, ttl)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.table_cleanup() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

table_status(table_name, time_column)[source]

Determine status of given table within current database.

threshold_check(key, threshold)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.threshold_check() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

threshold_save(eventid, keyid, group_name, severity, createtime)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.threshold_save() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

threshold_set(key, thresholdtime, relapsetime, ttl)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.threshold_set() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

thresholded_events_clean()[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.thresholded_events_clean() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

thresholded_events_count()[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.thresholded_events_count() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

thresholds_clean(threshold)[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.thresholds_clean() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

thresholds_count()[source]

This method is a convenience wrapper for underlying mentat.services.eventstorage.EventStorageCursor.thresholds_count() method.

It will automatically commit transaction for successfull database operation and rollback the invalid one.

watchdog_events(interval)[source]

Perform watchdog operation on event database: Check if any new events were added into the database within given time interval.

Parameters:interval (int) – Desired time interval in seconds.
Returns:True in case any events were stored within given interval, False otherwise.
Return type:bool
class mentat.services.eventstorage.EventStorageServiceManager(core_config, updates=None)[source]

Bases: object

Class representing a custom _EventStorageServiceManager_ capable of understanding and parsing Mentat system core configurations.

close()[source]

Close internal storage connection.

service()[source]

Return handle to storage connection service according to internal configurations.

Returns:Reference to storage service.
Return type:mentat.services.eventstorage.EventStorageService
class mentat.services.eventstorage.IPListAdapter(seq)[source]

Bases: object

Adapt a mentat.idea.sqldb.IPList to an SQL quotable object.

Resources: http://initd.org/psycopg/docs/advanced.html#adapting-new-python-types-to-sql-syntax

getquoted()[source]

Implementation of psycopg2 adapter interface.

prepare(conn)[source]

Implementation of psycopg2 adapter interface.

exception mentat.services.eventstorage.StorageIntegrityError[source]

Bases: mentat.services.eventstorage.EventStorageException

Class for custom event storage exceptions related to integrity errors.

mentat.services.eventstorage.build_query(parameters=None, qtype='select', columns=('id', 'detecttime', 'category', 'description', 'source_ip', 'target_ip', 'source_port', 'target_port', 'source_type', 'target_type', 'protocol', 'node_name', 'node_type', 'cesnet_resolvedabuses', 'cesnet_storagetime', 'cesnet_eventclass', 'cesnet_eventseverity', 'cesnet_inspectionerrors', 'event'))[source]

Build SQL database query according to given parameters.

Parameters:
  • parameters (dict) – Query parametersas complex dictionary structure.
  • qtype (str) – Type of the generated query (‘select’,’count’,’delete’).
Returns:

Generated query as psycopg2.sql.SQL and apropriate arguments.
Return type:

tuple
mentat.services.eventstorage.close()[source]

Close database connection on mentat.services.eventstorage.EventStorageService instance from module level manager.

mentat.services.eventstorage.init(core_config, updates=None)[source]

(Re-)Initialize mentat.services.eventstorage.EventStorageServiceManager instance at module level and store the refence within module.

Parameters:
  • core_config (dict) – Mentat core configuration structure.
  • updates (dict) – Optional configuration updates (same structure as core_config).
mentat.services.eventstorage.manager()[source]

Obtain reference to mentat.services.eventstorage.EventStorageServiceManager instance stored at module level.

Returns:Storage service manager reference.
Return type:mentat.services.eventstorage.EventStorageServiceManager
mentat.services.eventstorage.record_to_idea(val)[source]

Convert given SQL record object, as fetched from PostgreSQL database, directly into mentat.idea.internal.Idea object.

mentat.services.eventstorage.record_to_idea_ghost(val)[source]

Convert given SQL record object, as fetched from PostgreSQL database, directly into mentat.idea.internal.IdeaGhost object.

mentat.services.eventstorage.service()[source]

Obtain reference to mentat.services.eventstorage.EventStorageService instance from module level manager.

Returns:Storage service reference.
Return type:mentat.services.eventstorage.EventStorageService
mentat.services.eventstorage.set_manager(man)[source]

Set manager from outside of the module. This should be used only when you know exactly what you are doing.