mentat.idea.sqldb module

This module provides class for object representation and conversion of IDEA messages into their appropriate PostgreSQL records. These records can be then stored into database using the mentat.services.eventstorage event persistent storage service.

The resulting record is intended to be stored into PostgreSQL database using flat schema. This is a very simple custom schema and it was designed to tackle performance issues with full relational schema representation of IDEA messages. It is basically a single database table with fixed set of prepared indexed columns for the purposes of searching and the whole IDEA message is then stored as PostgreSQL’s native jsonb datatype inside the last table column.

The schema currently supports indexing of following IDEA message attributes:

  • ID

  • DetectTime

  • Category

  • Description

  • Source.IP (both v4 and v6)

  • Source.Port

  • Source.Type

  • Target.IP (both v4 and v6)

  • Target.Port

  • Target.Type

  • Protocol (both source and target, unique set)

  • Node.Name

  • Node.Type

  • _Mentat.ResolvedAbuses

  • _Mentat.StorageTime

As a side-effect of this approach, searching according to other IDEA message attributes is not possible.

This module is expected to work only with messages based on or compatible with the mentat.idea.internal.Idea class.

This module contains following message class:

Example usage:

>>> import mentat.idea.internal
>>> import mentat.idea.sqldb

# IDEA messages ussually come from regular dicts or JSON.
>>> idea_raw = {...}

# Just pass the dict as parameter to constructor to create internal IDEA.
>>> idea_msg = mentat.idea.internal.Idea(idea_raw)

# Just pass the IDEA message as parameter to constructor to create SQL record.
>>> idea_postgresql = mentat.idea.sqldb.Idea(idea_msg)
class mentat.idea.sqldb.IPList[source]

Bases: list

Custom list container for ipranges objects. This was implemented in order to support custom adaptation of IP objects into SQL query. Please see the mentat.services.eventstorage.IPListAdapter for more details. Please see the psycopg2 documentation for in-depth explanation.

class mentat.idea.sqldb.Idea(idea_event)[source]

Bases: object

Performs conversion of IDEA messages into flat relational model.

category = []
description = None
detecttime = None
eventclass = None
eventseverity = None
get_record()[source]

Return tuple containing object attributes in correct order for insertion into PostgreSQL database using the mentat.services.eventstorage service.

ident = None
inspectionerrors = []
jsonb = None
node_name = []
node_type = {}
protocol = {}
resolvedabuses = []
source_ip = []
source_ip_aggr_ip4 = None
source_ip_aggr_ip6 = None
source_port = []
source_type = {}
storagetime = None
target_ip = []
target_ip_aggr_ip4 = None
target_ip_aggr_ip6 = None
target_port = []
target_type = {}