mentat.idea.jsondict module

This module provides class for object representation and conversion of IDEA messages into JSON structures containing only primitive datat types (int, str, bool).

This module is expected to work only with messages based on or compatible with the mentat.idea.internal.Idea class, data conversions are very narrow regarding required input data types and will complain.

This module contains following message class:

  • mentat.idea.json.Idea

    Forward conversion into JSON data format.

Example usage:

>>> import mentat.idea.internal
>>> import mentat.idea.json

# IDEA messages ussually come from regular dicts or JSON
>>> idea_raw = {...}

# Just pass the dict as parameter to constructor
>>> idea_msg = mentat.idea.internal.Idea(idea_raw)

# When you want to convert IDEA message back into JSON primitive:
>>> idea_json = mentat.idea.json.Idea(idea_msg)
class mentat.idea.jsondict.AttachDict(init_data=None)[source]

Bases: TypedDict

Typed dictionary representing Attach substructure in IDEA message structure.

allow_unknown = True
typedef = {'Content': {'description': 'Attachment content.', 'type': <class 'str'>}, 'ContentCharset': {'description': 'Name of the content character set according to [[http://www.iana.org/assignments/character-sets/character-sets.xhtml|IANA list]]. If key is not defined, unspecified binary encoding is assumed.', 'type': <function Charset>}, 'ContentEncoding': {'description': 'Encoding of the content, if feasible. Nonexistent key means native JSON encoding.', 'type': <function Encoding>}, 'ContentID': {'description': 'If content of attachment is transferred separately (in underlaying container), this key contains list of external IDs of the content, so it can be paired back to message.', 'type': <function simplify.<locals>.<lambda>>}, 'ContentType': {'description': 'Internet Media Type of the attachment, according to [[http://tools.ietf.org/html/rfc2046|RFC 2046]] and related. Along with [[http://www.iana.org/assignments/media-types/media-types.xhtml|types standardized by IANA]] also non standard but widely used media types can be used (for examples see [[http://www.freeformatter.com/mime-types-list.html|MIME types list at freeformatter.com]]).', 'type': <function MediaType>}, 'ExternalURI': {'description': 'If content of attachment is available and/or recognizable from external source, this is list of defining URIs (usually URLs). May also be URN (according to [[http://tools.ietf.org/html/rfc2141|RFC 2141]]) in registered namespace ([[http://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml|IANA]]) or unregistered ad-hoc namespace bearing reasonable information value and uniqueness, such as "urn:mhr:55eaf7effadc07f866d1eaed9c64e7ee49fe081a", "magnet:?xt=urn:sha1:YNCKHTQCWBTRNJIV4WNAE52SJUQCZO5C".', 'type': <function simplify.<locals>.<lambda>>}, 'FileName': {'description': 'List of filenames of the attached file.', 'type': <function simplify.<locals>.<lambda>>}, 'Handle': {'description': 'Message unique identifier for reference through Attach elements.', 'type': <function Handle>}, 'Hash': {'description': 'Listof checksums of the content (for example "sha1:794467071687f7c59d033f4de5ece6b46415b633" or "md5:dc89f0b4ff9bd3b061dd66bb66c991b1").', 'type': <function simplify.<locals>.<lambda>>}, 'Note': {'description': 'Free text human readable additional note.', 'type': <class 'str'>}, 'Ref': {'description': 'List of references to known sources, related to attack and/or vulnerability, specific to this attachment. May be URL of the additional info, or URN (according to [[http://tools.ietf.org/html/rfc2141|RFC 2141]]) in registered namespace ([[http://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml|IANA]]) or unregistered ad-hoc namespace bearing reasonable information value and uniqueness, such as "urn:clamav:Win.Trojan.Banker-14334".', 'type': <function simplify.<locals>.<lambda>>}, 'Size': {'description': 'Length of the content.', 'type': <class 'int'>}, 'Type': {'description': 'List of attachment type tags.', 'type': <function simplify.<locals>.<lambda>>}}
mentat.idea.jsondict.Duration(val)[source]

Conversion of IDEA Timestamp datatype into appropriate string representation.

class mentat.idea.jsondict.Idea(init_data=None)[source]

Bases: IdeaBase

This class implements datatype conversions to format appropriate for raw JSON representation of IDEA messages.

Despite the fact it is based on idea.base.IdeaBase class, it is designed to process IDEA messages based on mentat.idea.internal.Idea class.

allow_unknown = True
typedef = {'AggrID': {'description': 'List of identifiers of messages, which are aggregated into more concise form by this message. Should be sent mostly by intermediary nodes, which detect duplicates, or aggregate events, spanning multiple detection windows, into one longer.', 'type': <function simplify.<locals>.<lambda>>}, 'AltNames': {'description': 'List of alternative identifiers; strings which help to pair the event to internal system information (for example tickets in request tracker systems).', 'type': <function simplify.<locals>.<lambda>>}, 'Attach': {'description': 'Array of additional attachments information and data.', 'type': <function simplify.<locals>.<lambda>>}, 'ByteCount': {'description': 'Number of bytes transferred.', 'type': <class 'int'>}, 'Category': {'description': 'List of event categories.', 'required': True, 'type': <function simplify.<locals>.<lambda>>}, 'CeaseTime': {'description': 'Deduced end of the event/attack.', 'type': <function Timestamp>}, 'Confidence': {'description': 'Confidence of detector in its own reliability of this particular detection. (0 - surely false, 1 - no doubts). If key is not presented, detector does not know (or has no capability to estimate the confidence).', 'type': <class 'float'>}, 'ConnCount': {'description': 'Number of individual connections attempted or taken place.', 'type': <class 'int'>}, 'CorrelID': {'description': 'List of identifiers of messages, which are information sources for creation of this message in case the message has been created based on correlation/analysis/deduction of other messages.', 'type': <function simplify.<locals>.<lambda>>}, 'CreateTime': {'description': 'Timestamp of the creation of the IDEA message. May point out delay between detection and processing of data.', 'type': <function Timestamp>}, 'Description': {'description': 'Short free text human readable description.', 'type': <class 'str'>}, 'DetectTime': {'description': 'Timestamp of the moment of detection of event (not necessarily time of the event taking place). This timestamp is mandatory, because every detector is able to know when it detected the information - for example when line about event appeared in the logfile, or when its information source says the event was detected, or at least when it accepted the information from the source.', 'required': True, 'type': <function Timestamp>}, 'EventTime': {'description': 'Deduced start of the event/attack, or just time of the event if its solitary.', 'type': <function Timestamp>}, 'FlowCount': {'description': 'Number of individual simplex (one direction) flows.', 'type': <class 'int'>}, 'Format': {'default': 'IDEA0', 'description': 'Identifier of the IDEA container.', 'required': True, 'type': <function Version>}, 'ID': {'default': <function <lambda>>, 'description': 'Unique message identifier.', 'required': True, 'type': <function ID>}, 'Node': {'description': 'List of detector or possible intermediary (event aggregator, correlator, etc.) descriptions, last visited first (as in e-mail Received headers).', 'type': <function simplify.<locals>.<lambda>>}, 'Note': {'description': 'Free text human readable addidional note, possibly longer description of incident if not obvious.', 'type': <class 'str'>}, 'PacketCount': {'description': 'Number of individual packets transferred.', 'type': <class 'int'>}, 'PredID': {'description': 'List of identifiers of messages, which are obsoleted and information in them is replaced by this message. Should be sent only by detection nodes to incorporate further data about ongoing event.', 'type': <function simplify.<locals>.<lambda>>}, 'Ref': {'description': 'List of references to known sources, related to attack and/or vulnerability. May be URL of the additional info, or URN (according to [[http://tools.ietf.org/html/rfc2141|RFC 2141]]) in registered namespace ([[http://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml|IANA]]) or unregistered ad-hoc namespace bearing reasonable information value and uniqueness, such as "urn:cve:CVE-2013-5634".', 'type': <function simplify.<locals>.<lambda>>}, 'RelID': {'description': 'List of otherwise related messages identifiers.', 'type': <function simplify.<locals>.<lambda>>}, 'Source': {'description': 'List of dictionaries of information concerning particular source of the problem.', 'type': <function simplify.<locals>.<lambda>>}, 'Target': {'description': 'List of dictionaries of information concerning particular target of the problem.', 'type': <function simplify.<locals>.<lambda>>}, 'WinEndTime': {'description': 'End of aggregation window in which event has been observed.', 'type': <function Timestamp>}, 'WinStartTime': {'description': 'Beginning of aggregation window in which event has been observed.', 'type': <function Timestamp>}, '_CESNET': {'description': 'Custom CESNET/Mentat abominations to IDEA definition', 'type': <function simplify.<locals>.<lambda>>}, '_Mentat': {'description': 'Custom CESNET/Mentat abominations to IDEA definition', 'type': <function simplify.<locals>.<lambda>>}, 'ts': {'description': 'CESNET specific timestamp as NTP timestamp', 'type': <function Timestamp>}, 'ts_u': {'description': 'CESNET specific timestamp as native Unix timestamp', 'type': <class 'int'>}}
class mentat.idea.jsondict.MentatDict(init_data=None)[source]

Bases: TypedDict

Typed dictionary representing _Mentat/_CESNET substructure in IDEA message structure.

allow_unknown = True
typedef = {'EventClass': {'description': 'Event class determined by inspection', 'type': <class 'str'>}, 'EventSeverity': {'description': 'Event severity determined by inspection', 'type': <class 'str'>}, 'EventTemplate': {'description': 'Template used to generate the event (deprecated)', 'type': <class 'str'>}, 'Impact': {'description': 'More user friendly description of event impact, used for reporting (IDMEF legacy)', 'type': <class 'str'>}, 'InspectionErrors': {'description': 'List of event pecularities found during inspection', 'type': <function simplify.<locals>.<lambda>>}, 'ResolvedAbuses': {'description': 'Abuse contacts related to any alert source', 'type': <function simplify.<locals>.<lambda>>}, 'SourceResolvedASN': {'description': 'AS numbers related to any alert source', 'type': <function simplify.<locals>.<lambda>>}, 'SourceResolvedCountry': {'description': 'Coutry ISO codes related to any alert source', 'type': <function simplify.<locals>.<lambda>>}, 'StorageTime': {'description': 'Unix timestamp of the moment the message was stored into database', 'type': <function Timestamp>}}
mentat.idea.jsondict.Net4(val)[source]

Conversion of IDEA Timestamp datatype into appropriate string representation.

mentat.idea.jsondict.Net6(val)[source]

Conversion of IDEA Timestamp datatype into appropriate string representation.

class mentat.idea.jsondict.NodeDict(init_data=None)[source]

Bases: TypedDict

Typed dictionary representing Node substructure in IDEA message structure.

allow_unknown = True
typedef = {'AggrWin': {'description': 'The size of the aggregation window, if applicable.', 'type': <function Duration>}, 'Name': {'description': 'Name of the detector, which must be reasonably unique, however still bear some meaningful sense. Usually denotes hierarchy of organisational units which detector belongs to and its own name.', 'type': <function NSID>}, 'Note': {'description': 'Free text human readable additional description.', 'type': <class 'str'>}, 'SW': {'description': 'List of the names of the detection software (optionally including version). For example "labrea-2.5-stable-1" or "HP TippingPoint 7500NX".', 'type': <function simplify.<locals>.<lambda>>}, 'Type': {'description': 'List of tags, describing various facets of the detector.', 'type': <function simplify.<locals>.<lambda>>}}
class mentat.idea.jsondict.SourceTargetDict(init_data=None)[source]

Bases: TypedDict

Typed dictionary representing Source and Target substructures in IDEA message structure.

allow_unknown = True
typedef = {'ASN': {'description': 'List of autonomous system numbers of this source/target.', 'type': <function simplify.<locals>.<lambda>>}, 'Anonymised': {'description': 'Establishes whether this source/target is willingly incomplete.', 'type': <class 'bool'>}, 'AttachHand': {'description': 'List of identifiers of attachments related to this source/target - contain "Handle"s of related attachments.', 'type': <function simplify.<locals>.<lambda>>}, 'Email': {'description': 'List of email address (for example Reply-To address in phishing message). Should be formatted according to [[http://tools.ietf.org/html/rfc5322#section-3.4|RFC 5322, section 3.4]] and related, however may not conform exactly, because values, extracted from logs, messages, DNS, etc. may themselves be malformed.', 'type': <function simplify.<locals>.<lambda>>}, 'Hostname': {'description': 'List of hostnames of this source/target. Should be FQDN, but may not conform exactly, because values, extracted from logs, messages, DNS, etc. may themselves be malformed. Empty array can be used to explicitly indicate that value has been inquired and not found (missing DNS name).', 'type': <function simplify.<locals>.<lambda>>}, 'IP4': {'description': 'List of IPv4 addresses of this source/target.', 'type': <function simplify.<locals>.<lambda>>}, 'IP6': {'description': 'List of IPv6 addresses of this source/target.', 'type': <function simplify.<locals>.<lambda>>}, 'Imprecise': {'description': 'Establishes whether this source/target is knowingly imprecise.', 'type': <class 'bool'>}, 'MAC': {'description': 'List of MAC addresses of this source/target.', 'type': <function simplify.<locals>.<lambda>>}, 'Netname': {'description': 'List of RIR database reference network identifier (for example "ripe:CESNET-BB2" or "arin:WETEMAA"). Common network identifiers are: ripe, arin, apnic, lacnic, afrinic. Empty array can be used to explicitly indicate that value has been inquired and not found (IP address from unassigned block).', 'type': <function simplify.<locals>.<lambda>>}, 'Note': {'description': 'Free text human readable additional note.', 'type': <class 'str'>}, 'Port': {'description': 'List of source or destination ports affected.', 'type': <function simplify.<locals>.<lambda>>}, 'Proto': {'description': 'List of protocols, concerning connections from/to this source/target.', 'type': <function simplify.<locals>.<lambda>>}, 'Ref': {'description': 'List of references to known sources, related to attack and/or vulnerability, specific to this source/target. May be URL of the additional info, or URN (according to [[http://tools.ietf.org/html/rfc2141|RFC 2141]]) in registered namespace ([[http://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml|IANA]]) or unregistered ad-hoc namespace bearing reasonable information value and uniqueness, such as "urn:cve:CVE-2013-2266".', 'type': <function simplify.<locals>.<lambda>>}, 'Router': {'description': 'List of router/interface path information. Intentionally organisation specific, router identifiers have usually no clear meaning outside organisational unit.', 'type': <function simplify.<locals>.<lambda>>}, 'Spoofed': {'description': 'Establishes whether this source/target is forged.', 'type': <class 'bool'>}, 'Type': {'description': 'List of source/target categories.', 'type': <function simplify.<locals>.<lambda>>}, 'URL': {'description': 'List of Unified Resource Locator of this source/target. Should be formatted according to [[http://tools.ietf.org/html/rfc1738|RFC 1738]], [[http://tools.ietf.org/html/rfc1808|RFC 1808]] and related, however may not conform exactly, because values, extracted from logs, messages, etc. may themselves be malformed.', 'type': <function simplify.<locals>.<lambda>>}}
mentat.idea.jsondict.Timestamp(val)[source]

Conversion of IDEA Timestamp datatype into appropriate string representation.

mentat.idea.jsondict.simplify(obj)[source]

Simplification method from TypedList to list and from TypedDict to dict.

This simple wrapper just returns the internal .data attribute after the successfull conversion instead of the object itself.