mentat.datatype.internal module

Internal datatype library.

Library contents

Todo

Documentation needs to be finished.

Warning

Still should be considered as work in progress and alpha code.

class mentat.datatype.internal.AbuseGroup(init_data=None)[source]

Bases: abc.TypedDict

Implementation of abuse group record structure.

allow_unknown = False
network_add(network)[source]

Update abuse group network list coming from given source.

networks_update(networks)[source]

Update abuse group network list coming from given source.

typedef = {'_id': {'required': True, 'type': <class 'str'>}, 'description': {'type': <class 'str'>}, 'id': {'type': <class 'str'>}, 'managers': {'default': <class 'list'>, 'required': True, 'type': <class 'abc.DBRefUsers'>}, 'networks': {'type': <class 'abc.NetworkRecordOld'>}, 'query': {'type': <class 'abc.SavedQuery'>}, 'rep_emails_critical': {'type': <class 'abc.String'>}, 'rep_emails_high': {'type': <class 'abc.String'>}, 'rep_emails_low': {'type': <class 'abc.String'>}, 'rep_emails_medium': {'type': <class 'abc.String'>}, 'rep_filters': {'type': <class 'abc.ReportingFilter'>}, 'rep_mode': {'default': 'summary', 'required': True, 'type': <function t_reporting_mode>}, 'rep_mute': {'default': False, 'required': True, 'type': <class 'bool'>}, 'rep_redirect': {'default': True, 'required': True, 'type': <class 'bool'>}, 'source': {'default': 'negistry', 'required': True, 'type': <class 'str'>}, 'subnet': {'type': <class 'abc.String'>}, 'subnet_cache': {'default': False, 'required': True, 'type': <class 'bool'>}, 'ts': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}}
class mentat.datatype.internal.EventStat(init_data=None)[source]

Bases: abc.TypedDict

Implementation of event stat structure.

allow_unknown = True
typedef = {'_id': {'required': True, 'type': <class 'str'>}, 'count': {'required': True, 'type': <class 'int'>}, 'external': {'required': True, 'type': <function Any>}, 'internal': {'required': True, 'type': <function Any>}, 'overall': {'required': True, 'type': <function Any>}, 'ts': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'ts_from': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'ts_to': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}}
class mentat.datatype.internal.NetworkRecord(init_data=None)[source]

Bases: abc.TypedDict

Base class for all NetworkRecord structures.

allow_unknown = False
fingerprint()[source]

Return network fingerprint (concatenation of network and source).

class mentat.datatype.internal.NetworkRecordIP4(init_data=None)[source]

Bases: mentat.datatype.internal.NetworkRecord

Implementation of IPv4 network record structure.

typedef = {'_resolved_abuses_chain': {'type': <class 'typedcols.Discard'>}, 'abuse_group': {'type': <class 'str'>}, 'createtime': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'description': {'type': <class 'str'>}, 'emails': {'type': <class 'abc.String'>}, 'emails_critical': {'type': <class 'abc.String'>}, 'emails_fallback': {'type': <class 'abc.String'>}, 'emails_high': {'type': <class 'abc.String'>}, 'emails_low': {'type': <class 'abc.String'>}, 'emails_medium': {'type': <class 'abc.String'>}, 'first': {'type': <class 'int'>}, 'id': {'type': <class 'str'>}, 'ip4_end': {'type': <class 'str'>}, 'ip4_start': {'type': <class 'str'>}, 'is_base': {'type': <class 'bool'>}, 'last': {'type': <class 'int'>}, 'netname': {'type': <class 'str'>}, 'network': {'required': True, 'type': <class 'str'>}, 'nrobj': {'required': True, 'type': <function t_ip_range>}, 'rank': {'type': <class 'int'>}, 'resolved_abuses': {'type': <class 'abc.String'>}, 'source': {'required': True, 'type': <class 'str'>}, 'type': {'required': True, 'type': <function t_network_record_type_ip4>}}
class mentat.datatype.internal.NetworkRecordIP6(init_data=None)[source]

Bases: mentat.datatype.internal.NetworkRecord

Implementation of IPv6 network record structure.

typedef = {'_resolved_abuses_chain': {'type': <class 'typedcols.Discard'>}, 'abuse_group': {'type': <class 'str'>}, 'createtime': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'description': {'type': <class 'str'>}, 'emails': {'type': <class 'abc.String'>}, 'emails_critical': {'type': <class 'abc.String'>}, 'emails_fallback': {'type': <class 'abc.String'>}, 'emails_high': {'type': <class 'abc.String'>}, 'emails_low': {'type': <class 'abc.String'>}, 'emails_medium': {'type': <class 'abc.String'>}, 'first': {'type': <class 'int'>}, 'id': {'type': <class 'str'>}, 'ip6_addr': {'type': <class 'str'>}, 'ip6_prefix': {'type': <class 'int'>}, 'is_base': {'type': <class 'bool'>}, 'last': {'type': <class 'int'>}, 'netname': {'type': <class 'str'>}, 'network': {'required': True, 'type': <class 'str'>}, 'nrobj': {'required': True, 'type': <function t_ip_range>}, 'rank': {'type': <class 'int'>}, 'resolved_abuses': {'type': <class 'abc.String'>}, 'source': {'required': True, 'type': <class 'str'>}, 'type': {'required': True, 'type': <function t_network_record_type_ip6>}}
class mentat.datatype.internal.Report(init_data=None)[source]

Bases: abc.TypedDict

Implementation of report structure.

allow_unknown = True
typedef = {'_id': {'required': True, 'type': <class 'str'>}, 'abuse': {'required': True, 'type': <class 'str'>}, 'analyzers': {'required': True, 'type': <function Any>}, 'categories': {'required': True, 'type': <function Any>}, 'category_sets': {'required': True, 'type': <function Any>}, 'cnt_alerts': {'required': True, 'type': <class 'int'>}, 'cnt_all': {'required': True, 'type': <class 'int'>}, 'cnt_analyzers': {'required': True, 'type': <class 'int'>}, 'cnt_categories': {'required': True, 'type': <class 'int'>}, 'cnt_category_sets': {'required': True, 'type': <class 'int'>}, 'cnt_detectors': {'required': True, 'type': <class 'int'>}, 'cnt_detectorsws': {'required': True, 'type': <class 'int'>}, 'cnt_flt': {'required': True, 'type': <class 'int'>}, 'cnt_flt_blk': {'required': True, 'type': <class 'int'>}, 'cnt_ips': {'required': True, 'type': <class 'int'>}, 'cnt_rlp': {'required': True, 'type': <class 'int'>}, 'cnt_thr': {'required': True, 'type': <class 'int'>}, 'cnt_thr_blk': {'required': True, 'type': <class 'int'>}, 'detectors': {'required': True, 'type': <function Any>}, 'detectorsws': {'required': True, 'type': <function Any>}, 'flag_archived': {'required': True, 'type': <class 'bool'>}, 'flag_jarchived': {'required': True, 'type': <class 'bool'>}, 'flag_mail_sent': {'required': True, 'type': <class 'bool'>}, 'frv': {'required': True, 'type': <function Any>}, 'id': {'required': True, 'type': <class 'str'>}, 'ips': {'required': True, 'type': <function Any>}, 'list_analyzers': {'required': True, 'type': <class 'abc.String'>}, 'list_categories': {'required': True, 'type': <class 'abc.String'>}, 'list_category_sets': {'required': True, 'type': <class 'abc.String'>}, 'list_detectors': {'required': True, 'type': <class 'abc.String'>}, 'list_detectorsws': {'required': True, 'type': <class 'abc.String'>}, 'list_ids': {'required': True, 'type': <class 'abc.String'>}, 'list_ips': {'required': True, 'type': <class 'abc.String'>}, 'mail_res': {'required': True, 'type': <class 'str'>}, 'mail_to': {'required': True, 'type': <class 'str'>}, 'mail_ts': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'message': {'required': True, 'type': <class 'str'>}, 'severity': {'required': True, 'type': <class 'str'>}, 'test_data': {'required': True, 'type': <class 'bool'>}, 'to': {'required': True, 'type': <class 'str'>}, 'ts': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'ts_from': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'ts_to': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'type': {'required': True, 'type': <class 'str'>}, 'ua_hash': {'required': True, 'type': <class 'str'>}}
class mentat.datatype.internal.ReportingFilter(init_data=None)[source]

Bases: abc.TypedDict

Implementation of reporting filter record structure.

typedef = {'_id': {'default': <function gen_sid>, 'required': True, 'type': <class 'str'>}, 'analyzers': {'type': <class 'abc.String'>}, 'categories': {'type': <class 'abc.String'>}, 'description': {'required': True, 'type': <class 'str'>}, 'enabled': {'type': <class 'bool'>}, 'filter': {'required': True, 'type': <class 'str'>}, 'hits': {'type': <class 'int'>}, 'ips': {'type': <class 'abc.String'>}, 'note': {'type': <class 'str'>}, 'ts': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'type': {'required': True, 'type': <function t_filter_type>}, 'validfrom': {'type': <function t_datetime>}, 'validto': {'type': <function t_datetime>}}
class mentat.datatype.internal.User(init_data=None)[source]

Bases: abc.TypedDict

Implementation of abuse group record structure.

allow_unknown = False
typedef = {'_id': {'required': True, 'type': <class 'str'>}, 'affiliations': {'default': <class 'list'>, 'type': <class 'abc.String'>}, 'certificate': {'type': <class 'str'>}, 'certificate_hash': {'type': <class 'str'>}, 'email': {'required': True, 'type': <class 'str'>}, 'groups': {'default': <class 'list'>, 'required': True, 'type': <class 'abc.DBRefGroups'>}, 'id': {'type': <class 'str'>}, 'name': {'required': True, 'type': <class 'str'>}, 'organization': {'required': True, 'type': <class 'str'>}, 'orggroups': {'default': <class 'list'>, 'type': <class 'abc.String'>}, 'query': {'default': <class 'list'>, 'type': <class 'abc.SavedQuery'>}, 'roles': {'default': <class 'list'>, 'required': True, 'type': <class 'abc.String'>}, 'ts': {'default': <built-in function time>, 'required': True, 'type': <function t_datetime>}, 'ts_last_login': {'type': <function t_datetime>}}
mentat.datatype.internal.gen_sid()[source]

Generate random unique subidentifier for NetworkRecord.

Returns

Unique identifier 8 characters long

Return type

str

mentat.datatype.internal.list_factory(name, item_type)[source]

Default implementation of list factory.

mentat.datatype.internal.list_types(flavour, cb_list_factory=None)[source]

Generate list type flavours from given scalar flavour definitions.

Parameters

  • flavour (dict) – Type flavour definitions

  • list_factory (callable or None) – List factory callable

Returns

list flavour

Return type

dict

mentat.datatype.internal.t_datetime(val)[source]

Convert/validate: Datetime.

Parameters

val (any) – Value to be converted/validated

Returns

Datetime object

Return type

datetime.datetime

Raises

ValueError – if the value could not be converted to datetime.datetime object

mentat.datatype.internal.t_dbref(val)[source]

Convert/validate: Database reference.

mentat.datatype.internal.t_filter_type(val)[source]

Convert/validate: Reporting filter type.

mentat.datatype.internal.t_ip_range(val)[source]

Convert/validate: IP range.

Parameters

val (any) – Value to be converted/validated

Returns

IP range

Return type

str

mentat.datatype.internal.t_net(val)[source]

Convert/validate: Convert anything to ipranges IPvX object class.

Parameters

val (any) – Value to be converted/validated

Returns

Object representing IPvX address/network/range

Return type

ipranges.IP4Net or ipranges.IP4Range or ipranges.IP4 or ipranges.IP6Net or ipranges.IP6Range or ipranges.IP6

Raises

ValueError – if the value could not be converted to ipranges object

mentat.datatype.internal.t_net4(val)[source]

Convert/validate: Convert anything to ipranges IPv4 object class.

Parameters

val (any) – Value to be converted/validated

Returns

Object representing IPv4 address/network/range

Return type

ipranges.IP4Net or ipranges.IP4Range or ipranges.IP4

Raises

ValueError – if the value could not be converted to ipranges object

mentat.datatype.internal.t_net6(val)[source]

Convert/validate: Convert anything to ipranges IPv6 object class.

Parameters

val (any) – Value to be converted/validated

Returns

Object representing IPv6 address/network/range

Return type

ipranges.IP6Net or ipranges.IP6Range or ipranges.IP6

Raises

ValueError – if the value could not be converted to ipranges object

mentat.datatype.internal.t_network_record(val, source=None)[source]

Convert/validate: Network record.

Parameters

val (any) – Value to be converted/validated

Returns

network record object.

Return type

NetworkRecord

Raises

ValueError – if the value is not valid network record

mentat.datatype.internal.t_network_record_old(val, source=None)[source]

Dummy convertor for legacy purposes.

mentat.datatype.internal.t_network_record_type_ip4(val)[source]

Convert/validate: Network record type.

Parameters

val (any) – Value to be converted/validated

Returns

network record type.

Return type

str

Raises

ValueError – if the value is not valid network record type

mentat.datatype.internal.t_network_record_type_ip6(val)[source]

Convert/validate: Network record type.

Parameters

val (any) – Value to be converted/validated

Returns

network record type.

Return type

str

Raises

ValueError – if the value is not valid network record type

mentat.datatype.internal.t_reporting_filter(val)[source]

Convert/validate: Reporting filter.

mentat.datatype.internal.t_reporting_mode(val)[source]

Convert/validate: Reporting mode.

mentat.datatype.internal.t_saved_query(val)[source]

Convert/validate: Saved query.

mentat.datatype.internal.to_net4(val)[source]

Convert any given value to ipranges.IP4Range.

Parameters

flavour (any) – Value to be converted

Returns

Converted value

Return type

ipranges.IP4Range

mentat.datatype.internal.to_net6(val)[source]

Convert any given value to ipranges.IP6Net.

Parameters

flavour (any) – Value to be converted

Returns

Converted value

Return type

ipranges.IP6Net

mentat.datatype.internal.typedef_abuse_group(flavour, list_flavour, addon=None)[source]

Typedef generator for abuse group records.

mentat.datatype.internal.typedef_event_stat(flavour, list_flavour, addon=None)[source]

Typedef generator for event statistics records.

mentat.datatype.internal.typedef_filter(flavour, list_flavour, addon=None)[source]

Typedef generator reporting filter records.

mentat.datatype.internal.typedef_network_record_ip4(flavour, list_flavour, addon=None)[source]

Typedef generator for IPv4 network records.

mentat.datatype.internal.typedef_network_record_ip6(flavour, list_flavour, addon=None)[source]

Typedef generator for IPv6 network records.

mentat.datatype.internal.typedef_report(flavour, list_flavour, addon=None)[source]

Typedef generator for report records.

mentat.datatype.internal.typedef_user(flavour, list_flavour, addon=None)[source]

Typedef generator for user records.