events

This pluggable module provides features related to IDEA event database. It enables users to execute custom queries via provided search form, browse the search results in paginated table, display the details of the events or download them locally to their own computers as JSON or YAML files. Finally a dashboard panel provides statistical overview of the overall event processing according to various aggregations.

Provided endpoints

/events/search
/api/events/search
/events/<item_id>/show
/api/events/<item_id>/show
/events/dashboard
/api/events/dashboard
/api/events/metadata
/events/<item_id>/download

Endpoint enabling users to download given IDEA event as JSON file.

  • Authentication: login required

  • Authorization: any role

  • Methods: GET

Features

Event detail

Relevant endpoints:

/events/<item_id>/show

  • Authentication: login required

  • Authorization: any role

  • Methods: GET

/api/events/<item_id>/show

Menu integration: none

This view provides detailed information about particular event. Following figure describes current state of the interface:

Hawat: Events - Event detail

Hawat: Events - Event detail

There are following features highlighted in the figure above:

  1. Event identification

At the top of the page there is an identification of the event composed of following data fields when available:

Description

Textual description of the event as provided by the detector. It is replaced with unique event ID when not available. (ref:IDEA)

ID

Unique event identifier as provided by the detector. (ref:IDEA)

Note

Additional information to the event as provided by the detector. Shown only when available. (ref:IDEA)

  1. Item actions

This toolbar contains actions available to the current user. Number of actions available depends on the user`s privileges.

Download

Download event as raw JSON file.

  1. Data panels

Content of the event is organized into three separate data panels.

  1. Context dropdown action menu

Many values displayed on the page have small context dropdown action menu buttons placed next to them. These menus contain list of actions that are available for that particular value/object type. These actions are provided for convenience and they are ussually queries for that value/object into different system module. For example context dropdown action menu for IP address may contain search actions for that same address in Events, Whois or GeoIP modules.

  1. Object additional data services

Some values/object types may be enriched with information from additional data services. Each IP address found within the event may be enriched with information from Whois, GeoIP, NERD, DNS or PassiveDNS modules. Due to the performance reasons this enrichment is performed asynchronously after the page is loaded using AJAX. There are limits for numbers of each object, that will be enriched automagically. Additional objects may be enriched on demand by clicking available button next to the value/object. Empty enrichment results are not shown.

Data panel: Properties

This data panel contains the most common and the most important event data attributes formatted nicelly into HTML structures with integrations to other system modules.

General properties

ID

Unique event identifier as provided by the detector. (ref:IDEA)

Detection time

Timestamp of the moment of detection of the event (not necessarily time of the event taking place). This timestamp is mandatory, because every detector is able to know when it detected the information - for example when line about event appeared in the logfile, or when its information source says the event was detected, or at least when it accepted the information from the source. (ref:IDEA)

The value is displayed with the timezone and localization according to the user`s preferences. These preferences may be set/changed in the user`s profile.

Creation time

Timestamp of the creation of the event. May point out delay between detection and processing of data. (ref:IDEA)

The value is displayed with the timezone and localization according to the user`s preferences. These preferences may be set/changed in the user`s profile.

Creation delay

Calculated time difference between the creation time and detection time. Shown only if creation time timestamp is present.

Event time

Deduced start of the event/attack, or just time of the event if its solitary. (ref:IDEA)

The value is displayed with the timezone and localization according to the user`s preferences. These preferences may be set/changed in the user`s profile.

Cease time

Deduced end of the event/attack. (ref:IDEA)

The value is displayed with the timezone and localization according to the user`s preferences. These preferences may be set/changed in the user`s profile.

Event duration

Calculated time difference between the cease time and event time. Shown only if both event time and cease time timestamps are present.

Aggregation window start time

Beginning of aggregation window in which event has been observed. (ref:IDEA)

The value is displayed with the timezone and localization according to the user`s preferences. These preferences may be set/changed in the user`s profile.

Aggregation window end time

End of aggregation window in which event has been observed. (ref:IDEA)

The value is displayed with the timezone and localization according to the user`s preferences. These preferences may be set/changed in the user`s profile.

Aggregation window size

Calculated size of the aggregation time window as a difference between the aggregation window start and end times. Shown only if both aggregation window start and end time timestamps are present.

Storage time

Timestamp of the moment the event was stored into the database. This data attribute is internal to Mentat system. (ref)

The value is displayed with the timezone and localization according to the user`s preferences. These preferences may be set/changed in the user`s profile.

Storage delay

Calculated time difference between the storage time and detection time. This value may point out the delay due to the processing through Warden and Mentat systems.

Categories

Category of the event. (ref:IDEA), (ref:IDEA:categories)

Event severity

Deduced severity of the event. This data attribute is internal to Mentat system. It is calculated by the classification instance of mentat-inspector.py event processing module based on built-in customizable set of rules. (ref)

Event class

Deduced classification of the event. This data attribute is internal to Mentat system. It is calculated by the classification instance of mentat-inspector.py event processing module based on built-in customizable set of rules. (ref)

Abuse groups

List of all abuse groups resolved from all source addresses found within the event. (ref)

Source countries

List of all countries resolved from all source addresses found within the event. (ref)

Source autonomous systems

List of all autonomous system numbers resolved from all source addresses found within the event. (ref)

Sources

Targets

Detectors

Data panel: YAML data

This data panel contains textual dump of the event into the YAML format. It may be used to thoroughly inspect the contents of the event and view the less common event data attributes that may not be displayed in Properties data panel.

Data panel: JSON data

This data panel contains textual dump of the event into the JSON format. It may be used to thoroughly inspect the contents of the event and view the less common event data attributes that may not be displayed in Properties data panel.

Event dashboard

Relevant endpoints:

/events/dashboard

  • Authentication: login required

  • Authorization: any role

  • Methods: GET

/api/events/dashboard

Menu integration: / Dashboards / Event

This view provides overall statistical information about IDEA events processed during particular time period. Statistical data are produced in advance by the mentat-statistician.py module for 5 minute intervals and stored into database to separate table. This view then uses these precalculated datasets to produce grand overall for given time period by aggregating all relevant sub-datasets.

There are some limitations to precalculations. For example after aggregation is made by certain event attribute only TOPx values are stored to database, the rest is is not omitted but instead aggregated into single value under subkey __REST__.

View provides three separate datasets:

  • Overall statistics

    This dataset contains data for all processed events.

  • Internal statistics

    This dataset contains data only for events from known networks. Mentat can be configured to recognize some networks as more interesting from the point of view of users and administrators of that particular installation/instance.

  • External statistics

    This dataset contains data only for events NOT from known networks.

For each of these datasets following aggregations are available:

  • Total events processed

  • Number of events per abuse

  • Number of events per analyzer

  • Number of events per ASN

  • Number of events per category

  • Number of events per category set

  • Number of events per class

  • Number of events per country

  • Number of events per detector

  • Number of events per detector software

  • Number of events per IP

  • Number of events per severity

Results are presented in the form of timeline and pie charts and tables.

From the developer perspective the data preprocessing and chart and table rendering is done using excelent D3 JavaScript library.

Web API

For general information about web API please refer to section Web API.

Following is a list of all currently available API endpoints. These endpoints provide results as JSON document instead of full HTML page.

API endpoint: show

Relevant endpoint:

/api/events/<item_id>/show

  • Authentication: login required

  • Authorization: any role

  • Methods: GET, POST

Response format

JSON document, that will be received as a response for the search, can contain following keys:

item

  • Description: This subkey is present in case requested item exists. It contains the IDEA message according to the IDEA specification.

  • Datatype: IDEA message as dictionary

item_id

  • Description: This subkey is present in case requested item exists. It contains the identifier of the message.

  • Datatype: string

search_widget_item_limit

  • Description: This subkey is always present in the response. It is intended for internal purposes.

  • Datatype: integer

status

  • Description: This subkey is present in case there were any errors in the submitted request. So in another words the presence of this subkey is an indication of failure. This subkey contains the HTTP status code of the error.

  • Datatype: integer

message

  • Description: This subkey is present in case there were any errors in the submitted request. So in another words the presence of this subkey is an indication of failure. This subkey contains the human readable message describing the error that occured.

  • Datatype: string

Example usage with curl:

$ curl -X POST -d "api_key=your%AP1_k3y" "https://.../api/events/event_id/show"

API endpoint: dashboard

Relevant endpoint:

/api/events/dashboard

  • Authentication: login required

  • Authorization: any role

  • Methods: GET, POST

The URL for web API interface is available as normal endpoint to the user of the web interface. This fact can be used to debug the queries interactively and then simply copy them to another application. One might for example start with filling in the search form in the /events/dashboard endpoint. Once you are satisfied with the result, you can simply switch the base URL to the /api/events/dashboard endpoint and you are all set.

Available query parameters:

Following parameters may be specified as standard HTTP query parameters:

dt_from

  • Description: Lower event detection time boundary

  • Datatype: Datetime in the format YYYY-MM-DD HH:MM:SS, for example 2018-01-01 00:00:00

  • Default: Previous midnight

  • Note: Default value kicks in in case the parameter is not specified, explicitly use empty value for boundless search

dt_to

  • Description: Upper event detection time boundary

  • Datatype: Datetime in the format YYYY-MM-DD HH:MM:SS, for example 2018-01-01 00:00:00

Response format

JSON document, that will be received as a response for the search, can contain following keys:

form_data

  • Description: This subkey is present in case search operation was triggered. It contains a dictionary with all query parameters described above and their appropriate processed values.

  • Datatype: dictionary

form_errors

  • Description: This subkey is present in case there were any errors in the submitted search form and the search operation could not be triggered. So in another words the presence of this subkey is an indication of search failure. This subkey contains list of all form errors as pairs of strings: name of the form field and error description. The error description is localized according to the user`s preferences.

  • Datatype: list of tuples of strings

  • Example: [["dt_from", "Not a valid datetime value"]]

statistics

  • Description: This subkey is present in case search operation was triggered. It contains the actual result of the search. Following subkeys can be found in this dictionary:

    • count - Number of statistical datasets used to calculate this result dataset

    • dt_from - Lower time boundary of the result dataset

    • dt_to - Upper time boundary of the result dataset

    • stats_external - Separate statistics calculated for events from external networks

    • stats_internal - Separate statistics calculated for events from internal networks

    • stats_overall - Overall statistics

    • timeline_cfg - Pre-calculated optimized timeline configurations

    Each of the stats_* subkeys may in turn contain following subkeys: abuses, analyzers, asns, categories, category_sets, classes, countries, detectors, detectorsws, ips, severities. Each of these subkeys represents aggregation of events by particular attribute. Additionally there are following counters: cnt_alerts and cnt_events (synonyms, total number of original events), cnt_recurring (number of recurring events) and cnt_unique (number of new unique events). Finally there is a timeline subkey, which contains eveything that was described so far, but rendered to the timeline.

  • Datatype: list of dictionaries

items_count

  • Description: This subkey is present in case search operation was triggered. It contains the number of original datasets that have been processed to produce final dataset statistics.

  • Datatype: integer

query_params

  • Description: This subkey is always present in the response. It contains processed search query parameters that the user actually explicitly specified.

  • Datatype: dictionary

  • Example: {"dt_from": "", "submit": "Search"}

time_marks

  • Description: This subkey is present in case search operation was triggered. It contains list of time marks that can be used to calculate the duration of various processing steps like queriing database, processing and rendering the result.

  • Datatype: list of lists

searched

  • Description: This subkey is present in case search operation was triggered. It is a simple indication of the successfull search operation.

  • Datatype: boolean always set to True

search_widget_item_limit

  • Description: This subkey is always present in the response. It is intended for internal purposes.

  • Datatype: integer

view_icon

  • Description: This subkey is always present in the response. It is intended for internal purposes.

  • Datatype: string

view_title

  • Description: This subkey is always present in the response. It is intended for internal purposes.

  • Datatype: string

Example usage with curl:

$ curl -X POST -d "api_key=your%AP1_k3y" "https://.../api/events/dashboard?submit=Search"

Note

Please be aware, that this endpoint provides raw statistical dataset. The Event dashboard endpoint performs some additional client-side processing with JavaScript to transform datasets provided by this endpoint into format appropriate for chart and table rendering.

API endpoint: metadata

Relevant endpoint:

/api/events/metadata

  • Authentication: login required

  • Authorization: any role

  • Methods: GET, POST

The main reason for existence of this endpoint is the ability to somehow retrieve all possible and correct values for various IDEA message attributes. These lists can be then used for example for rendering some UI widgets.

Response format

JSON document, that will be received as a response for the search, can contain following keys:

categories

  • Description: This subkey contains all possible values for IDEA message categories.

  • Datatype: list of strings

classes

  • Description: This subkey contains all possible values for IDEA message classes.

  • Datatype: list of strings

detector_types

  • Description: This subkey contains all possible values for IDEA message detector types.

  • Datatype: list of strings

detectors

  • Description: This subkey contains all possible values for IDEA message detector names.

  • Datatype: list of strings

host_types

  • Description: This subkey contains all possible values for IDEA message host types.

  • Datatype: list of strings

inspection_errs

  • Description: This subkey contains all possible values for IDEA message inspection errors.

  • Datatype: list of strings

protocols

  • Description: This subkey contains all possible values for IDEA message protocols.

  • Datatype: list of strings

severities

  • Description: This subkey contains all possible values for IDEA message severities.

  • Datatype: list of strings

source_types

  • Description: This subkey contains all possible values for IDEA message source types.

  • Datatype: list of strings

target_types

  • Description: This subkey contains all possible values for IDEA message target types.

  • Datatype: list of strings

Example usage with curl:

$ curl -X POST -d "api_key=your%AP1_k3y" "https://.../api/events/metadata"