mentat-netmngr.py

This Mentat module is a script providing functions for abuse group network management for Mentat system database.

This script is implemented using the pyzenkit.zenscript framework and so it provides all of its core features. See the documentation for more in-depth details.

Note

Still work in progress, use with caution.

Usage examples

# Display help message and exit.
mentat-netmngr.py --help

# Run in debug mode (enable output of debugging information to terminal).
mentat-netmngr.py --debug

# Run with increased logging level.
mentat-netmngr.py --log-level debug

Available script commands

status (default)
Detect and display the state of internal whois database contents according to the data in given reference whois file.
update
Attempt to update the state of internal whois database contents according to the data in given reference whois file.
convert-exceptions
Attempt to convert given list of exception files into valid whois file.

Custom configuration

Custom command line options

--whois-file file-path

Path to reference whois file containing network data.

Type: string, default: None

Custom config file options

exception_files

List of paths to exception files and their appropriate abuse groups.

Example configuration:

"exception_files": [
    {
        "path":  "/path/to/file",
        "abuse": "abuse_group_id"
    }
],

Type: list of dicts, default: None

Exception file format

The exception file is an ordinary text file containing single IPv(4|6) address|network|range on each line. Blank lines and lines beginning with # are ignored. Whois exception files are very easy to be generated and they are meant for specifiing whois resolving exceptions. For example you may use it to describe hosts with addresses from the domain of one particular abuse group, but actualy belonging to different group. This might be the case of routers belonging to service provider but residing within the network address space of the customer. Another example may be nodes of some cloud computing service that have addresses from address space of the cloud computing organization member.

Whois file format

Whois file is an ordinary text file containg whois information in specific structured way. It is recognized by the mentat.services.whois.FileWhoisModule and can be used for whois resolving.

The structure of the data comes from the export format of CESNET’s Negistry tool, which is an internal custom copy of relevant RIPE whois data. It is JSON based format. Following example content describes multiple valid syntaxes for describing network records:

{
    "__whois_about__" "Generated by Custom tool(tm) at 2018-01-25 01:00:01.765858.",
    "__whois_type__": "custom",

    # Option 1: Pass IP4 start and end addresses
    "78.128.128.0 - 78.128.255.255": {
        "ip4_start": "78.128.128.0",
        "ip4_end": "78.128.255.255",
        "netname": "CZ-TEN-34-20070410",
        "resolved_abuses": [
            "abuse@cesnet.cz"
        ]
    },

    # Option 2: Pass network CIDR or range and type
    "78.128.212.64 - 78.128.212.127": {
        "network": "78.128.212.64/26",
        "type": "ipv4",
        "netname": "CESNET-HSM4",
        "descr": [
          "CESNET, z.s.p.o.",
          "Ostrava"
        ],
        "resolved_abuses": [
          "abuse@cesnet.cz"
        ]
    },

    # Option 3: Pass IP6 address and prefix
    "2001:718::/29": {
        "ip6_addr": "2001:718::",
        "ip6_prefix": 29,
        "netname": "CZ-TEN-34-20010521",
        "description": "Extensive network description",
        "resolved_abuses": [
            "abuse@cesnet.cz"
        ]
    },

    # Option 4: Pass only IPv(4|6) network|range without type for autodetection (slower)
    "2001:718::/29": {
        "network": "2001:718::/29",
        "netname": "CZ-TEN-34-20010521",
        "resolved_abuses": [
          "abuse@cesnet.cz"
        ]
    },
    ...
}

The __whois_type__ meta attribute has a deeper meaning. It is used throughout the database records to mark their origin. It is possible to keep track which tool provided which whois network records.

The netname, descr and description attributes are optional and will be used/stored into database, if present.

The resolved_abuses attribute is mandatory and must contain list of abuse groups (abuse contacts) for that particular network record.

For the purposes of cooperation with the Negistry tool following header is also recognized and automatically sets up the __whois_type__ header to value negistry:

{
    "__negistry_about__": "Generated by Negistry at 2018-01-25 01:00:01.765858.",
    ...
}

Configuration

The application supports multiple means for adjusting the internal configurations. When appropriate the default values for each configuration is hardcoded in module source code. However there are several options to change the value:

  • Override the internal default value when instantinating the application object by passing different value to object constructor.
  • Pass the different value by configuration file.
  • Pass the different value by command line option.

The configuration values are assigned from the sources mentioned above in that particular order, so the value given by command line option overwrites the value written in configuration file.

Command line options

Configuration can be passed down to application by command line options. These options have the highest priority and will overwrite any other configuration values. Depending on the base object of the application different set of options is available.

Common application options

Following configuration options are available for all applications based on pyzenkit.baseapp:

--help
Display help and usage description and exit (flag)
--name alternative-name

Alternative name for application instead of default $0, using which names for log, runlog, pid, status and other files will be generated.

Type: string, default: $0

--quiet

Run in quiet mode (flag).

Do not write anything to stdout or stderr.

Type: boolean, default: False

--verbose

Increase application output verbosity (flag, repeatable).

Type: boolean, default: False

--log-file file-name

Name of the log file.

Type: string, default: autodetected

--log-level level

Logging level [debug, info, warning, error, critical].

Type: string, default: info

--runlog-dir dir-name

Name of the runlog directory.

Type: string, default: autodetected

--runlog-dump

Dump runlog to stdout when done processing (flag).

Type: boolean, default: False

--runlog_log

Write runlog to logging service when done processing (flag)

Type: boolean, default: False

--pstate_file file-name

Name of the persistent state file.

Type: string, default: autodetected

--pstate_dump

Dump persistent state to stdout when done processing (flag).

Type: boolean, default: False

--pstate_log

Write persistent state to logging service when done processing (flag).

Type: boolean, default: False

--action action

Execute given quick action and exit. List of available actions can be displayed with --help option.

Type: string, default: None

--user name-or-id

Name/gid of the system user for process permissions.

Type: string, default: None

--group name-or-id

Name/gid of the system group for process permissions.

Type: string, default: None

Common script options

Following configuration options are available on top of common applicationsoptions for all applications based on pyzenkit.zenscript:

--regular

Operational mode: regular script execution (flag). Conflicts with --shell option.

Type: boolean, default: False

--shell

Operational mode: manual script execution from shell (flag). Conflicts with --regular option.

Type: boolean, default: False

--command name

Name of the script command to be executed.

Type: string, default: autodetected

--interval interval

Execution interval. This value should correspond with related cron script.

Type: string, default: daily

--adjust_thresholds

Round-up time interval threshols to interval size (flag).

Type: boolean, default: False

--time_high time

Upper time interval threshold.

Type: float, default: time.time

Configuration files and directories

Configuration can be passed down to application using a combination of configuration file or configuration directory. The configuration file

The available configuration keys are very similar to command line options and the names differ only in the use of _ character instead of -. However there is a certain set of configuration keys that is available only through command line options and not through configuration file and vice versa.