Installation

The Mentat system is available as native Python package from PyPI. However there is only so much you can do with setuptools, so to make the installation more convenient for users we are also providing package for Debian based systems via our custom internal repository. We are trying hard to keep the installation process as simple as possible, however at the same time we try not to force too many decisions on the users of the system. For that reason the Debian package for Mentat system does not pull and automatically install all required dependencies, but you will have to install couple of additional Debian system packages and/or Python libraries manually according to your setup and preferences.

Note

As an example try to compare version of PostgreSQL database that is available via official Debian repositories with version available via repository operated by PostgreSQL development team. That is the reason why we did not add that directly to the list of Debian package dependencies.

For your convenience there is a section dedicated to installation of prerequisites:

  1. Installation guide - Prerequisites

To sum it up currently you have following installation options:

  1. Installation guide - Manual installation

  2. Installation guide - Installation from Git repository

  3. Installation guide - Ansible roles

Note

We are using HTTPS to serve our internal Debian package repository. You need to have the apt-transport-https package installed on your target Debian host to be able to access it.

Dependencies

System dependencies

Following is a list of most important system dependencies:

BerkeleyDB

BerkeleyDB (sometimes referred to as simply “BDB”) is an embedded open-source, database storage library. The simplicity arises from the fact that it is a basic key-value store and not a full-fledged database system that provides querying and schema constraints. It offers flexibility by being schema-less and by providing convenient mechanisms to include and discard features.

In Mentat system it is used for caching various data.

This dependency will be istalled for you automatically by Debian packaging system, because the package version in Debian repository is good enough for our needs.

PostgreSQL

PostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness. It runs on all major operating systems, including Linux, UNIX (AIX, BSD, HP-UX, macOS, Solaris), and Windows. It is fully ACID compliant, has full support for foreign keys, joins, views, triggers, and stored procedures (in multiple languages). It includes most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP. It also supports storage of binary large objects, including pictures, sounds, or video. It has native programming interfaces for C/C++, Java, .Net, Perl, Python, Ruby, Tcl, ODBC, among others, and exceptional documentation.

In Mentat system it is used as a backend service for persistent data storage.

You have to install this dependency manually according to your preferences. Alternativelly you can use prepared minimalistic Ansible role postgresql (see section Installation guide - Ansible roles below for more details).

Additionally, it is necessary to install the IP4R PostgreSQL extension to enable support for better handling of mixed IP addresses and ranges.

You may also refer to section Install - PostgreSQL 13 below for more precise installation instructions.

RRDTool

The OpenSource industry standard, high performance data logging and graphing system for time series data.

In Mentat system it is used for storing and displaying continuous system processing performance statistics.

This dependency will be istalled for you automatically by Debian packaging system, because the package version in Debian repository is good enough for our needs.

MaxMind Geolite2 databases

GeoLite2 databases are free IP geolocation databases comparable to, but less accurate than, MaxMind’s GeoIP2 databases. The GeoLite2 Country, City, and ASN databases are updated weekly, every Tuesday.

In Mentat system these databases are used for IDEA event enrichment with IP geolocation data.

Please follow official documentation to install and configure the geoipupdate utility, use it to download latest version of free City, Country and ASN databases and setup cron job to periodically update local database files.

You may also refer to section Install - geoipupdate below for more precise installation instructions.

Python dependencies

Following is a list of most important and/or notable Python dependencies:

geoip2

Python binding for MaxMind GeoIP2 API.

Jinja2

A small but fast and easy to use stand-alone template engine written in pure Python.

Babel

A collection of tools for internationalizing Python applications.

WTForms

WTForms is a flexible forms validation and rendering library for Python web development.

ply

Python Lex & Yacc.

psycopg2

Python driver for PostgreSQL.

SQLAlchemy

Python universal DBAL and ORM toolkit.

alembic

Database migration tool.

rrdtool

Python bindings for RRDTool.

Flask

Awesome Python microframework for web development.

ipranges

Python library for working with IP addressess.

typedcols

Python library providing typed collections.

idea-format

Python library for working with IDEA messages.

Pydgets

Python3 command line widget library.

PyZenKit

Collection of usefull tools and utilities for creating console applications, scripts and system services (daemons) in Python 3.

Pynspect

Python library for filtering, querying or inspecting almost arbitrary data structures.

When installing from our Debian packages, all these dependencies are installed for you automatically into a separate Python virtual environment (we try hard not to mess up your host system). In case you need to install/reinstall them manually, you can use pip3 and prepared Python requirements file /etc/mentat/requirements.pip:

$ . /var/mentat/venv/bin/activate
(venv) $ pip3 install -r /etc/mentat/requirements.pip
(venv) $ deactivate

Installation guide - Prerequisites

This section is included here solely for your convenience. Please be aware, that official installation instructions in the official documentation is always best source of information. Please refer to that documentation in case you encounter any errors.

Install - pip

# Run these commands as root:
$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ python3 get-pip.py

Install - PostgreSQL 13

# Run these commands as root:
$ echo "deb http://apt.postgresql.org/pub/repos/apt/ buster-pgdg main" | tee /etc/apt/sources.list.d/pgdg.list
$ wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
$ apt update
$ apt install postgresql-13 postgresql-13-ip4r postgresql-client-13 postgresql-server-dev-13 libpq-dev pgadmin4

Install - Yarn

# Run these commands as root:
$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
$ apt update
$ apt install build-essential nodejs npm yarn

Install - Grunt

# Run these commands as root:
$ npm install -g grunt-cli

Install - geoipupdate

First please sign up for free account to enable downloading of free Geolite2 database files. Then generate access key with support for GeoIP Update protocol, you will be asked for this during the key creation process. You will also be given option to download configuration file for the geoipupdate utility with you account configuration details already prefilled.

Next find appropriate package of geoipupdate utility for your system on official releases page and install in locally on your target system. Following commands are only examples for your convenience, please choose latest available version.

# Run these commands as root:
$ wget https://github.com/maxmind/geoipupdate/releases/download/v4.1.5/geoipupdate_4.1.5_linux_amd64.deb
$ dpkg -i geoipupdate_4.1.5_linux_amd64.deb

Now you must configure the geoipupdate utility. Default configuration file on Debian systems is located at /etc/GeoIP.conf. Either replace the default version with the one downloaded after access key creation, or tweak to contents to look similar to these:

# GeoIP.conf file for `geoipupdate` program, for versions >= 3.1.1.
# Used to update GeoIP databases from https://www.maxmind.com.
# For more information about this config file, visit the docs at
# https://dev.maxmind.com/geoip/geoipupdate/.

# `AccountID` is from your MaxMind account.
AccountID xxxxxx

# `LicenseKey` is from your MaxMind account
LicenseKey yyyyyyyyyyyyyy

# `EditionIDs` is from your MaxMind account.
EditionIDs GeoLite2-ASN GeoLite2-City GeoLite2-Country

Now download the GeoIP databases to your target host. Keep the default database directory pointed at /usr/share/GeoIP/. This path will be preconfigured as default in Mentat system.

# Run these commands as root:
$ geoipupdate --verbose

Then create new crontab file to periodically update the local database files. Let`s call it /etc/cron.d/geoipupdate with contents similar to these:

# Run every Wednesday at 6am.
0 6 * * 3   root /usr/bin/geoipupdate

Installation guide - Manual installation

The installation of base system is pretty straightforward, please follow these simple steps (all commands should be executed as root):

# Step 1: Install pip Python package manager. See official documentation, or refer
# to section `Installation guide - Prerequisites` above.

# Step 2: Install and configure PostgreSQL database. See official documentation,
# or refer to section `Installation guide - Prerequisites` above.

# Step 3: Download apt list file for mentat production package repository:
$ wget -O /etc/apt/sources.list.d/mentat-production.list https://alchemist.cesnet.cz/mentat/deb/mentat.list

# Step 4: Download and install GnuPG signing key for mentat production package repository:
$ wget -O - https://alchemist.cesnet.cz/mentat/deb/mentat-production.gpg | apt-key add -

# Step 5: Refresh apt cache:
$ apt update

# Step 6: List all packages available in newly added repository (optional):
$ grep "Package:" /var/lib/apt/lists/alchemist.cesnet.cz_*mentat*_Packages | uniq

# Step 7: Install the Mentat system:
$ apt install mentat-ng

After these steps there are several post-installation tasks to be performed:

# To be safe activate the Python virtual environment:
$ . /var/mentat/venv/bin/activate

# Step 8: Initialize PostgreSQL database schema:
(venv) $ mentat-dbmngr.py --command init

# Step 9: Stamp both metadata end event databases with latest migration revisions:
(venv) $ hawat-cli db stamp head
(venv) $ /etc/mentat/scripts/sqldb-migrate.sh stamp head

# Step 10: Precache various event search form select option dictionaries:
(venv) $ mentat-precache.py --allow-empty

# Step 11: Optimize PostgreSQL settings for used databases (optional).
# Please refer to 'Database' documentation section fro more deatils.
(venv) $ /etc/mentat/scripts/sqldb-optimize.sh

# Deactivate now unnecessary virtual environment:
(venv) $ deactivate

The Mentat system is now successfully installed. You may try to start everything up:

$ mentat-controller.py
$ mentat-controller.py --command start
$ hawat-cli run --host localhost --port 5000

The hawat-cli run command in the example above launches built-in development web server providing the Hawat web interface. It should be used only for demonstration or development purposes and not accessible from the outside world. You must use a production level webserver like Apache2 for serving the web interface, please refer to section Installation guide - Web interface with Apache2 below for more details.

Installation guide - Installation from Git repository

This type of installation is intended either for developers, that want to be able to execute the application locally, or for deploying the application for testing/debugging/demonstration purposes. Prior to installation you must provide following prerequisites on the target host system:

The installation is pretty straightforward, please follow these simple steps:

# Step 0: Install required Debian packages
$ sudo apt install sudo adduser make build-essential python3 python3-venv libpython3-dev libpq-dev rrdtool librrd-dev libdb5.3 libdb5.3-dev nodejs npm

# Step 1: Install pip Python package manager. See official documentation, or refer
# to section `Installation guide - Prerequisites` above.

# Step 2: Install and configure PostgreSQL database. See official documentation,
# or refer to section `Installation guide - Prerequisites` above.

# Step 3: Install Node.JS, npm and Yarn. See official documentation, or refer
# to section `Installation guide - Prerequisites` above.

# Step 4: Install Grunt task manager. See official documentation, or refer to
# section `Installation guide - Prerequisites` above.

# Step 5: Clone the git repository. Please note, that following command uses
# HTTPS and you will therefore have read-only access to the repository. In
# case you will be working and contributing code, you must either send patches
# via email, or be granted read-write access to our repository. In that case
# the repository URL will be something like [username]@homeproj.cesnet.cz:mentat-ng,
# please adjust following command accordingly.
$ git clone --recurse-submodules https://homeproj.cesnet.cz/git/mentat-ng.git/ mentat-ng
$ cd mentat-ng

# Step 6: Switch to development branch (optional, depending on the use case)
$ git checkout devel

# Step 7: Check for presence of all prerequisites:
$ make deps-prerequisites

# Step 8: Bootstrap Python virtual environment for development:
$ make venv

# Step 9: Activate virtual environment before any further work:
$ . venv/bin/activate

# Step 10: Perform the installation magic:
(venv) $ make develop

After these steps there are several post-installation tasks to be performed:

# Do not forget to activate the virtual environment:
$ . venv/bin/activate

# Step 11: Initialize PostgreSQL database schema:
(venv) $ mentat-dbmngr.py --command init

# Step 12: Stamp both metadata end event databases with latest migration revisions:
(venv) $ hawat-cli db stamp head
(venv) $ ./scripts/sqldb-migrate.sh stamp head

# Step 13: Precache various event search form select option dictionaries:
(venv) $ mentat-precache.py --allow-empty

# Step 14: Optimize PostgreSQL settings for used databases (optional).
# Please refer to 'Database' documentation section fro more deatils.
(venv) $ /etc/mentat/scripts/sqldb-optimize.sh

The Mentat system is now successfully installed. You may try to start everything up:

# Do not forget to activate the virtual environment:
$ . venv/bin/activate

(venv) $ mentat-controller.py
(venv) $ mentat-controller.py --command start
(venv) $ hawat-cli run --host localhost --port 5000

The hawat-cli run command in the example above launches built-in development web server providing the Hawat web interface. It should be used only for demonstration or development purposes and not accessible from the outside world. You must use a production level webserver like Apache2 for serving the web interface, please refer to section Installation guide - Web interface with Apache2 below for more details.

The whole application uses the ./chroot subdirectory of current working directory as a kind of lightweight chroot jail and places all runtime files inside it.

Please also note the existence of .env configuration file, that was created for you during the installation. This file contains various local custom configurations, that are loaded using python-dotenv.

Installation guide - Ansible roles

Note

Please do not use Ansible roles for installation during migration from previous stable version of Mentat based on Perl and MongoDB. Some of the configuration files changed and you may encounter weird errors in case original configuration stays active. Part of the migration process is assessing changes in configuration files and this process is not automated and needs to be done by hand.

If you are fond of Ansible, there is great news for you: we have prepared Ansible roles to make the installation as simple as possible. These roles are available via official Ansible Galaxy site. They can be found easily using the galaxy search form. For the sake of completeness this is a short guide how to quickly use them, but you can use them as usual in your own role ecosystem.

First there is a basic role called mentat. This role will perform only basic Mentat installation. It covers steps 1 through 11 of manual installation with some additional actions like starting Mentat system and enabling it at system boot. Additionally it will install the PostgreSQL database for you through native dependency on postgresql role:

# Step 1: Create working directory:
mkdir ansible-mentat && cd ansible-mentat && mkdir roles

# Step 2: Download the roles:
ansible-galaxy install honzamach.mentat --roles-path=./roles

# Step 3: Pull the Mentat playbook to local directory:
cp roles/honzamach.mentat/role_mentat.yml .

# Step 4: Create inventory file 'inventory' with content similar to this:
[servers-production]
your.server.hostname

[servers-mentat]
your.server.hostname

# Step 5: Execute the playbook:
ansible-playbook -i inventory role_mentat.yml

And that is it. After a lot of crunching Mentat system should be installed on your target device. Again, this playbook installs only basic system without configuration of Apache webserver or creating any demonstration accounts.

Note

Please browse through the tasks of both playbooks to known what actions are actually being done on taget host. Ansible is designed to be human readable and easily understandable, so it does not make much sense to describe set of actions for each playbook. That would only duplicate the information.

Installation guide - Web interface with Apache2

If you wish to use the web interface Hawat, you wil have to install and configure a web server. You may use any server you like as long as it supports execution of WSGI Python3 scripts. Do not forget to setup some initial user accounts, either use the demonstration fixtures, or setup real user accounts. Otherwise you will not be able to log in. Also, please note that Hawat application is not reentrant, so employed web server must allow process base dispatch (not threaded or event based).

Following examples demonstrate necessary Apache2 configuration when Mentat system is installed on target system with Debian packages. Please review and adjust the content of all listed configuration files according to your needs, they are not ready for deployment out of the box:

/etc/apache2/sites-available/site_mentat.conf

Main Mentat web interface configuration file for Apache web server.

/etc/mentat/mentat-hawat.py.conf

Main Mentat web interface configuration file.

# Step 1: Install Apache web server:
apt-get install apache2 libapache2-mod-wsgi-py3

# Step 2: Copy example Apache site configuration file and adjust it according to your needs:
cp /etc/mentat/apache/site_mentat.conf.htpasswd.example /etc/apache2/sites-available/site_mentat.conf

# Step 3: Configure Apache2 to run as mentat user in '/etc/apache2/envvars' file:
sed -i.bak s/APACHE_RUN_USER=www-data/APACHE_RUN_USER=mentat/g /etc/apache2/envvars
sed -i.bak s/APACHE_RUN_GROUP=www-data/APACHE_RUN_GROUP=mentat/g /etc/apache2/envvars
rm /etc/apache2/envvars.bak

# Step 4: Enable/disable appropriate Apache sites and modules:
a2enmod rewrite
a2enmod ssl
a2enmod wsgi
a2dissite 000-default.conf
a2dissite default-ssl.conf
a2ensite site_mentat.conf

# Step 5: Enforce process based Apache worker
a2dismod mpm_worker
a2dismod mpm_event
a2enmod mpm_prefork

# !!!!! THIS STEP IS SUPER IMPORTANT !!!!!
# Step 6: In file /etc/mentat/mentat-hawat.py.conf change default SECRET_KEY and
# in production deployment make sure, that ENABLED_BLUEPRINTS key does not
# contain 'hawat.blueprints.auth_dev', or that is is commented out. Otherwise
# you would enable anyone impersonate any other user without password.
# !!!!! THIS STEP IS SUPER IMPORTANT !!!!!
vim /etc/mentat/mentat-hawat.py.conf

# Step 7: Restart Apache service:
systemctl restart apache2.service

# Step 8: OPTION A: Insert demonstration data into Mentat`s metadata database
# (this will insert some demo accounts, groups, networks and filters into db
# and can be used in conjuction with htpasswd file above):
mentat-dbmngr.py --command fixtures-add

# Step 9: OPTION B: Create initial user account/s in Mentat`s metadata database
# (please adjust the attributes, do not simply copy and paste):
mentat-dbmngr.py --command user-add login=superman "fullname=Clark Kent" email=kent@dailyplanet.com "organization=Daily Planet, inc." roles=user,admin

Warning

Please make sure you have read step 6 in the recipe above. The auth_dev authentication module in particular is a HUGE security risk when enabled in production installation and accessible from network.

Warning

For demonstration purposes Mentat package ships with preconfigured htpasswd file containing following user accounts (login - password):

  • user - user

  • manager - manager

  • developer - developer

  • admin - admin

This can be used in conjunction with database fixtures above (step 17, option A) to immediately start exploring the Mentat system. Please make sure NOT to use it in production environment!!!

Following recipe is another example installation of Apache web server, this time with Shibboleth SSO login service (this SSO login service is used for example in Czech eduID identity federation):

# Step 1: Install Apache web server:
apt-get install apache2 libapache2-mod-wsgi-py3 libapache2-mod-shib2

# Step 2: Copy example Apache site configuration file and adjust it according to your needs:
cp /etc/mentat/apache/site_mentat.conf.shibboleth.example /etc/apache2/sites-available/site_mentat.conf

# Step 3: Configure Apache2 to run as mentat user in '/etc/apache2/envvars' file:
sed -i.bak s/APACHE_RUN_USER=www-data/APACHE_RUN_USER=mentat/g /etc/apache2/envvars
sed -i.bak s/APACHE_RUN_GROUP=www-data/APACHE_RUN_GROUP=mentat/g /etc/apache2/envvars
rm /etc/apache2/envvars.bak

# Step 4: Enable/disable appropriate Apache sites and modules:
a2enmod rewrite
a2enmod ssl
a2enmod shib2
a2enmod wsgi
a2dissite 000-default.conf
a2dissite default-ssl.conf
a2ensite site_mentat.conf

# Step 5: Enforce process based Apache worker
a2dismod mpm_worker
a2dismod mpm_event
a2enmod mpm_prefork

# !!!!! THIS STEP IS SUPER IMPORTANT !!!!!
# Step 6: Change default SECRET_KEY in /etc/mentat/mentat-hawat.py.conf:
# !!!!! THIS STEP IS SUPER IMPORTANT !!!!!
vim /etc/mentat/mentat-hawat.py.conf

# Step 7: Restart Apache service:
systemctl restart apache2.service

# Step 8: Create initial user account/s in Mentat`s metadata database
# (please adjust the attributes, do not simply copy and paste):
mentat-dbmngr.py --command user-add login=superman "fullname=Clark Kent" email=kent@dailyplanet.com "organization=Daily Planet, inc." roles=user,admin

In case you need more local configuration file adjustments you may use environment variable FLASK_CONFIG_FILE, which may point to the additional configuration file with configuration overrides.

In case you are installing web interface for development version installed directly from Git, there is another example configuration file in repository ./conf/apache/site_mentat_dev.conf.shibboleth.example and another WSGI script ./bin/mentat-hawat-dev.wsgi, that are more suitable for this kind of deployment. Both of these files expect Mentat repository to be cloned into /home/mentat/mentat-ng directory. The former file ilustrates the possibility of additional security setting by hiding the whole web interface behind the authentication service to secure Flask`s built-in development console from open Internet. The latter file ilustrates custom WSGI script that makes use of local .env file and python-dotenv feature to easily customize the web interface configuration in development environment.

Directory layout

Location

Content

/etc/mentat

Configuration files for Mentat modules.

/etc/cron.d

Some Mentat modules are launched periodically via cron. When enabled, you may find them here prefixed with mentat-*.

/var/mentat

Working directory for all Mentat modules. Pid files, runlogs, log files and other runtime data is located inside this directory.

/usr/local/bin

Executable files.

What is next?

OK. You have your Mentat system successfully installed, so what is next?

  • If you are upgrading from previous version of Mentat system, you might wish to read and follow the Upgrading documentation page.

  • If you are migrating from previous version of Mentat system, you might wish to read and follow the Migration documentation page.

  • If you want to take quick tour of the Mentat system, you might wish to read and follow the Quickstart documentation page.

  • If you want to read our recommendations and tips about administration and management of the Mentat system, you might wish to read and follow the Administration documentation page.

Troubleshooting

Installing HTTPS driver for apt

You might encounter following error when trying to install mentat-ng Debian package:

E: The method driver /usr/lib/apt/methods/https could not be found.

The problem is, that our Debian are server over HTTPS and you have to install appropriate driver for apt:

apt-get install apt-transport-https

Installing pip3 manually using get-pip.py

If you run into issues with pip3, because your native packaging system contains older versions and you are attempting to install a package requiring newer version, you might opt to uninstall the Debian version of pip3 and install latest manually using get-pip.py script: