Installation

The Mentat system is available as native Python package from PyPI. However there is only so much you can do with setuptools, so to make the installation more convenient for users we are also providing package for Debian based systems via our custom internal repository. We are trying hard to keep the installation process as simple as possible, however at the same time we try not to force too many decisions on the users of the system. For that reason the Debian package for Mentat system does not pull and automatically install all required dependencies, but you will have to install couple of additional Debian system packages and/or Python libraries manually according to your setup and preferences.

Note

As an example try to compare version of PostgreSQL database that is available via official Debian repositories with version available via repository operated by PostgreSQL development team. That is the reason why we did not add that directly to the list of Debian package dependencies.

For your convenience there is a section dedicated to installation of prerequisites:

  1. Installation guide - Prerequisites

To sum it up currently you have following installation options:

  1. Installation guide - Manual installation

  2. Installation guide - Installation from Git repository

  3. Installation guide - Ansible roles

Note

We are using HTTPS to serve our internal Debian package repository. You need to have the apt-transport-https package installed on your target Debian host to be able to access it.

Dependencies

System dependencies

Following is a list of most important system dependencies:

RRDTool

The OpenSource industry standard, high performance data logging and graphing system for time series data.

In Mentat system it is used for storing and displaying continuous system processing performance statistics.

This dependency will be istalled for you automatically by Debian packaging system, because the package version in Debian repository is good enough for our needs.

PostgreSQL

PostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness. It runs on all major operating systems, including Linux, UNIX (AIX, BSD, HP-UX, macOS, Solaris), and Windows. It is fully ACID compliant, has full support for foreign keys, joins, views, triggers, and stored procedures (in multiple languages). It includes most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP. It also supports storage of binary large objects, including pictures, sounds, or video. It has native programming interfaces for C/C++, Java, .Net, Perl, Python, Ruby, Tcl, ODBC, among others, and exceptional documentation.

In Mentat system it is used as a backend service for persistent data storage.

You have to install this dependency manually according to your preferences. Alternativelly you can use prepared minimalistic Ansible role postgresql (see section Installation guide - Ansible roles below for more details).

Additionally, it is necessary to install the IP4R PostgreSQL extension to enable support for better handling of mixed IP addresses and ranges.

Python dependencies

Following is a list of most important and/or notable Python dependencies:

geoip2

Python binding for MaxMind GeoIP2 API.

Jinja2

A small but fast and easy to use stand-alone template engine written in pure Python.

Babel

A collection of tools for internationalizing Python applications.

WTForms

WTForms is a flexible forms validation and rendering library for Python web development.

ply

Python Lex & Yacc.

psycopg2

Python driver for PostgreSQL.

SQLAlchemy

Python universal DBAL and ORM toolkit.

alembic

Database migration tool.

rrdtool

Python bindings for RRDTool.

Flask

Awesome Python microframework for web development.

ipranges

Python library for working with IP addressess.

typedcols

Python library providing typed collections.

idea-format

Python library for working with IDEA messages.

Pydgets

Python3 command line widget library.

PyZenKit

Collection of usefull tools and utilities for creating console applications, scripts and system services (daemons) in Python 3.

Pynspect

Python library for filtering, querying or inspecting almost arbitrary data structures.

When installing from our Debian packages, all these dependencies are installed for you automatically into a separate Python virtual environment (we try hard not to mess up your host system). In case you need to install/reinstall them manually, you can use pip3 and prepared Python requirements file /etc/mentat/requirements.pip:

$ . /var/mentat/venv/bin/activate
(venv) $ pip3 install -r /etc/mentat/requirements.pip
(venv) $ deactivate

Installation guide - Prerequisites

This section is included here solely for your convenience. Please be aware, that official installation instructions in the official documentation is always best source of information. Please refer to that documentation in case you encounter any errors.

# Run these commands as root:
$ echo "deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main" | tee /etc/apt/sources.list.d/pgdg.list
$ wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
$ aptitude update
$ aptitude install postgresql-11 postgresql-11-ip4r postgresql-client-11 postgresql-server-dev-11 libpq-dev pgadmin4
# Run these commands as root:
$ curl -sL https://deb.nodesource.com/setup_8.x | bash -
$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
$ aptitude update
$ aptitude install build-essential nodejs yarn
# Run these commands as root:
$ npm install -g grunt-cli

Installation guide - Manual installation

The installation of base system is pretty straightforward, please follow these simple steps (all commands should be executed as root):

# Step 1: Install and configure PostgreSQL database. See official documentation,
# or refer to appropriate section above.

# Step 2: Download apt list file for mentat production package repository:
$ wget -O /etc/apt/sources.list.d/mentat-production.list https://alchemist.cesnet.cz/mentat/deb/mentat-production.list

# Step 3: Download and install GnuPG signing key for mentat production package repository:
$ wget -O - https://alchemist.cesnet.cz/mentat/deb/mentat-production.gpg | apt-key add -

# Step 4: Refresh apt cache:
$ apt-get update

# Step 5: List all packages available in newly added repository (optional):
$ grep "Package:" /var/lib/apt/lists/alchemist.cesnet.cz_*mentat*_Packages | uniq

# Step 6: Install the Mentat system:
$ apt-get install mentat-ng

After these steps there are several post-installation tasks to be performed:

# To be safe activate the Python virtual environment:
$ . /var/mentat/venv/bin/activate

# Step 8: Initialize PostgreSQL database schema:
(venv) $ mentat-dbmngr.py --command init

# Step 9: Stamp both metadata end event databases with latest migration revisions:
(venv) $ hawat-cli db stamp head
(venv) $ /etc/mentat/scripts/sqldb-migrate.sh stamp head

# Step 10: Precache various event search form select option dictionaries:
(venv) $ mentat-precache.py --allow-empty

# Deactivate now unnecessary virtual environment:
(venv) $ deactivate

The Mentat system is now successfully installed. You may try to start everything up:

$ mentat-controller.py
$ mentat-controller.py --command start
$ hawat-cli run --host localhost --port 5000

The hawat-cli run command in the example above launches built-in development web server providing the Hawat web interface. It should be used only for demonstration or development purposes and not accessible from the outside world. You must use a production level webserver like Apache2 for serving the web interface, please refer to section Installation guide - Web interface with Apache2 below for more details.

Installation guide - Installation from Git repository

This type of installation is useful mainly either for developers, or for deploying the application for testing/debugging/demonstration purposes. Prior to installation you must provide following prerequisites on the target host system:

The installation of base system is pretty straightforward, please follow these simple steps:

# Step 1: Install and configure PostgreSQL database. See official documentation,
# or refer to appropriate section above.

# Step 2: Install Yarn package manager for Node.JS. See official documentation,
# or refer to appropriate section above.

# Step 3: Install Grunt task manager. See official documentation, or refer to
# appropriate section above.

# Step 4: Clone the git repository. Please note, that following command uses
# HTTPS and you will therefore have read-only access to the repository. In
# case you will be working and contributing code, you must either send patches
# via email, or be granted read-write access to our repository. In that case
# the repository URL will be something like [username]@homeproj.cesnet.cz:mentat-ng,
# please adjust following command accordingly.
$ git clone --recurse-submodules https://homeproj.cesnet.cz/git/mentat-ng.git/ mentat-ng
$ cd mentat-ng

# Step 5: Switch to development branch (optional, depending on the use case)
$ git checkout devel

# Step 6: Check for presence of all prerequisites:
$ make deps-prerequisites

# Step 7: Bootstrap Python virtual environment for development:
$ make venv

# Step 8: Activate virtual environment before any further work:
$ . venv/bin/activate

# Step 9: Perform the installation magic:
(venv) $ make develop

After these steps there are several post-installation tasks to be performed:

# Do not forget to activate the virtual environment:
$ . venv/bin/activate

# Step 10: Initialize PostgreSQL database schema:
(venv) $ mentat-dbmngr.py --command init

# Step 11: Stamp both metadata end event databases with latest migration revisions:
(venv) $ hawat-cli db stamp head
(venv) $ ./scripts/sqldb-migrate.sh stamp head

# Step 12: Precache various event search form select option dictionaries:
(venv) $ mentat-precache.py --allow-empty

The Mentat system is now successfully installed. You may try to start everything up:

# Do not forget to activate the virtual environment:
$ . venv/bin/activate

(venv) $ mentat-controller.py
(venv) $ mentat-controller.py --command start
(venv) $ hawat-cli run --host localhost --port 5000

The hawat-cli run command in the example above launches built-in development web server providing the Hawat web interface. It should be used only for demonstration or development purposes and not accessible from the outside world. You must use a production level webserver like Apache2 for serving the web interface, please refer to section Installation guide - Web interface with Apache2 below for more details.

The whole application uses the ./chroot subdirectory of current working directory as a kind of lightweight chroot jail and places all runtime files inside it.

Please also note the existence of .env configuration file, that was created for you during the installation. This file contains various local custom configurations, that are loaded using python-dotenv.

Installation guide - Ansible roles

Note

Please do not use Ansible roles for installation during migration from previous stable version of Mentat based on Perl and MongoDB. Some of the configuration files changed and you may encounter weird errors in case original configuration stays active. Part of the migration process is assessing changes in configuration files and this process is not automated and needs to be done by hand.

If you are fond of Ansible, there is great news for you: we have prepared Ansible roles to make the installation as simple as possible. These roles are available via official Ansible Galaxy site. They can be found easily using the galaxy search form. For the sake of completeness this is a short guide how to quickly use them, but you can use them as usual in your own role ecosystem.

First there is a basic role called mentat. This role will perform only basic Mentat installation. It covers steps 1 through 11 of manual installation with some additional actions like starting Mentat system and enabling it at system boot. Additionally it will install the PostgreSQL database for you through native dependency on postgresql role:

# Step 1: Create working directory:
mkdir ansible-mentat && cd ansible-mentat && mkdir roles

# Step 2: Download the roles:
ansible-galaxy install honzamach.mentat --roles-path=./roles

# Step 3: Pull the Mentat playbook to local directory:
cp roles/honzamach.mentat/role_mentat.yml .

# Step 4: Create inventory file 'inventory' with content similar to this:
[servers-production]
your.server.hostname

[servers-mentat]
your.server.hostname

# Step 5: Execute the playbook:
ansible-playbook -i inventory role_mentat.yml

And that is it. After a lot of crunching Mentat system should be installed on your target device. Again, this playbook installs only basic system without configuration of Apache webserver or creating any demonstration accounts.

If you would like to quickly bootstrap the demonstration installation of Mentat system including Hawat web interface and some preconfigured user accounts, groups etc., you may use another prepared role called mentat_demo. This role uses dependency on mentat role described above to perform basic installation and then builds on top of that:

# Step 1: Create working directory:
mkdir ansible-mentat && cd ansible-mentat && mkdir roles

# Step 2: Download the roles:
ansible-galaxy install honzamach.mentat_demo --roles-path=./roles

# Step 3: Pull the Mentat demo playbook to local directory:
cp roles/honzamach.mentat_demo/role_mentat_demo.yml .

# Step 4: Create inventory file 'inventory' with content similar to this:
[servers-mentat-demo]
your.server.hostname

# Step 5: Execute the playbook:
ansible-playbook -i inventory role_mentat_demo.yml

And voilà! Your Mentat demo server is up and running!

Note

Please browse through the tasks of both playbooks to known what actions are actually being done on taget host. Ansible is designed to be human readable and easily understandable, so it does not make much sense to describe set of actions for each playbook. That would only duplicate the information.

Installation guide - Web interface with Apache2

If you wish to use the web interface Hawat, you wil have to install and configure a web server. You may use any server you like as long as it supports execution of WSGI Python3 scripts. Do not forget to setup some initial user accounts, either use the demonstration fixtures, or setup real user accounts. Otherwise you will not be able to log in.

Following examples demonstrate necessary Apache2 configuration when Mentat system is installed on target system with Debian packages. Please review and adjust the content of all listed configuration files according to your needs, they are not ready for deployment out of the box:

/etc/apache2/sites-available/site_mentat.conf

Main Mentat web interface configuration file for Apache web server.

/etc/mentat/mentat-hawat.py.conf

Main Mentat web interface configuration file.

# Step 1: Install Apache web server:
apt-get install apache2 libapache2-mod-wsgi-py3

# Step 2: Copy example Apache site configuration file and adjust it according to your needs:
cp /etc/mentat/apache/site_mentat.conf.htpasswd.example /etc/apache2/sites-available/site_mentat.conf

# Step 3: Configure Apache2 to run as mentat user in '/etc/apache2/envvars' file:
sed -i.bak s/APACHE_RUN_USER=www-data/APACHE_RUN_USER=mentat/g /etc/apache2/envvars
sed -i.bak s/APACHE_RUN_GROUP=www-data/APACHE_RUN_GROUP=mentat/g /etc/apache2/envvars
rm /etc/apache2/envvars.bak

# Step 4: Enable/disable appropriate Apache sites and modules:
a2enmod rewrite
a2enmod ssl
a2enmod wsgi
a2dissite 000-default.conf
a2dissite default-ssl.conf
a2ensite site_mentat.conf

# !!!!! THIS STEP IS SUPER IMPORTANT !!!!!
# Step 5: Change default SECRET_KEY in /etc/mentat/mentat-hawat.py.conf:
# !!!!! THIS STEP IS SUPER IMPORTANT !!!!!
vim /etc/mentat/mentat-hawat.py.conf

# Step 6: Restart Apache service:
systemctl restart apache2.service

# Step 7: OPTION A: Insert demonstration data into Mentat`s metadata database
# (this will insert some demo accounts, groups, networks and filters into db
# and can be used in conjuction with htpasswd file above):
mentat-dbmngr.py --command fixtures-add

# Step 8: OPTION B: Create initial user account/s in Mentat`s metadata database
# (please adjust the attributes, do not simply copy and paste):
mentat-dbmngr.py --command user-add login=superman "fullname=Clark Kent" email=kent@dailyplanet.com "organization=Daily Planet, inc." roles=user,admin

Warning

For demonstration purposes Mentat package ships with preconfigured htpasswd file containing following user accounts (login - password):

  • user - user

  • manager - manager

  • developer - developer

  • admin - admin

This can be used in conjunction with database fixtures above (step 17, option A) to immediately start exploring the Mentat system. Please make sure NOT to use it in production environment!!!

Following recipe is another example installation of Apache web server, this time with Shibboleth SSO login service (this SSO login service is used for example in Czech eduID identity federation):

# Step 1: Install Apache web server:
apt-get install apache2 libapache2-mod-wsgi-py3 libapache2-mod-shib2

# Step 2: Copy example Apache site configuration file and adjust it according to your needs:
cp /etc/mentat/apache/site_mentat.conf.shibboleth.example /etc/apache2/sites-available/site_mentat.conf

# Step 3: Configure Apache2 to run as mentat user in '/etc/apache2/envvars' file:
sed -i.bak s/APACHE_RUN_USER=www-data/APACHE_RUN_USER=mentat/g /etc/apache2/envvars
sed -i.bak s/APACHE_RUN_GROUP=www-data/APACHE_RUN_GROUP=mentat/g /etc/apache2/envvars
rm /etc/apache2/envvars.bak

# Step 4: Enable/disable appropriate Apache sites and modules:
a2enmod rewrite
a2enmod ssl
a2enmod shib2
a2enmod wsgi
a2dissite 000-default.conf
a2dissite default-ssl.conf
a2ensite site_mentat.conf

# !!!!! THIS STEP IS SUPER IMPORTANT !!!!!
# Step 5: Change default SECRET_KEY in /etc/mentat/mentat-hawat.py.conf:
# !!!!! THIS STEP IS SUPER IMPORTANT !!!!!
vim /etc/mentat/mentat-hawat.py.conf

# Step 6: Restart Apache service:
systemctl restart apache2.service

# Step 7: Create initial user account/s in Mentat`s metadata database
# (please adjust the attributes, do not simply copy and paste):
mentat-dbmngr.py --command user-add login=superman "fullname=Clark Kent" email=kent@dailyplanet.com "organization=Daily Planet, inc." roles=user,admin

In case you need more local configuration file adjustments you may use environment variable FLASK_CONFIG_FILE, which may point to the additional configuration file with configuration overrides.

In case you are installing web interface for development version installed directly from Git, there is another example configuration file in repository ./conf/apache/site_mentat_dev.conf.shibboleth.example and another WSGI script ./bin/mentat-hawat-dev.wsgi, that are more suitable for this kind of deployment. Both of these files expect Mentat repository to be cloned into /home/mentat/mentat-ng directory. The former file ilustrates the possibility of additional security setting by hiding the whole web interface behind the authentication service to secure Flask`s built-in development console from open Internet. The latter file ilustrates custom WSGI script that makes use of local .env file and python-dotenv feature to easily customize the web interface configuration in development environment.

Directory layout

Location

Content

/etc/mentat

Configuration files for Mentat modules.

/etc/cron.d

Some Mentat modules are launched periodically via cron. When enabled, you may find them here prefixed with mentat-*.

/var/mentat

Working directory for all Mentat modules. Pid files, runlogs, log files and other runtime data is located inside this directory.

/usr/local/bin

Executable files.

What is next?

OK. You have your Mentat system successfully installed, so what is next?

  • If you are upgrading from previous version of Mentat system, you might wish to read and follow the Upgrading documentation page.

  • If you are migrating from previous version of Mentat system, you might wish to read and follow the Migration documentation page.

  • If you want to take quick tour of the Mentat system, you might wish to read and follow the Quickstart documentation page.

  • If you want to read our recommendations and tips about administration and management of the Mentat system, you might wish to read and follow the Administration documentation page.

Troubleshooting

Installing HTTPS driver for apt

You might encounter following error when trying to install mentat-ng Debian package:

E: The method driver /usr/lib/apt/methods/https could not be found.

The problem is, that our Debian are server over HTTPS and you have to install appropriate driver for apt:

apt-get install apt-transport-https

Installing pip3 manually using get-pip.py

If you run into issues with pip3, because your native packaging system contains older versions and you are attempting to install a package requiring newer version, you might opt to uninstall the Debian version of pip3 and install latest manually using get-pip.py script: