Upgrading

This document describes system upgrade process from previous Mentat production versions within the 2.x series. For upgrading from 0.4.20 to 2.x series please see the Migration section.

Warning

Prerequisite for upgrading is an existing and working installation of all Mentat system packages.

Upgrading Mentat system

Please proceed according to the following recipe to safelly upgrade your installation to latest version:

# Launch tmux or screen.
tmux

# Step 0: Activate maintenance mode in case the downtime will be noticable for users:
# First update timestamps of maintenance start and maintenance end:
$ vim /etc/mentat/apache/maintenance/.htaccess
# Now bring the Mentat system web interface down and maintenance site up:
$ a2enmod substitute
$ a2dissite site_mentat-ng.conf
$ a2ensite site_maintenance.conf
$ systemctl restart apache2

# Step 1: It is recommended to stop Mentat daemons and cronjobs before upgrade:
$ mentat-controller.py --command stop
$ mentat-controller.py --command disable

# Step 2: Perform the actual upgrade:
$ aptitude update
$ aptitude upgrade

# Step 3: To be safe activate the Python virtual environment for Mentat system:
$ . /var/mentat/venv/bin/activate

# Step 4: Make sure your database schema is up to date. Please be aware, that
# these operations may need a lot of time to complete depending on the size
# of your database:
(venv) $ time mentat-dbmngr.py --command init
(venv) $ time hawat-cli db upgrade
(venv) $ time /etc/mentat/scripts/sqldb-migrate.sh upgrade head
# At this point it could be wise to verify the state of the database and
# perform some maintenance tasks to prevent from thrashing. Please be aware,
# that these operations may need a lot of time to complete depending on the
# size of your database:
(venv) $ psql mentat_events
\timing on
VACUUM FREEZE VERBOSE;
CLUSTER VERBOSE;
ANALYZE VERBOSE;
(venv) $ psql mentat_main
\timing on
VACUUM FREEZE VERBOSE;
CLUSTER VERBOSE;
ANALYZE VERBOSE;

# Step 5: Deactivate now unnecessary virtual environment:
(venv) $ deactivate

# Step 6: Start all your Mentat daemons and cronjobs again:
$ mentat-controller.py --command start
$ mentat-controller.py --command enable

# Step 7: Restart the web server that is serving web interface:
$ a2dismod substitute
$ a2dissite site_maintenance.conf
$ a2ensite site_mentat-ng.conf
$ systemctl restart apache2

Upgrading PostgreSQL from 10.x to 11.x

Following checklist describes the steps necessary to upgrade the PostgreSQL database from version 10.x to 11.x.

Warning

Please be aware, that the database upgrade is NOT a straightforward operation. It can take a lot of time depending on the size of the current database, because the data files need to be converted to new format.

# Launch tmux or screen.
tmux

# Step 0: Activate maintenance mode:
# First update timestamps of maintenance start and maintenance end:
$ vim /etc/mentat/apache/maintenance/.htaccess
# Now bring the Mentat system web interface down and maintenance site up:
$ a2enmod substitute
$ a2dissite site_mentat-ng.conf
$ a2ensite site_maintenance.conf
$ systemctl restart apache2

# Step 1: Stop all processes touching the PostgreSQL database:
$ systemctl stop apache2
$ mentat-controller.py --command stop
$ mentat-controller.py --command disable
$ systemctl stop postgresql

# Step 2: Install PostgreSQL 11:
$ aptitude update
$ aptitude install postgresql-11 postgresql-11-ip4r postgresql-server-dev-11 postgresql-client-11

# Step 3: Verify the installation success (output included):
$ pg_lsclusters
:Ver Cluster Port Status Owner    Data directory              Log file
:10  main    5432 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
:11  main    5433 online postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log

# Step 4: PostgreSQL was started during installation, stop it again:
$ systemctl stop postgresql

# Step 5: Drop the default PostgreSQL 11 cluster created during installation:
$ pg_dropcluster 11 main

# Step 6: Verify the clusters (output included):
$ pg_lsclusters
:Ver Cluster Port Status Owner    Data directory              Log file
:10  main    5432 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log

# Step 7: Perform the data migration (slow to complete):
$ pg_upgradecluster --method=upgrade 10 main

# Step 8: Drop the PostgreSQL 10 data as there are two copies (10+11):
$ pg_dropcluster --stop 10 main

# Step 9: Remove the old PostgreSQL version:
$ aptitude purge postgresql-10 postgresql-10-ip4r postgresql-server-dev-10 postgresql-client-10

# Step 10: Start the DB (maintenance still required, not ready for system uptime):
$ systemctl start postgresql

# Step 11: From the PostgreSQL shell (psql):
# The CLUSTER is optional, it takes time but can shrink the DB size considerably if not done recently
VACUUM VERBOSE;
-- CLUSTER VERBOSE events;
ANALYZE VERBOSE;

# Step 12: This is a good time for restart (optional). New kernel? Long uptime & non-ECC RAM?
$ reboot

# Step 13: Now the system is ready for production, start it up
$ systemctl restart postgresql
$ systemctl start apache2
$ mentat-controller.py --command enable
$ mentat-controller.py --command start

# Step 14: Restart the web server that is serving web interface:
$ a2dismod substitute
$ a2dissite site_maintenance.conf
$ a2ensite site_mentat-ng.conf
$ systemctl restart apache2

After these steps it is necessary to update following configuration files:

/etc/mentat/mentat-cleanup.py.conf

Change configuration db_path to point to correct filesystem location. In default Debian installations it should look something like this:

"db_path": "/var/lib/postgresql/11/main",

Upgrading PostgreSQL from 11.x to 12.x

Following checklist describes the steps necessary to upgrade the PostgreSQL database from version 11.x to 12.x.

Warning

Please be aware, that the database upgrade is NOT a straightforward operation. It can take a lot of time depending on the size of the current database, because the data files need to be converted to new format.

# Launch tmux or screen.
tmux

# Step 0: Activate maintenance mode:
# First update timestamps of maintenance start and maintenance end:
$ vim /etc/mentat/apache/maintenance/.htaccess
# Now bring the Mentat system web interface down and maintenance site up:
$ a2enmod substitute
$ a2dissite site_mentat-ng.conf
$ a2ensite site_maintenance.conf
$ systemctl restart apache2

# Step 1: Stop all processes touching the PostgreSQL database:
$ sudo systemctl stop warden_filer_cesnet_receiver.service
$ sudo systemctl disable warden_filer_cesnet_receiver.service
$ sudo mentat-controller.py --command stop
$ sudo mentat-controller.py --command disable
$ systemctl restart postgresql

### There can be no DB writes beyond this point as we are about to drop indices to ensure data integrity!

# Step 2: Connect to current database:
$ psql mentat_events
DROP INDEX events_detecttime_idx;
DROP INDEX events_combined_idx;
DROP INDEX events_cesnet_storagetime_idx;
DROP INDEX events_cesnet_eventseverity_idx;
ALTER TABLE events DROP CONSTRAINT events_pkey;
VACUUM FREEZE VERBOSE;
CHECKPOINT;

# Step 3: Stop PostgreSQL:
$ sudo systemctl stop postgresql

# Step 4: Install PostgreSQL 12:
$ sudo apt-get update
$ sudo apt-get install postgresql-12 postgresql-12-ip4r postgresql-server-dev-12 postgresql-client-12

# Step 5: Migration:
$ sudo pg_lsclusters
Ver Cluster Port Status Owner    Data directory              Log file
11  main    5432 online postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log
12  main    5433 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log

$ sudo systemctl stop postgresql

$ sudo pg_dropcluster 12 main

$ sudo pg_lsclusters
Ver Cluster Port Status Owner    Data directory              Log file
11  main    5432 down   postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log

# This will require *temporarily* setting wal_level to 'logical' (in postgresql.conf) - it is set to 'minimal' if you followed configuration advice from docs
# Alternatively one can ommit the --link parameter, but that requires free space for a 1:1 copy and of course also takes much longer
$ sudo pg_upgradecluster --method=upgrade --link 11 main

$ sudo pg_dropcluster 11 main

# Step 6: Remove PostgreSQL 11 and all prior versions:
$ sudo apt-get remove --purge postgresql-11 postgresql-client-11 postgresql-server-dev-11 postgresql-11-ip4r postgresql-9.4 postgresql-9.5 postgresql-9.6 postgresql-10

# Step 7: Start PostgreSQL:
$ sudo systemctl start postgresql

# Step 8: Recreate indices:
REINDEX DATABASE mentat_events;
ALTER TABLE events ADD PRIMARY KEY (id);
CREATE INDEX IF NOT EXISTS events_detecttime_idx ON events USING BTREE (detecttime);
CREATE INDEX IF NOT EXISTS events_cesnet_storagetime_idx ON events USING BTREE (cesnet_storagetime);
CREATE INDEX IF NOT EXISTS events_cesnet_eventseverity_idx ON events USING BTREE (cesnet_eventseverity) WHERE cesnet_eventseverity IS NOT NULL;
CREATE INDEX IF NOT EXISTS events_combined_idx ON events USING GIN (category, node_name, protocol, source_port, target_port, source_type, target_type, node_type, cesnet_resolvedabuses, cesnet_inspectionerrors);
CHECKPOINT;
ANALYZE VERBOSE;

# Step 9: This is a good time for restart (optional). New kernel? Long uptime & non-ECC RAM?
$ reboot

# Step 10: Start Mentat and all other services:
$ systemctl restart postgresql
$ sudo mentat-controller.py --command enable
$ sudo mentat-controller.py --command start
$ sudo systemctl start warden_filer_cesnet_receiver.service
$ sudo systemctl enable warden_filer_cesnet_receiver.service

# Step 11: Restart the web server that is serving web interface:
$ a2dismod substitute
$ a2dissite site_maintenance.conf
$ a2ensite site_mentat-ng.conf
$ systemctl restart apache2

After these steps it is necessary to update following configuration files:

/etc/mentat/mentat-cleanup.py.conf

Change configuration db_path to point to correct filesystem location. In default Debian installations it should look something like this:

"db_path": "/var/lib/postgresql/12/main",

Upgrading underlying Debian system

After upgrading underlying Debian system please execute following set of commands to repair installation of Mentat system:

$ rm -rf /var/mentat/venv
$ apt install mentat-ng --reinstall

What is next?

You have just successfully upgraded Mentat system to latest version, so what is next?