Upgrading

This document describes system upgrade process from previous Mentat production versions within the 2.x series. For upgrading from 0.4.20 to 2.x series please see the Migration section.

Warning

Prerequisite for upgrading is an existing and working installation of all Mentat system packages.

Upgrading Mentat system

Please proceed according to the following recipe to safelly upgrade your installation to latest version:

# Launch tmux or screen.
tmux

# Step 0: Activate maintenance mode in case the downtime will be noticable for users:
# First update timestamps of maintenance start and maintenance end:
$ vim /etc/mentat/apache/maintenance/.htaccess
# Now bring the Mentat system web interface down and maintenance site up:
$ a2enmod substitute
$ a2dissite site_mentat-ng.conf
$ a2ensite site_maintenance.conf
$ systemctl restart apache2

# Step 1: It is recommended to stop Mentat daemons and cronjobs before upgrade:
$ mentat-controller.py --command stop
$ mentat-controller.py --command disable

# Step 2: Perform the actual upgrade:
$ aptitude update
$ aptitude upgrade

# Do not forget to review changes in configuration files/structure and
# adapt accordingly.

# Step 3: To be safe activate the Python virtual environment for Mentat system:
$ . /var/mentat/venv/bin/activate

# Step 4: Make sure your database schema is up to date. Please be aware, that
# these operations may need a lot of time to complete depending on the size
# of your database:
(venv) $ time hawat-cli db upgrade
(venv) $ time /etc/mentat/scripts/sqldb-migrate.sh upgrade head
(venv) $ time /etc/mentat/scripts/sqldb-optimize.sh
# At this point it could be wise to verify the state of the database and
# perform some maintenance tasks to prevent from thrashing. Please be aware,
# that these operations may need a lot of time to complete depending on the
# size of your database:
(venv) $ psql mentat_events
\timing on
VACUUM FREEZE VERBOSE;
CLUSTER VERBOSE;
ANALYZE VERBOSE;
(venv) $ psql mentat_main
\timing on
VACUUM FREEZE VERBOSE;
CLUSTER VERBOSE;
ANALYZE VERBOSE;

# Step 5: Deactivate now unnecessary virtual environment:
(venv) $ deactivate

# Step 6: Start all your Mentat daemons and cronjobs again:
$ mentat-controller.py --command start
$ mentat-controller.py --command enable

# Step 7: Restart the web server that is serving web interface:
$ a2dismod substitute
$ a2dissite site_maintenance.conf
$ a2ensite site_mentat-ng.conf
$ systemctl restart apache2

For your convenience there is a maintenance-mode.sh script that you can use to quickly turn maintenance mode ON/OFF. Upgrading steps are then much more simple:

# Launch tmux or screen.
tmux

# Step 1: Activate maintenance mode:
$ /etc/mentat/scripts/maintenance-mode.sh on

# Now perform steps 2-5 from the checklist above

# Step 6: Deactivate maintenance mode:
$ /etc/mentat/scripts/maintenance-mode.sh off

Upgrading to Mentat 2.12

  • Query lengths have exceeded the default postgres limit for query storage, and it is required to increase the track_activity_query_size to at least 32kB so the My queries listings continue to work properly. That means the postgresql configuration file should be modified so that:

track_activity_query_size=32768

To actually enable the changes the postgresql service needs to be restarted.

# Restart the postgres service
$ systemctl restart postgresql

# In order to reconnect the closed database connection, the apache2 service
# might also require a restart
$ systemctl restart apache2

Upgrading to Mentat 2.8

  • Rulesets for two inspectors have been merged into one instance. That concerns mentat-inspector.py, mentat-inspector-b.py.conf and mentat-controller.py.conf. If you use vanilla configuration from .deb, package manager will replace the old configuration files with new ones automatically. If you used different installation method or made some modifications into these configuration files, you are encouraged to review the changes and decide whether you want to stay with split configuration or also proceed with the merge.

Upgrading PostgreSQL from 13.x to 14.x

Following checklist describes the steps necessary to upgrade the PostgreSQL database from version 13.x to 14.x.

Warning

Please be aware that running Mentat is supported only on a specific major version of PostgreSQL (usually the latest), as the Mentat’s database access code is tuned to it and sometimes also requires the new features.

For upgrading PostgreSQL from older versions, please refer to the documentation of previous versions of Mentat.

Warning

Please be aware that the database upgrade is NOT a straightforward operation. It can take a lot of time depending on the size of the current database, because the data files need to be converted to new format.

Upgrade to the latest version of Mentat prior to upgrading PostgreSQL.

# Step 0. Launch tmux or screen
    $ tmux

# Step 1. Activate the maintenance mode website
    # First update timestamps of maintenance start and maintenance end:
    $ sudo vim /etc/mentat/apache/maintenance/.htaccess
    # Now bring the Mentat system web interface down and the maintenance site up:
    $ sudo a2enmod substitute
    $ sudo a2dissite site_mentat-ng.conf
    $ sudo a2ensite site_maintenance.conf
    $ sudo systemctl restart apache2

# Step 2. Shut down the Mentat's import pipeline and stabilize the DB
    $ sudo systemctl stop warden_filer_cesnet_receiver.service
    $ sudo systemctl disable warden_filer_cesnet_receiver.service
    $ sudo mentat-controller.py --command stop
    $ sudo mentat-controller.py --command disable
    $ sudo systemctl restart postgresql

# Step 3. Vacuum the database
    # Typically, peer authentication is set up for user postgres in the DB
    $ sudo -u postgres vacuumdb -F -j 16 -v -a

# Step 4. Stop PostgreSQL
    $ sudo systemctl stop postgresql

# Step 5. Install PostgreSQL 14
    $ sudo apt-get update
    $ sudo apt-get install postgresql-14 postgresql-14-ip4r postgresql-server-dev-14 postgresql-client-14

# (OPTIONAL) Step 6. Update IP4R extension
    psql -d mentat_main -c "ALTER EXTENSION IP4R update"
    psql -d mentat_events -c "ALTER EXTENSION IP4R update"

# Step 7. Back up the default PostgreSQL v14 configuration file
    # This is used later in step 9.
    $ cp /etc/postgresql/14/main/postgresql.conf ~/postgresql_14_default.conf

# Step 8. Migration
    $ sudo pg_lsclusters
    Ver Cluster Port Status Owner    Data directory              Log file
    13  main    5432 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
    14  main    5433 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log

    $ sudo systemctl stop postgresql

    $ sudo pg_dropcluster 14 main

    $ sudo pg_lsclusters
    Ver Cluster Port Status Owner    Data directory              Log file
    13  main    5432 down   postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log

    # This will require *temporarily* setting wal_level to 'logical' (in postgresql.conf), as
    # it is set to 'minimal' if you followed configuration advice from the docs.
    # Alternatively one can ommit the --link parameter, but that requires free space for
    # a 1:1 copy and of course also takes much longer.
    $ sudo sed -i -E 's/^(wal_level\s*=\s*)[a-z]+/\1logical/' /etc/postgresql/13/main/postgresql.conf
    $ sudo pg_upgradecluster --method=upgrade --link 13 main

    $ sudo pg_dropcluster 13 main

# Step 9. Remove PostgreSQL 13 and potential leftovers from previous versions
    $ sudo apt-get remove --purge postgresql-13 postgresql-client-13 postgresql-server-dev-13 postgresql-13-ip4r postgresql-13 postgresql-12 postgresql-client-12 postgresql-server-dev-12 postgresql-12-ip4r postgresql-12 postgresql-client-11 postgresql-server-dev-11 postgresql-11-ip4r postgresql-10 postgresql-9.6 postgresql-9.5 postgresql-9.4

# Step 10. Update the configuration file
    # Related to #6480. This is the most laborous step, which I is not yet automated.
    # Also, sometimes the options are just reordered, what complicates the merge process.
    $ sudo vimdiff /etc/postgresql/14/main/postgresql.conf ~/postgresql_14_default.conf

    # Change the setting for wal_level back to minimal if it was changed in step 7.
    $ sudo sed -i -E 's/^(wal_level\s*=\s*)[a-z]+/\1minimal/' /etc/postgresql/14/main/postgresql.conf

# Step 11a. Reboot the system
    # OPTIONAL: This is a good time to reboot the machine if desired (kernel update,
    # long runtime & non-ECC RAM). Alternatively, just follow with 10b.
    $ sudo reboot

# Step 11b. Start PostgreSQL
    # Only if step 10a was skipped.
    $ sudo systemctl start postgresql

# Step 12a. Cleanup & optimization
    # As PostgreSQL upgrade is done roughly once a year (that is the cadence of major version
    # release), it is a good point to do VACUUM FULL. Alternatively, if downtime has to be
    # minimized at all costs, continue with step 11b.
    # Skipping will save about 10 minutes. It is not recommended.
    $ sudo -u postgres vacuumdb -f -j 16 -v -a -z

# Step 12b. The mandatory ANALYZE
    # Only if step 11a was skipped.
    # At least an ANALYZE run is required as the statistics are not carried over
    # during the upgrade.
    $ sudo -u postgres vacuumdb -Z -j 16 -a

# Step 13. Start Mentat
    $ sudo mentat-controller.py --command enable
    $ sudo mentat-controller.py --command start
    $ sudo systemctl enable warden_filer_cesnet_receiver.service
    $ sudo systemctl start warden_filer_cesnet_receiver.service

# Step 14. Deactivate the maintenance mode website
    $ sudo a2dismod substitute
    $ sudo a2dissite site_maintenance.conf
    $ sudo a2ensite site_mentat-ng.conf
    $ sudo systemctl restart apache2

Upgrading to authenticated version of GeoIP service

Since 30.12.2019 the MaxMind IP geolocation service changed significantly its access policies for downloading free versions of IP geolocation databases. These

Following actions are necessary to make event enrichment with IP geolocation data working again:

  1. Follow installation instructions in section Install - geoipupdate.

  2. Upgrade to Mentat system version 2.6.x and review all default configuration file changes. The changes occured in /etc/mentat/mentat-controller.py.conf and /etc/mentat/core/services.json.conf configuration files. If you are reading this before Mentat version 2.6.x is released please make following configuration changes manually:

diff --git a/conf/mentat-controller.py.conf b/conf/mentat-controller.py.conf
index 493903e2..4e454073 100644
--- a/conf/mentat-controller.py.conf
+++ b/conf/mentat-controller.py.conf
@@ -134,7 +134,7 @@
         { "name": "mentat-cleanup-py" },

-        # Utility for fetching current versions of IP geolocation databases.
-        { "name": "fetch-geoipdb-sh" },
-
         # Provide periodical informational report emails about overall performance of Mentat system.
         { "name": "mentat-informant-py" },

diff --git a/conf/core/services.json.conf b/conf/core/services.json.conf
index 81988e55..ca38cdba 100644
--- a/conf/core/services.json.conf
+++ b/conf/core/services.json.conf
@@ -25,9 +25,9 @@
         # GeoIP service settings.
         #
         "geoip": {
-            "asndb": "/var/opt/opensourcedbs/GeoLite2-ASN.mmdb",
-            "citydb": "/var/opt/opensourcedbs/GeoLite2-City.mmdb"
-            #"countrydb": "/var/opt/opensourcedbs/GeoLite2-Country.mmdb"
+            "asndb": "/usr/share/GeoIP/GeoLite2-ASN.mmdb",
+            "citydb": "/usr/share/GeoIP/GeoLite2-City.mmdb"
+            #"countrydb": "/usr/share/GeoIP/GeoLite2-Country.mmdb"
         },
  1. Restart Mentat system:

$ mentat-controller.py --command stop
$ mentat-controller.py --command start
$ mentat-controller.py --command enable
$ systemctl restart apache2
  1. Finally make sure all obsolete and deprecate files are gone:

rm -f /etc/cron.d/fetch-geoipdb-sh
rm -f /etc/mentat/cron/fetch-geoipdb-sh.cron
rm -f /etc/mentat/scripts/fetch-geoipdb.sh

Upgrading underlying Debian system

After upgrading underlying Debian system please execute following set of commands to repair installation of Mentat system:

$ rm -rf /var/mentat/venv
$ apt install mentat-ng --reinstall

What is next?

You have just successfully upgraded Mentat system to latest version, so what is next?