Upgrading

This document describes system upgrade process from previous Mentat production versions within the 2.x series. For upgrading from 0.4.20 to 2.x series please see the Migration section.

Warning

Prerequisite for upgrading is an existing and working installation of all Mentat system packages.

Upgrading Mentat system

Please proceed according to the following recipe to safelly upgrade your installation to latest version:

# Step 0: Activate maintenance mode in case the downtime will be noticable for users:
# First update timestamps of maintenance start and maintenance end:
$ vim /etc/mentat/apache/maintenance/.htaccess
# Now bring the Mentat system web interface down and maintenance site up:
$ a2enmod substitute
$ a2dissite site_mentat-ng.conf
$ a2ensite site_maintenance.conf
$ systemctl restart apache2

# Step 1: It is recommended to stop Mentat daemons and cronjobs before upgrade:
$ mentat-controller.py --command stop
$ mentat-controller.py --command disable

# Step 2: Perform the actual upgrade:
$ aptitude update
$ aptitude upgrade

# Step 3: To be safe activate the Python virtual environment for Mentat system:
$ . /var/mentat/venv/bin/activate

# Step 4: Make sure your database schema is up to date. Please be aware, that
# these operations may need a lot of time to complete depending on the size
# of your database:
(venv) $ time mentat-dbmngr.py --command init
(venv) $ time hawat-cli db upgrade
(venv) $ time /etc/mentat/scripts/sqldb-migrate.sh upgrade head
# At this point it could be wise to verify the state of the database and
# perform some maintenance tasks to prevent from thrashing. Please be aware,
# that these operations may need a lot of time to complete depending on the
# size of your database:
(venv) $ psql mentat_events
\timing on
VACUUM FREEZE;
CLUSTER;
ANALYZE;

# Step 5: Deactivate now unnecessary virtual environment:
(venv) $ deactivate

# Step 6: Start all your Mentat daemons and cronjobs again:
$ mentat-controller.py --command start
$ mentat-controller.py --command enable

# Step 7: Restart the web server that is serving web interface:
$ a2dismod substitute
$ a2dissite site_maintenance.conf
$ a2ensite site_mentat-ng.conf
$ systemctl restart apache2

Upgrading PostgreSQL from 10.x to 11.x

Following checklist describes the steps necessary to upgrade the PostgreSQL database from version 10.x to 11.x. Please be aware, that the database upgrade is not a straightforward operation. It can take a lot of time depending on the size of the current database, because the data files need to be converted to new format.

# Stop all processes touching the PostgreSQL database:
$ systemctl stop apache2
$ mentat-controller.py --command stop
$ mentat-controller.py --command disable
$ systemctl stop postgresql

# Install PostgreSQL 11 & friends:
$ aptitude update
$ aptitude install postgresql-11 postgresql-11-ip4r postgresql-server-dev-11

# Verify the installation success (output included):
$ pg_lsclusters
:Ver Cluster Port Status Owner    Data directory              Log file
:10  main    5432 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
:11  main    5433 online postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log

# PostgreSQL was started during installation, stop it again:
$ systemctl stop postgresql

# Drop the default PostgreSQL 11 cluster created during installation:
$ pg_dropcluster 11 main

# Verify the clusters (output included):
$ pg_lsclusters
:Ver Cluster Port Status Owner    Data directory              Log file
:10  main    5432 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log

# Perform the data migration (slow to complete):
$ pg_upgradecluster --method=upgrade 10 main

# Drop the PostgreSQL 10 data as there are two copies (10+11):
$ pg_dropcluster --stop 10 main

# Remove the old PostgreSQL version & friends:
$ aptitude purge postgresql-10 postgresql-client-10 postgresql-server-dev-10

# Start the DB (maintenance still required, not ready for system uptime):
$ systemctl start postgresql

# From the PostgreSQL shell (psql):
# The CLUSTER is optional, it takes time but can shrink the DB size considerably if not done recently
VACUUM VERBOSE;
-- CLUSTER VERBOSE events;
ANALYZE VERBOSE;

# This is a good time for restart (optional). New kernel? Long uptime & non-ECC RAM?
$ reboot

# Now the system is ready for production, start it up
$ systemctl start apache2
$ mentat-controller.py --command enable
$ mentat-controller.py --command start

What is next?

You have just successfully upgraded Mentat system to latest version, so what is next?