.. _section-migration: Migration ================================================================================ This document describes migration process from previous Mentat production release ``0.4.20`` to ``2.x`` series. For upgrading from within the ``2.x`` series please see the :ref:`section-upgrading` section. .. warning:: This manual is valid only for Mentat version up to ``2.3.x``. In version ``2.4.x`` support for migration was dropped. This page will remain here until next version and then it too will be gone. Please perform migration as soon as possible. New version of Mentat system comes with quite a lot of new features and most importantly uses `PostgreSQL `__ as database backend. Consider doing clean installation on different host to reduce number of leftover deprecated files. You have following migration options: #. :ref:`section-migration-remote` #. :ref:`section-migration-local` Remote migration is a superset of local migration, you have to move all the necessary data to remote host and then continue according to the local migration checklist. .. _section-migration-remote: Migration to remote host -------------------------------------------------------------------------------- In case you are migrating to another server, there are additional actions that need to be taken besides those when migrating to new version on a single host. The most important part is to synchronize data and configurations. But you also need to take into consideration possible host renaming and readdressing, so you might need to alter DNS records as well. For the data transfer, it is sufficient to use tools like ``scp`` or ``rsync`` and copy all relevant data to target destination. Make sure to turn the whole Mentat system off first on both servers to ensure data integrity. To make the downtime as short as possible you may consider using ``rsync`` multiple times, first iteration will be used to perform crude synchronization, than stop the Mentat system and perform final data synchronization. As an example you may use following checklist: .. code-block:: shell # Step 0: Update TTL on relevant DNS records to reasonably low value in case # you are also renaming. This should not be necessary when you are also # readdressing (swapping two servers altogether). # Step 1: Make sure Mentat system is stopped on target host server (both real-time # and cronjob modules). If you are using Warden as IDEA message source, # stop that as well. root@target$ service warden_filer_receiver stop root@target$ mentat-controller.py --command stop root@target$ mentat-controller.py --command disable # Step 2: Pre-synchronize data directory. You may tweak and use example script # mentat-sync-files.sh described in the next section for more complex # file synchronization process. root@source$ rsync --archive --update --delete --progress /var/mentat root@target:/var # Step 3: Stop the Mentat system on source host (both real-time and cronjob modules). # If you are using Warden as IDEA message source, stop that as well. # This is necessary so that you have clean initial state and for data # integrity. root@target$ service warden_filer_receiver stop root@source$ mentat-controller.py --command stop root@source$ mentat-controller.py --command disable # Step 4: Re-synchronize data directory. Again you may tweak and use example script # mentat-sync-files.sh described in the next section for more complex # file synchronization process. root@source$ rsync --archive --update --delete --progress /var/mentat root@target:/var # Step 5: Migrate the MongoDB database to target host (sadly this step requires # that MongoDB is installed on target host as well, you may remove it # after successfull migration). You may tweak and use example scripts # mentatdb-dump-all.sh and mentat-sync-mongodb.sh for more complex # database synchronization process. root@source$ mkdir -p /var/tmp/mentatdb_dump_all root@source$ cd /var/tmp/mentatdb_dump_all root@source$ mongodump --db mentat_stats --collection statistics root@source$ mongodump --db mentat --collection users root@source$ mongodump --db mentat --collection groups root@source$ mongodump --db mentat --collection reports_ng root@source$ cd /var/tmp root@source$ tar -czvf mentatdb_dump_all.tar.gz mentatdb_dump_all root@source$ rm -rf /var/tmp/mentatdb_dump_all root@source$ scp /var/tmp/mentatdb_dump_all.tar.gz root@target:/var/tmp/ root@target$ cd /var/tmp root@target$ tar -xzvf /var/tmp/mentatdb_dump_all.tar.gz root@target$ cd /var/tmp/mentatdb_dump_all/dump root@target$ mongorestore --drop --db mentat_stats mentat_stats/statistics.bson root@target$ mongorestore --drop --db mentat mentat/users.bson root@target$ mongorestore --drop --db mentat mentat/groups.bson root@target$ mongorestore --drop --db mentat mentat/reports_ng.bson root@target$ rm -f /var/tmp/mentatdb_dump_all.tar.gz root@target$ rm -rf /var/tmp/mentatdb_dump_all That is it, you now have all the data on target host. Please continue to the section :ref:`section-migration-local` to finish the migration process. .. _section-migration-local: Local migration -------------------------------------------------------------------------------- At this point you should have the new version of Mentat system installed and if you were migrating to remote host, you should have all the steps from section :ref:`section-migration-remote` completed. So now you have all the data on the target host server and basically all that remains is to perform database migration from `MongoDB `__ to `PostgreSQL `__. There are migration scripts prepared to do just that, so the whole migration process is as simple as executing them. .. code-block:: shell # Step 0: Reconfigure Mentat system by comparing old and new configuration files. # This should be done manually, because most of the modules have new # configuration options and you should consider tweaking some of them. # This step is really important, you may encounter weird errors in case # some outdated configuration stays active, especially in following files: # - mentat-backup.py.conf # - mentat-cleanup.py.conf # - mentat-controller.py.conf # - mentat-dbmngr.py.conf # - mentat-netmngr.py.conf # - mentat-reporter.py.conf # - mentat-statistician.py.conf # Step 1: Migrate system metadata tables (users, groups, reports, statistics, etc.). /etc/mentat/scripts/sqldb-migrate-data.py # Step 2: Migrate IDEA events. This step is optional, depending on your setup # it might take A LOT of time and it might be better to just skip it # and start from scratch with empty event database, or migrate just # a chunk of the whole database. /etc/mentat/scripts/sqldb-migrate-events.py # Step 3: Remove possible local reports generated by local reporter. rm -rf /var/mentat/reports/reporter # Step 4: Install reports generated by legacy reporter. New reporter does use # a different naming scheme for report attachments (to be more consistent), # so it is necessary to rename the old ones. mv /var/mentat/reports/reporter-ng /var/mentat/reports/reporter find /var/mentat/reports/reporter -type f -exec rename 's/_/-/g' {} \; /etc/mentat/scripts/mentat-reports-order.py # Step 5: Get rid of old log and runlog files (optional, but recommended, you # may encounter weird errors when these do not get cleaned). find /var/mentat/log -name=*.log* -delete find /var/mentat/run -name=*.runlog -delete find /var/mentat/run -name=*.pstate -delete find /var/mentat/run -name=*.state -delete After successfull local migration it is time to start everything up again and make sure everything is in order. .. code-block:: shell # Step 6: Since you are still technically in downtime, this might be a good # opportunity to reboot the host server and make sure averything boots # back up, install new server firmware etc. # Step 7: Start the Mentat system (both real-time and cronjob modules). # If you are using Warden as IDEA message source, start that as well. mentat-controller.py --command enable mentat-controller.py --command start service warden_filer_receiver start # Step 8: Make sure messages are passing through all real-time messages # processing modules by inspecting the log files. Look for obvious # errors and warnings. tail -f /var/mentat/log/mentat-inspector.py.log tail -f /var/mentat/log/mentat-enricher.py.log tail -f /var/mentat/log/mentat-storage.py.log grep ERROR /var/mentat/log/mentat-inspector.py.log grep ERROR /var/mentat/log/mentat-enricher.py.log grep ERROR /var/mentat/log/mentat-storage.py.log # Step 9: Access the web interface and check that everything is in order. # Step 10: If you migrated to another host and mangled with DNS, do not forget # to put things back as they were. .. _section-migration-scripts: Usefull scripts -------------------------------------------------------------------------------- This section contains usefull migration scripts for your convenience. We have used many scripts and cronjobs during migration of our production Mentat installation and here are those that might be usefull in general. mentatdb-dump-all.sh ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Install this script somewhere on the source Mentat host, for example as file ``/root/mentatdb-dump-all.sh``. You can use it to perform local dump of all Mentat `MongoDB `__ database collections that are relevant for migration. .. code-block:: shell #!/bin/bash # UTILITY SCRIPT FOR LOCAL DUMPS OF MONGODB DATABASE COLLECTIONS RELEVANT FOR # MIGRATION TO MENTAT-NG 2.X # # Author: Jan Mach # License: MIT # # Adjust these settings according to your needs: MENTATDUMPDIR="/var/tmp" MENTATDUMPFILENAME="mentatdb_dump_all" echo "" echo "[SRC] Cleanup after possible previous execution." rm -f "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}.tar.gz" rm -rf "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}" echo "" echo "[SRC] Preparing work environment." mkdir -p "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}" cd "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}" echo "" echo "[SRC] Dumping relevant database collections." mongodump --db mentat_stats --collection statistics mongodump --db mentat --collection users mongodump --db mentat --collection groups mongodump --db mentat --collection reports_ng cd "${MENTATDUMPDIR}" tar -czvf "${MENTATDUMPFILENAME}.tar.gz" "${MENTATDUMPFILENAME}" echo "" echo "[SRC] Post-execution cleanup." rm -rf "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}" echo "" echo "[SRC] Your database migration dump is ready: ${MENTATDUMPDIR}/${MENTATDUMPFILENAME}.tar.gz" mentat-sync-mongodb.sh ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Install this script somewhere on the target Mentat host, for example as file ``/root/mentat-sync-mongodb.sh``. You can use it to perform restoration of all essential Mentat `MongoDB `__ database collections. .. code-block:: shell #!/bin/bash # UTILITY SCRIPT FOR FETCHING LATEST MONGODB DATABASE DUMP FROM CURRENT PRODUCTION SERVER # AND IMPORTING IT INTO LOCAL MONGODB DATABASE DURING MIGRATION TO MENTAT-NG 2.X # # Author: Jan Mach # License: MIT # # Adjust these settings according to your needs: MENTATMONGODBDUMPSCRIPT="/root/mentatdb-dump-all.sh" MENTATPRODSSHUSER="root" MENTATPRODSSHSERVER="mentat.domain.org" MENTATDUMPDIR="/var/tmp" MENTATDUMPFILENAME="mentatdb_dump_all" echo "" echo "[TGT] Creating fresh dump on current production server." ssh "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}" "${MENTATMONGODBDUMPSCRIPT}" echo "" echo "[TGT] Cleanup after possible previous execution." rm -f "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}.tar.gz" rm -rf "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}" echo "" echo "[TGT] Fetching latest database dump from production server ${MENTATPRODSSHSERVER}." scp "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}:${MENTATDUMPDIR}/${MENTATDUMPFILENAME}.tar.gz" "${MENTATDUMPDIR}/" echo "" echo "[TGT] Extracting fetched database dump." cd "${MENTATDUMPDIR}" tar -xzvf "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}.tar.gz" echo "" echo "[TGT] Importing relevant database collections." cd "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}/dump" mongorestore --drop --db mentat_stats mentat_stats/statistics.bson mongorestore --drop --db mentat mentat/users.bson mongorestore --drop --db mentat mentat/groups.bson mongorestore --drop --db mentat mentat/reports_ng.bson echo "" echo "[TGT] Post-execution cleanup." rm -f "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}.tar.gz" rm -rf "${MENTATDUMPDIR}/${MENTATDUMPFILENAME}" Additionally you might want to install Cronjob for performing periodical dumps and then automatically pull and import those dumps on target Mentat host. This can be usefull for trial period before actual migration to test whether everything is working: .. code-block:: shell root@source:~# crontab -e # And add something similar to this: # Create, fetch and import latest MongoDB dump every four hours. 5 */4 * * * /root/mentat-sync-mongodb.sh mentat-sync-files.sh ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Install this script somewhere on the target Mentat host, for example as file ``/root/mentat-sync-files.sh``. You can use it to perform filesystem data migration and installation tasks. .. code-block:: shell #!/bin/bash # UTILITY SCRIPT FOR PERIODICAL MENTAT FILESYSTEM DATA SYCHRONIZATION AND # INSTALLATION DURING MIGRATION TO MENTAT-NG 2.X # # Author: Jan Mach # License: MIT # # Adjust these settings according to your needs: MENTATPRODSSHUSER="root" MENTATPRODSSHSERVER="mentat.domain.org" if [ "x${1}x" = "x--skip-installx" ]; then SKIPINSTALL="yes" else SKIPINSTALL="no" fi echo "" echo "[TGT] Synchronizing Mentat data directories (except reports)." rsync --progress --archive --update --delete --force --exclude=www --exclude=log --exclude=spool --exclude=reports --exclude=run --exclude=maintenance --exclude=cache "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}:/var/mentat/" /var/mentat/ echo "" echo "[TGT] Synchronizing Mentat report directories." rsync --progress --archive --update --delete --force --exclude=reporter "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}:/var/mentat/reports/" /var/mentat/reports if [ "${SKIPINSTALL}" = "yes" ]; then echo "[TGT] Skipping legacy report installation." else echo "" echo "[TGT] Installing reports generated by legacy reporter." rsync --progress --archive --update --delete --force /var/mentat/reports/reporter-ng/ /var/mentat/reports/reporter find /var/mentat/reports/reporter -type f -exec rename 's/_/-/g' {} \; /etc/mentat/scripts/mentat-reports-order.py echo "" echo "[TGT] Cleanup of local log, runlog, state and pstate files." find /var/mentat/log -name '*.log*' -delete find /var/mentat/run -name '*.runlog' -delete find /var/mentat/run -name '*.pstate' -delete find /var/mentat/run -name '*.state' -delete fi Additionally you might want to install Cronjob for performing periodical executions of this script to showrten the actual data migration time: .. code-block:: shell root@source:~# crontab -e # And add something similar to this: # Synchonize Mentat filesystem data every four hours. 35 * * * * /root/mentat-sync-files.sh --skip-install mentat-tweakdb.sql ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Install this script somewhere on the target Mentat host, for example as file ``/root/mentat-tweakdb.sql``. You can use it to perform additional tweaks of database contents, for exampple for changing the default reporting settings for all groups. .. code-block:: sql UPDATE settings_reporting SET timezone = 'Europe/Prague'; UPDATE settings_reporting SET locale = 'cs'; Then execute it like this: .. code-block:: shell psql -f mentat-tweakdb.sql mentat_main mentat-sync-config.sh ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Install this script somewhere on the target Mentat host, for example as file ``/root/mentat-sync-config.sh``. You can use it as inspiration for copying various important configuration files between old and new production servers. After executing this script you will find the most important configurations alongside the local ones for convenient migration process. .. code-block:: shell #!/bin/bash # UTILITY SCRIPT FOR FETCHING MOST IMPORTANT CONFIGURATION FROM CURRENT PRODUCTION # SERVER TO LOCAL ONE (MIGRATION TARGET) DURING MIGRATION TO MENTAT-NG 2.0 # # Author: Jan Mach # License: MIT # # Adjust these settings according to your needs: MENTATPRODSSHUSER="root" MENTATPRODSSHSERVER="mentat.domain.org" echo "" echo "[TGT] Switching Shibboleth configurations." rsync --progress --archive --update --delete --force "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}:/etc/shibboleth/" /etc/shibboleth-new/ chown _shibd:_shibd /etc/shibboleth-new/sp-*.pem cp -r /etc/shibboleth /etc/shibboleth-old rsync --progress --archive --update --delete --force /etc/shibboleth/ "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}:/etc/shibboleth-new/" ssh "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}" chown _shibd:_shibd /etc/shibboleth-new/sp-*.pem ssh "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}" cp -r /etc/shibboleth /etc/shibboleth-old echo "" echo "[TGT] Switching server certificates." rsync --progress --archive --update --delete --force "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}:/etc/ssl/servercert/" /etc/ssl/servercert-new/ cp -r /etc/ssl/servercert /etc/ssl/servercert-old rsync --progress --archive --update --delete --force /etc/ssl/servercert/ "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}:/etc/ssl/servercert-new/" ssh "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}" cp -r /etc/ssl/servercert /etc/ssl/servercert-old echo "" echo "[TGT] Copying Mentat configuration from current production server ${MENTATPRODSSHSERVER}." rsync --progress --archive --update --delete --force "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}:/etc/mentat/" /etc/mentat-oldprod/ echo "" echo "[TGT] Copying Warden receiver configuration from current production server ${MENTATPRODSSHSERVER}." rsync --progress --archive --update --delete --force "${MENTATPRODSSHUSER}@${MENTATPRODSSHSERVER}:/etc/warden_filer.cfg" /etc/warden_filer.cfg-new .. _section-migration-cleanup: Cleanup -------------------------------------------------------------------------------- When you are comfortable enough, after successfull migration you may remove old Perl based Mentat packages: .. code-block:: shell aptitude purge libcesnet-toolkit-perl mentat-client mentat-common mentat-dev mentat-hawat mentat-server Another component, that is not required for Mentat system ``2.x`` series is `MongoDB `__. Depending on the repository you have installed it from, you might execute command similar to these: .. code-block:: shell # Native Debian package: aptitude purge mongodb # MongoDB, inc. Debian package: aptitude purge mongodb-org Also make sure to remove now unnecessary APT source files: .. code-block:: shell rm /etc/apt/sources.list.d/cesnet-certs.list rm /etc/apt/sources.list.d/mentat.list rm /etc/apt/sources.list.d/mongodb-org.list Following directories are not used in Mentat version ``2.0.0`` and later: .. code-block:: shell rm -rf /var/mentat/reports/briefer rm -rf /var/mentat/reports/reporter-ng rm -rf /var/mentat/www .. _section-migration-next: What is next? -------------------------------------------------------------------------------- You have just successfully migrated Mentat system to latest version, so what is next? * If you want to take quick tour of the Mentat system, you might wish to read and follow the :ref:`section-quickstart` documentation page.