Development

This is the documentation for developers of the Mentat library itself, or developers of components and modules usable by or pluggable into the Mentat system.

Getting the code

We are using the Git SCM system to manage our codebase. Please follow instructions in section Installation guide - Installation from Git repository to install the project into your desired location. By following these instructions a Python virtual environment will be initialized for you, all requirements (including those required only for development) will be installed and finally the project itself will be installed locally in editable mode.

General guidelines

  • Let PEP 20 be the guide for your mind.

  • Let PEP 8 be the guide for your hand.

  • Let you and PEP 257 and PEP 287 be the guide for others.

  • Use Sphinx-doc format to document the code.

  • Pull and merge often.

  • Use devel branch for small updates and bugfixes.

  • For bigger features fork devel, merge after accepting, delete branch.

  • Use release branch only for code that is ready to be released into production.

  • Use master branch only for production level and stable code.

  • master and release branches must not break unittests, lint or build in general. devel branch should not.

  • Unless you have been explicitly allowed to, do not use master and release branches.

  • New feature should be accompanied with unit tests.

  • Do not introduce new dependencies into core library. Dependent code should go into its own submodule, so dependency can be runtime and enforced by system administrator if necessary, but not by library.

  • Reuse existing (even soft) dependencies. There is no need to use three competing IP address libraries. However, do not prevent application developer to use different one in his app, should he need to.

Development essentials

There is a project master Makefile in the root of the project repository which can perform various essential or useful development tasks. You can get the full list of all available make commands/targets by executing one of the following commands:

$ make
$ make help

Of course you need to have make utility installed on your system, on Debian-based system you can use following command:

$ aptitude install make

It is recommended to always install the project into Python virtual environment. If you have followed the installation instructions in section Installation guide - Installation from Git repository this virtual environment was set up for you in ./venv subdirectory. Before any development work do not forget to activate the environment with:

# Activate virtual environment before any development work:
$ . venv/bin/activate

# Deactivate virtual environment when it is not needed anymore with:
(venv) $ deactivate

Development prerequisites

There are several development prerequisites, that already have to be present on your development machine. These prerequisites are not installed automatically for you, because the installation is too complex, too customizable or simply best to be performed by the user himself. These prerequisites currently are:

  • Python 3: Please use version similar to current stable Python3 release on current stable Debian release (current).

  • Pip: Python package manager, we recommend installation with get-pip.py to get the latest release, you may sometimes encounter weird bugs when installing newer packages with old versions of pip.

  • Yarn: NPM package manager for web interface libraries. This package manager is responsible for managing frontend development libraries like jQuery, Bootstrap, D3 etc.

  • Grunt: JavaScript task runner for web interface development. It is responsible for taking care of frontend related tasks like JavaScript and CSS minification

  • PostgreSQL 11: Relational database, used for persistent data storage. Please use version 11 wherever possible.

You can check for presence of all of these dependencies with this handy make target:

# Check for presence of all prerequisites:
$ make deps-prerequisites

In case you get any errors please follow the official documentation to install the missing prerequisite.

Dependencies

There is a number of makefile targets called deps and deps- that are responsible for helping with dependency management. Please study the make help output to view the list of available targets.

If your code requires some additional third party dependencies please follow these procedures:

  • Debian dependencies

    • Debian dependencies must be specified in the deploy/mentat/ctrl/control.tmpl file. Please study the documentation to correctly use Debian dependency system.

  • Python dependencies

    • Use pip command to manage Python dependencies.

    • Dependencies that are always required must be listed in conf/requirements.pip file (including version) and in conf/requirements-latest.pip file (without version).

    • Dependencies that are required only for development must be listed in conf/requirements-dev.pip file (including version) and in conf/requirements-latest-dev.pip file (without version).

    • In this project it is preferred to use pip for native Python dependencies. In some cases you may however choose to use Debian package for pulling in the required library instead.

  • Web interface dependencies

    • Use yarn command to manage frontend dependencies.

Example workflow for adding Python dependency:

# Install library locally:
(venv) $ pip install flask

# Get the version of the library:
(venv) $ pip freeze | grep -i flask

# Now write the library name with version to `conf/requirements.pip` and
# without version to `conf/requirements-latest.pip`.

# In case the library is required only for development write the library name
# with version to `conf/requirements-dev.pip` and without version to
# `conf/requirements-latest-dev.pip`.

# Make sure the dependency gets installed also using the makefile target:
(venv) $ make deps-python
(venv) $ make deps-python-dev

Example workflow for adding frontend dependency:

# Install dependency with yarn:
(venv) $ yarn add jquery

# Install development dependency with yarn:
(venv) $ yarn add grunt --dev

# Make sure the dependency gets installed also using the makefile target:
(venv) $ make deps-webui

For upgrading all the dependencies to latest versions you may use following make targets:

# Activate virtual environment before any development work:
$ . venv/bin/activate

(venv) $ make deps-python-upgrade
(venv) $ make deps-python-upgrade-dev
(venv) $ make deps-webui-upgrade

Running development version of Mentat system

Mentat system can be executed from within the Git repository. We worked hard to make sure it will not wander around your filesystem. The key to make everything work is the environment variable APP_ROOT_PATH, which controls the base root of the application. It is basically a soft implementation of a very lightweight chroot. Application can adjust various filesystem paths to the value of this variable. During the preparation of development environment a local ./chroot subdirectory was prepared automatically for you. You may use it in a following fashion:

# A: Use it manually, be carefull to provide the APP_ROOR_PATH as absolute path:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command start

# B: For your convenience there is also a makefile target:
(venv) $ make run-mentat-dev

For your convenience there is a very handy makefile target ctrl-mentat-dev, which is capable of passing commands to your local development instance of mentat-controller.py. You may use it in a following fashion:

# This:
(venv) $ make ctrl-mentat-dev COMMAND=start
# Is same as this:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command start

# This:
(venv) $ make ctrl-mentat-dev COMMAND=status
# Is same as this:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command status

# This:
(venv) $ make ctrl-mentat-dev COMMAND=stop
# Is same as this:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command stop

Make sure to read following documentation sections to understand the usage of the Mentat system and its various components:

Running development web server

The web interface for this project is written in excellent Flask microframework, that comes with built-in webserver for development. It can be launched in following ways:

# A: You may use the Flask built-in command in a following way:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) FLASK_APP=hawat FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(realpath ./hawat.local.conf) flask run

# B: You may custom command line interface to launch webserver in development
# mode and with development configuration:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(realpath ./hawat.local.conf) hawat-cli run

# C: Use following makefile target to do the same as the three above with less
# typing:
(venv) $ make run-webui-dev

There are following environment variables you may use to tweak the application launch according to your needs:

  • FLASK_DEBUG

    This configuration controls state of the internal debugger independently on the FLASK_ENV setting. It is a boolean value and should be either True or False. Default value is False.

  • FLASK_ENV

    This configuration controls application environment setting. This is a string value and should be either development or production. Default value is production.

  • FLASK_CONFIG

    This configuration controls the name of the configuration class from mydojo.config module that will be used to configure the application. Valid value is one of the mydojo.config.CONFIG_MAP. Default value is default.

  • FLASK_CONFIG_FILE

    This configuration controls the name of the configuration file that will be used to further configure the application. Values in this file are applied last and will override anything in the configuration classes from mydojo.config. Default value is empty. It must point to existing file if set, otherwise an exception will be raised. Please use absolute path to the file to avoid any surprises.

Note

The FLASK_CONFIG_FILE is especially handy for customizing the local application configuration during development process or during deployment.

For more information please study following resources:

Documentation

The project documentation consists of the part generated directly from the source code docstrings and of the part written manually. It is generated using the Sphinx-doc tool into various formats. Please use RST markup features where appropriate to increase readability and cross-reference to related content. It should however still be possible to view the documentation of all Python modules in Pythonic way via pydoc3 and the result should still be more or less readable. Please test it immediately with:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ pydoc3 ./path/to/module.py

You may generate and review the documentation locally by executing the following command:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ make docs

# View the documentation in your default web browser:
(venv) $ make docs-view

# When necessary you may also remove all documentation related artifacts and
# rebuild:
(venv) $ make clean-build-docs
(venv) $ make docs
(venv) $ make docs-view

Documentation will be generated into doc/sphinx/_build/html/manual.html.

Important resources for you to study:

Internationalization and translations

The web interface and some other parts of the system are localized to provide best experience for target user. Following libraries are used to accomplish this task:

The web interface translations are included in the hawat module. The most important files are following:

  • lib/hawat/babel.cfg - Babel configuration file

  • lib/hawat/messages.pot - Extracted translations, generated automatically

  • lib/hawat/translations/ - Directory containing translations to various languages

Strings in the python source code are marked for translation when you wrap them in one of the following functions: gettext(), lazy_gettext(), tr_(). The last one is defined internally and is used for translating constants or enums. Strings in the Jinja2 templates are marked for translation when you wrap them with gettext() or _() functions.

After adding new strings into the web interface that will need translating please follow this procedure:

# Pull (extract and update) all translation strings into message catalogs:
(venv) $ make hpybabel-update

# Now please edit the translation files. For example for czech locale please
# edit file ``lib/hawat/translations/cs/messages.po``.

# When you are happy with your translations compile the message catalogs with:
(venv) $ make hpybabel-compile

Similarly the make mpybabel-update and make hpybabel-compile targets can be used to compile translations of Mentat reporting templates. These are located in following directories:

  • conf/templates/informant/: Report templates for mentat-informant.py.

  • conf/templates/reporter/: Report templates for mentat-reporter.py.

  • conf/templates/utest/: Templates for library unit tests.

Checking code with Pyflakes

You may check the whole codebase with Pyflakes tool by executing following command:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ make pyflakes

Or you may check just the single file by executing following command:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ cd lib
(venv) $ pyflakes path/to/module.py

Important resources:

Checking code with Pylint

You may check the whole codebase with Pylint tool by executing following command:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ make pylint

Or you may check just the single file by executing following command:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ cd lib
(venv) $ pylint --rcfile=../.pylintrc-lib path/to/module.py

Important resources:

Running unit tests

You may run prepared unit tests on the whole codebase by executing the following command:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ make test

Important resources:

Building web interface

The web interface development requires certain specific tasks like copying third party libraries from node_modules directory to correct locations, JavaScript and CSS minifications etc. When you are developing web interface following makefile target will be very handy to you:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ make build-webui

Database schema migrations

Event database migrations

Due to the performance reasons the event database abstraction layer is implemented directly on top of the psycopg2 driver. To be consistent with metadata database migrations we are using separate configured instance of Alembic database migration utility. The migration environment is located in migrations-events subdirectory.

To create new migration during development follow these steps:

cd migrations-events
alembic revision -m "revision description"

Now edit the generated revision file to suit your needs. You may wish to use following resources as reference:

Migration can be then invoked locally from within the migration environment directory:

cd migrations-events
alembic upgrade head
alembic history

To enable execution of database migrations on target systems after installation from package there is a simple wrapper script /etc/mentat/scripts/sqldb-migrate-e.sh:

/etc/mentat/scripts/sqldb-migrate-e.sh upgrade head
/etc/mentat/scripts/sqldb-migrate-e.sh history

Important resources:

Metadata database migrations

Important resources:

Versioning

This project uses the semantic versioning. When the production level packages are being built and deployed, the automated build system takes the project version directly from following files (paths are relative to project root):

  • lib/mentat/__init__.py

When building the release or development level packages, the automated build system appends an internal build number as additional subversion. This way each build produces unique version string and unique package. This feature can be used during development to reduce the need for incrementing the version numbers manually between each builds.

Tagging

Each major and minor version release must be tagged within the repository. Please use only annotated or signed tags and provide short comment for the release. Before tagging please view existing tags so that you can attempt to maintain the style of the tag messages.

# List all existing tags
git tag -l -n999

# Create new annotated tag and provide message
git tag -a v2.0.0

# Push tags to remote servers (if you are permitted to do so)
git push origin v2.0.0
git push buildbot v2.0.0

# Number of commits between last two versions:
$ git rev-list --count v1.0.0..v0.0.1

# Total changes between last two versions:
$ git log --numstat --pretty="%H" v1.0.0..v0.0.1 | awk 'NF==3 {plus+=$1; minus+=$2} END {printf("+%d, -%d\n", plus, minus)}'

Building Python packages

If you want to build native Python packages locally please use following makefile target:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ make build-whl

Generated packages will be placed into ./dist subdirectory.

Building Debian packages

If you want to build native Debian packages locally please use following makefile target:

# Always make sure your virtual environment is activated:
$ . venv/bin/activate

# Run tests:
(venv) $ make build-deb

Generated packages will be placed into ./deploy/mentat/ subdirectory.

Examples

Implementing example daemon module

Before going further please read the documentation and study source code of following libraries:

Now save following content into the file /etc/mentat/examples/mentat-demopiper.py:

import pyzenkit
import mentat.const
import mentat.daemon.piper

class DemoPrintComponent(pyzenkit.zendaemon.ZenDaemonComponent):

    def get_events(self):
        return [
            {
                'event': 'message_process',
                'callback': self.cbk_event_message_process,
                'prepend': False
            }
        ]

    def cbk_event_message_process(self, daemon, args):
        daemon.logger.info(
            "Processing message: '{}': '{}'".format(
                args['id'], str(args['data']).strip()
            )
        )
        daemon.queue.schedule('message_commit', args)
        self.inc_statistic('cnt_printed')
        return (daemon.FLAG_CONTINUE, None)

class DemoPiperDaemon(mentat.daemon.piper.PiperDaemon):

    def __init__(self):
        super().__init__(
            name        = 'mentat-demopiper.py',
            description = 'DemoPiperDaemon - Demonstration daemon',
            path_bin    = '/usr/local/bin',
            path_cfg    = '/tmp',
            path_log    = '/var/mentat/log',
            path_run    = '/var/mentat/run',
            path_tmp    = '/tmp',

            default_config_dir    = None,
            default_queue_in_dir  = '/var/mentat/spool/mentat-demopiper.py',
            default_queue_out_dir = None,

            schedule = [
                ('message_enqueue', {'data': '{"testA1": 1, "testA2": 2}'}),
                ('message_enqueue', {'data': '{"testB1": 1, "testB2": 2}'}),
                (mentat.const.DFLT_EVENT_START,)
            ],
            schedule_after = [
                (mentat.const.DFLT_INTERVAL_STATISTICS, mentat.const.DFLT_EVENT_LOG_STATISTICS)
            ],

            components = [
                DemoPrintComponent()
            ]
        )

if __name__ == "__main__":
DemoPiperDaemon().run()

Now let`s create configuration file /tmp/mentat-demopiper.py. It must contain a valid JSON dictionary, that may or may not be empty, so it must contain at least following:

# Configuration for module
{}

Note, that you may use single-line comments. Any line, that beginswith # is ignored. However there may be only white characters on the line before the comment.

Now add your module somewhere into the message processing pipeline. For the simplicity let`s put it after the default mentat-storage.py module, so that we have to make only one change in existing configuration files. Replace the existing value for queue_out_dir with following line in /etc/mentat/mentat-storage.py.conf file:

"queue_out_dir": "/var/mentat/spool/mentat-demopiper.py",

And finally add your new module to the /etc/mentat/mentat-controller.py.conf file into the key modules, so that you can start and stop it together with the rest of the modules:

    {
    "exec": "mentat-demopiper.py",
    "args": [
        # Enable debug information before daemonization
        "--debug"
        # Force logging level ['debug', 'info', 'warning', 'error', 'critical']
        "--log-level=debug"
    ]
},

Place it on top of the list so that it gets started first since it is the last module in the message processing chain.

Now everything is ready for you to start everything up:

# Create symlink to example
ln -s /etc/mentat/examples/mentat-demopiper.py /usr/local/bin/mentat-demopiper.py
# Stop all currently running components
mentat-controller.py --command stop
# Start all currently components
mentat-controller.py --command start
# Generate test messages
mentat-ideagen.py --count 10
# View log file
tail -f /var/mentat/log/mentat-demopiper.py.log