pyzenkit.baseapp module

This module provides base implementation of generic console application represented by the pyzenkit.baseapp.BaseApp class with many usefull features including (but not limited to) the following:

Application configuration service

The base application provides tools for loading configurations from multiple sources and merging them all into single dictionary. This is then available to developers as public class attribute config.

Currently the following configuration sources are available:

  • Optional configuration via configuration directory. All JSON files in given directory are loaded and then merged together.

  • Optional configuration via single JSON configuration file.

  • Optional configuration via command line arguments and options.

Command line argument parsing service

The base application preconfigures an instance of standard argparse.ArgumentParser class for parsing command line arguments and prepopulates in with a set of built-in options and arguments. This instance can be then further modified and enhanced in subclass by the user.

Logging service

The base application is capable of automated setup of logging.Logger logging service. Logger parameters like threshold level, or target file name are fully configurable. Currently following destinations are supported:

  • Optional logging to console.

  • Optional logging to text file.

Persistent state service

The base application contains optional persistent state feature, which is intended for storing or passing data between multiple executions of the same application. The feature is implemented as a simple dictionary, that is populated from simple JSON file on startup and written back on teardown.

Application runlog service

The base application provides optional runlog service, which is a intended to provide storage for relevant data and results during the processing and enable further analysis later. The feature is implemented as simple dictionary, that is written into JSON file on teardown.

Plugin system

The base application provides tools for writing and using plugins, that can be used to further enhance the functionality of application and improve code reusability by composing the application from smaller building blocks.

Application actions

The base application provides tools for quick actions. These actions are intended to be used for global application management tasks such as vieving or validating configuration without executing the application itself, listing or evaluating runlogs and so on. There is a number of built-in actions and more can be implemented very easily.

The application is designed to provide all of these usefull features by default and with as less as possible work, while also maintaining high customizability. This goal is achived by following measures:

  • Most of the hardcoded features are somehow customizable by configuration file keys or command line options.

  • There are many callback hooks prepared for subclasses, that can be used to add the desired functionality.

  • If you are more familiar with the code, you may override the default implementation of almost any method and provide your own functionality, or call the parent implementation at some point in the new method.

Please browse the source code, there is an example implementation in pyzenkit.baseapp.DemoBaseApp class.

Application usage modes

Applications created using this framework can be utilized in two work modes:

  • run

  • plugin

In the run mode all application features are initialized and configured and desired action or application processing code is immediatelly executed.

In the plugin mode the application is only initialized and configured and any other interactions must be performed manually by calling appropriate methods. This approach enables users to plug one apllication into another one on a wider scope. One example use case of this feature may be the implementation of an user command line interface that controls multiple applications (for example like git executable controls almost everything in Git universe).

Application actions

Applications created based on this framework come with built-in support for actions. The action is a designation for some kind of support task, that does not relate to actual purpose of the application. The actions however are executed within the same environment and have access to the whole application, so they are perfect for simple tasks like configuration validation. The actions are also much more simple and when executing an action the application does not go through all life-cycle stages (see below).

Currently following actions are built-in and supported:

config-view

Display current configuration tree.

runlog-dump

Simple dump of chosen application runlog.

runlog-view

View chosen application runlog (nicer display with calculated statistics).

runlogs-dump

Simple dump of all application runlogs.

runlogs-list

View list of all application runlogs.

runlogs-evaluate

View evaluated statistics of all application runlogs.

Actions may be executed by selecting desired action by command line argument:

path/to/baseapp.py --action config-view
path/to/baseapp.py --action=config-view

Action may be selected permanently inside configuration file under the dictionary key action.

Application life-cycle

Depending on the mode of operation (run or plugin) the application code goes through a different set of stages during its life span. See subsections below for more details on each life-cycle stage. The life-cycle is implemented partly in the pyzenkit.baseapp.BaseApp.__init__() method and mostly in pyzenkit.baseapp.BaseApp.run() method.

In the plugin mode the following stages are performed:

  • __init__

  • setup

In the run mode the following stages are performed in case some action is being handled:

  • __init__

  • setup

  • action

In the run mode the following stages are performed in case of normal processing:

  • __init__

  • setup

  • process

  • evaluate

  • teardown

Stage __init__

The __init__ stage is responsible for creating and basic default initialization of application object. No exception should occur or be raised during the initialization stage and the code should always work regardles of environment setup, so no files should be opened, etc. Any exception during this stage will intentionally not get handled in any way and will result in full traceback dump and immediate application termination.

All more advanced setup tasks should be performed during the setup stage, which is capable of intelligent catching and displaying/logging of any exceptions.

This stage is implemented in the BaseApp.__init__() method and there are following substages in this stage:

  • init command line argument parser: BaseApp._init_argparser()

  • parse command line arguments: BaseApp._parse_cli_arguments()

  • initialize application name: BaseApp._init_name()

  • initialize filesystem paths: BaseApp._init_paths()

  • initialize application runlog: BaseApp._init_runlog()

  • initialize default configurations: BaseApp._init_config()

  • subclass hook for additional initializations: BaseApp._sub_stage_init()

Any of the previous substages may be overriden in a subclass to enhance or alter the functionality, but always be sure of what you are doing.

Stage setup

The setup stage is responsible for bootstrapping and configuring of the whole application. Any exception, that is the instance of ZenAppSetupException will be catched, only simple message will be displayed to the user and application will terminate. This use case represents the case when the error is on the user`s side, for example non-existent configuration file, etc. In any other case the application will terminate with full traceback print.

This stage is implemented in the BaseApp._stage_setup() method and there are following substages in this stage:

  • setup configuration: BaseApp._stage_setup_configuration()

  • setup user and group privileges: BaseApp._stage_setup_privileges()

  • setup logging: BaseApp._stage_setup_logging()

  • setup persistent state: BaseApp._stage_setup_pstate()

  • setup plugins: BaseApp._stage_setup_plugins()

  • subclass hook for additional setup: BaseApp._sub_stage_setup()

  • setup dump: BaseApp._stage_setup_dump()

Any of the previous substages may be overriden in a subclass to enhance or alter the functionality, but always be sure of what you are doing.

Stage action

The action stage takes care of executing built-in actions. See appropriate section above for list of all available built-in actions.

More actions can be implemented very easily. The action callback methods just have to follow these requirements to be autodetected by the application engine:

  • Action callback must be method without any mandatory arguments.

  • Action callback method name must begin with cbk_action_ prefix.

  • Action name in the method name after the prefix must also be snake_cased`.

  • Action name will be calculated by replacing _ with -.

Following are examples of valid action callbacks:

cbk_action_test(self):         # Will be mapped to 'test' action.
cbk_action_another_test(self): # Will be mapped to 'another-test' action.

When a method is implemented according to these guidelines, it will be automatically recognized and executable.

This stage is implemented in the BaseApp._stage_action() method.

Please see the implementation of existing built-in actions for examples.

Stage process

The process stage is supposed to perform the required task(s). This stage is implemented in the BaseApp._stage_process() method. It is a wrapper method, that takes care of exception catching and the application must provide its own implementation of template method BaseApp._sub_stage_process(), which is being called from the wrapper and should contain actual processing code.

Stage evaluate

The evaluate stage is supposed to perform any required evaluations of current runlog. Currently, there are no built-in evaluations.

Stage teardown

The teardown stage is supposed to perform any cleanup tasks before the application exits.

This stage is implemented in the BaseApp._stage_teardown() method and there are following substages in this stage:

  • subclass hook for additional teardown actions: BaseApp._sub_stage_teardown()

  • save persistent state: BaseApp._stage_teardown_pstate()

  • save runlog: BaseApp._stage_teardown_runlog()

Programming API

The application features mentioned in this document are available to the user/developer either as public attributes of the pyzenkit.baseapp.BaseApp class, or as public methods:

  • public attributes:

  • public methods:

    • BaseApp.c() - Shortcut method for accessing the self.config.get(key, default)

    • BaseApp.cc() - Shortcut method for accessing the self.config[self.CORE].get(key, default)

    • BaseApp.p() - Method for printing to terminal honoring the verbose setting.

    • BaseApp.dbgout() - Method for printing additional debug messages honoring the debug setting.

    • BaseApp.excout() - Method for printing exception without traceback and terminating the application.

    • BaseApp.error() - Register given error into application.

    • BaseApp.json_dump() - Utility method for dumping data structure into JSON string.

    • BaseApp.json_load() - Utility method for loading JSON file.

    • BaseApp.json_save() - Utility method for writing data structure into JSON file.

Subclass extension hooks

The base application provides following extension hooks, that can be used in subclasses to enhance the base functionality. There is one hook for each application life-cycle stage:

  • BaseApp._sub_stage_init()

  • BaseApp._sub_stage_setup()

  • BaseApp._sub_stage_process()

  • BaseApp._sub_stage_teardown()

There are also couple of methods for runlog analysis and evaluation, that can be used to enhance the default output of runlog-* built-in actions:

  • BaseApp._sub_runlog_analyze()

  • BaseApp._sub_runlog_format_analysis()

  • BaseApp._sub_runlogs_evaluate()

  • BaseApp._sub_runlogs_format_evaluation()

class pyzenkit.baseapp.BaseApp(**kwargs)[source]

Bases: object

Base implementation of generic executable application. This class attempts to provide robust and stable framework, which can be used to writing all kinds of scripts or daemons. Although is is usable, this is however a low level framework and should not be used directly, use the pyzenkit.zenscript or pyzenkit.zendaemon modules for writing custom scripts or daemons respectively. That being said, the pyzenkit.baseapp.DemoBaseApp class is an example implementation of using this class directly without any additional overhead.

CONFIG_ACTION = 'action'
CONFIG_CFG_DIR = 'config_dir'
CONFIG_CFG_DIR_S = 'config_dir_silent'
CONFIG_CFG_FILE = 'config_file'
CONFIG_CFG_FILE_S = 'config_file_silent'
CONFIG_DEBUG = 'debug'
CONFIG_GROUP = 'group'
CONFIG_INPUT = 'input'
CONFIG_LIMIT = 'limit'
CONFIG_LOG_FILE = 'log_file'
CONFIG_LOG_LEVEL = 'log_level'
CONFIG_NAME = 'name'
CONFIG_PID_FILE = 'pid_file'
CONFIG_PLUGINS = 'plugins'
CONFIG_PSTATE_DUMP = 'pstate_dump'
CONFIG_PSTATE_FILE = 'pstate_file'
CONFIG_PSTATE_LOG = 'pstate_log'
CONFIG_QUIET = 'quiet'
CONFIG_RUNLOG_DIR = 'runlog_dir'
CONFIG_RUNLOG_DUMP = 'runlog_dump'
CONFIG_RUNLOG_LOG = 'runlog_log'
CONFIG_USER = 'user'
CONFIG_VERBOSITY = 'verbosity'
CORE = '__core__'
CORE_LOGGING = 'logging'
CORE_LOGGING_LEVEL = 'level'
CORE_LOGGING_LEVELC = 'level_console'
CORE_LOGGING_LEVELF = 'level_file'
CORE_LOGGING_TOCONS = 'to_console'
CORE_LOGGING_TOFILE = 'to_file'
CORE_PSTATE = 'pstate'
CORE_PSTATE_SAVE = 'save'
CORE_RUNLOG = 'runlog'
CORE_RUNLOG_SAVE = 'save'
FLAG_DEBUG = False
PATH_BIN = 'bin'
PATH_CFG = 'cfg'
PATH_LOG = 'log'
PATH_RUN = 'run'
PATH_TMP = 'tmp'
PATH_VAR = 'var'
PTRN_ACTION_CBK = 'cbk_action_'
PTRN_APP_NAME = '^[_a-zA-Z][-_a-zA-Z0-9.]*$'
RC_FAILURE = 1
RC_SUCCESS = 0
RESULT_FAILURE = 'failure'
RESULT_SUCCESS = 'success'
RLANKEY_AGE = 'age'
RLANKEY_COMMAND = 'command'
RLANKEY_DURATIONS = 'durations'
RLANKEY_DURPOST = 'dur_post'
RLANKEY_DURPRE = 'dur_pre'
RLANKEY_DURPROC = 'dur_proc'
RLANKEY_DURRUN = 'dur_run'
RLANKEY_EFFECTIVITY = 'effectivity'
RLANKEY_LABEL = 'label'
RLANKEY_RESULT = 'result'
RLANKEY_RUNLOG = 'runlog'
RLEVKEY_ANALYSES = 'analyses'
RLEVKEY_AVGDURPROC = 'avg_dur_proc'
RLEVKEY_AVGDURRUN = 'avg_dur_run'
RLEVKEY_AVGEFFECT = 'avg_effectivity'
RLEVKEY_MAXDURPROC = 'max_dur_proc'
RLEVKEY_MAXDURRUN = 'max_dur_run'
RLEVKEY_MAXEFFECT = 'max_effectivity'
RLEVKEY_MINDURPROC = 'min_dur_proc'
RLEVKEY_MINDURRUN = 'min_dur_run'
RLEVKEY_MINEFFECT = 'min_effectivity'
RLKEY_ARGV = 'argv'
RLKEY_ERRORS = 'errors'
RLKEY_NAME = 'name'
RLKEY_PID = 'pid'
RLKEY_RC = 'rc'
RLKEY_RESULT = 'result'
RLKEY_TMARKS = 'time_marks'
RLKEY_TS = 'ts'
RLKEY_TSFSF = 'ts_fsf'
RLKEY_TSSTR = 'ts_str'
argparser

[PUBLIC] Initialize command line argument parser.

c(key, default=None)[source]

Shortcut method: Get given configuration value, shortcut for:

self.config.get(key, default)

Parameters
  • key (str) – Name of the configuration value.

  • default – Default value to be returned when key is not set.

Returns

Configuration value fogr given key.

cbk_action_config_view()[source]

ACTION: Parse and view application configurations.

cbk_action_runlog_dump()[source]

ACTION: Dump given application runlog.

cbk_action_runlog_view()[source]

ACTION: View details of given application runlog.

cbk_action_runlogs_dump()[source]

ACTION: View list of all available application runlogs.

cbk_action_runlogs_evaluate()[source]

ACTION: Evaluate previous application runlogs.

cbk_action_runlogs_list()[source]

ACTION: View list of all available application runlogs.

cc(key, default=None)[source]

Shortcut method: Get given core configuration value, shortcut for:

self.config[self.CORE].get(key, default)

Core configurations are special configurations under configuration key __CORE__, which may only either be hardcoded, or calculated from other configurations.

Parameters
  • key (str) – Name of the core configuration value.

  • default – Default value to be returned when key is not set.

Returns

Core configuration value fogr given key.

config

[PUBLIC] Application configuration dictionary.

static dbgout(message)[source]

Routine for printing additional debug messages. The given message will be printed only in case the static class variable FLAG_DEBUG flag is set to True. This can be done either by explicit assignment in code, or using command line argument --debug, which is evaluated ASAP and sets the variable to True. The message will be printed to sys.stderr.

Parameters

message (str) – Message do be written.

description

[PUBLIC] Default application help description.

static draw_progress_bar(percent, bar_len=50)[source]

Draw progress bar on standard output terminal.

error(error, retc=None, trcb=None)[source]

Register given error, that occured during application run. Registering in the case of this method means printing the error message to logging facility, storing the message within the appropriate runlog data structure, generating the traceback when required and altering the runlog result and return code attributes accordingly.

Parameters
  • error (str) – Error message to be written.

  • retc (int) – Requested return code with which to terminate the application.

  • trcb (Exception) – Optional exception object.

static excout(exception, retc=None)[source]

Routine for displaying the exception message to the user without traceback and terminating the application. This routine is intended to display information about application errors, that are not caused by the application code itself (like missing configuration file, non-writable directories, etc.) and that can not be logged because of the fact, that the logging service was not yet initialized. For that reason this method is used to handle exceptions during the __init__ and setup stages.

Parameters
  • exception (Exception) – Exception object.

  • retc (int) – Requested return code with which to terminate the application.

execute_command(command, can_fail=False)[source]

Execute given shell command.

static format_progress_bar(percent, bar_len=50)[source]

Format progress bar from given values.

classmethod get_resource_path(fs_path, *more_chunks)[source]

Return filesystem path to application resource with APP_ROOT_PATH taken into consideration. If fs_path is absolute the APP_ROOT_PATH will be ignored as usual.

classmethod get_resource_path_fr(fs_path, *more_chunks)[source]

Force given application filesystem path to be relative to APP_ROOT_PATH.

static json_dump(data, **kwargs)[source]

Dump given data structure into JSON string.

Parameters
Returns

Data structure as JSON string.

Return type

str

static json_load(json_file)[source]

Load data structure from given JSON file.

Parameters

json_file (str) – Name of the JSON file to read from.

Returns

Loaded data structure.

Return type

dict

static json_save(json_file, data, **kwargs)[source]

Save given data structure into given JSON file.

Parameters
  • json_file (str) – Name of the JSON file to write to.

  • data (dict) – Data to be dumped to JSON.

  • kwargs – Optional arguments to pass to pyzenkit.jsonconf.json_save() method.

Returns

Always returns True.

Return type

bool

logger

[PUBLIC] Preconfigured logging.Logger object.

name

[PUBLIC] Name of the application, autodetected, or forced by object constructor arguments.

p(string, level=0)[source]

Print given string to sys.stdout with respect to quiet and verbosity settings.

Parameters
  • string (str) – String to print.

  • level (int) – Required minimal verbosity level to print the message.

paths

[PUBLIC] Application paths that will be used to construct various absolute file paths.

plugin()[source]

APPLICATION MODE: Plugin mode - Main processing method.

This method allows the object to be used as plugin within larger framework. Only the necessary setup is performed.

pstate

[PUBLIC] Persistent state dictionary.

retc

[PUBLIC] Final return code as integer.

run()[source]

APPLICATION MODE: Standalone application mode - Main processing method.

Run as standalone application, performs all stages of object life-cycle:

1. setup stage 2.1 action stage 2.2.1 processing stage 2.2.2 evaluation stage 2.2.3 teardown stage

runlog

[PUBLIC] Application processing runlog.

runlog_analyze(runlog)[source]

Analyze given runlog.

runlog_format_analysis(analysis)[source]

Format given runlog analysis.

runlogs_evaluate(runlogs)[source]

Evaluate given runlogs.

runlogs_format_evaluation(evaluation)[source]

Format runlog evaluation.

runlogs_list(**kwargs)[source]

List all available runlogs.

time_mark(ident, descr)[source]

Mark current time with additional identifier and description to application runlog.

Parameters
  • ident (str) – Time mark identifier.

  • descr (str) – Time mark description.

Returns

Time mark data structure.

Return type

dict

class pyzenkit.baseapp.DemoBaseApp(name=None, description=None)[source]

Bases: BaseApp

Minimalistic class for demonstration purposes. Study implementation of this class for tutorial on how to use this framework.

exception pyzenkit.baseapp.ZenAppEvaluateException(description, **params)[source]

Bases: ZenAppException

Describes problems or errors that occur during the evaluate phase.

exception pyzenkit.baseapp.ZenAppException(description, **params)[source]

Bases: Exception

Base class for all ZenApp custom exceptions.

When appropriate, these exceptions will be catched, error will be displayed to the user and the application will attempt to gracefully terminate without dumping the traceback visibly to the user. These exceptions should be used for anticipated errors, which can occur during normal application execution and do not mean there is anything wrong with the code itself, for example missing configuration file, etc…

class pyzenkit.baseapp.ZenAppPlugin[source]

Bases: object

Base class for all ZenApp application plugins. Plugins can be used to further enhance the code reusability by composing the application from smaller building blocks.

configure(app)[source]

Callback to be called during configuration phase (after initialization).

init_argparser(app, argparser, **kwargs)[source]

Callback to be called during argparser initialization phase.

init_config(app, config, **kwargs)[source]

Callback to be called during default configuration initialization phase.

init_runlog(app, runlog, **kwargs)[source]

Callback to be called during runlog initialization phase.

setup(app)[source]

Callback to be called during setup phase (after setup).

exception pyzenkit.baseapp.ZenAppProcessException(description, **params)[source]

Bases: ZenAppException

Describes problems or errors that occur during the process phase.

exception pyzenkit.baseapp.ZenAppSetupException(description, **params)[source]

Bases: ZenAppException

Describes problems or errors that occur during the setup phase.

exception pyzenkit.baseapp.ZenAppTeardownException(description, **params)[source]

Bases: ZenAppException

Describes problems or errors that occur during the teardown phase.