pyzenkit.zendaemon module

This module provides base implementation of daemon service represented by the pyzenkit.zendaemon.ZenDaemon class. It builds on top of pyzenkit.baseapp module and adds couple of other usefull features:

  • Fully automated daemonization process.

  • Event driven design.

  • Support for handling arbitrary signals.

  • Support for modularity with daemon components.

Daemonization

Daemonization is a process of transforming foreground application into a background always running service. The pyzenkit.zendaemon.ZenDaemon class has this feature built in and configurable with command line options, or configuration files/directories. Please see documentation page section-pyzenkit-configuration.

Daemonization is implemented on top of the pyzenkit.daemonizer utility library, please refer to its documentation for more details.

Event driven design and event queue

The daemon application has the event driven design. The ZenDaemon._sub_stage_process() method is implemented to perform an infinite event loop. There are events being emited from different parts of the application, which are then being ordered into event queue. Each of these events is then handled with appropriate callback method.

Event callback methods must be registered into daemon application to be recognized. Multiple event callbacks my be registered for certain single event. In this case those callbacks will be called in order of registration and a result of the previous one will be passed as input of the next. In other words callbacks form a pipeline and event will be pushed through that. Each callback method has the opportunity to break the pipeline/chain by returning apropriate flag.

The naming convention for event callback method is the following:

  • Event callback must be method, which accepts reference to pyzenkit.zendaemon.ZenDaemon daemon as first argument and dict args as second argument.

  • Event callback method name must begin with cbk_event_ prefix.

  • Event name in method name after the prefix must also be snake_cased`.

Note, that event name in callback method name is not used in any way for mapping callbacks to events (like in the case of actions), the callbacks are explicitly registered to handle particular events. It is however a great best practice and it is very clear then which callback handles which event.

Following are examples of valid event callbacks:

cbk_event_test(self, daemon, args)
cbk_event_another_test(self, daemon, args)

Each daemon application has an instance of the EventQueueManager as public attribute, which represents the event queue. There are following methods available for scheduling events into the queue:

Signal handling

Each daemon service should be capable of receiving and handling external signals. Currently support for following signals is built-in:

SIGHUP

Reload configuration and reconfigure the application.

Note

This feature is not yet fully implemented.

SIGINT

Finish up all current work and attempt to perform gracefull stop. In case you send this signal multiple times (currently the magical number is three), you will force the event loop to be exited prematurelly. This feature can be usefull in case you are shutting the daemon down by hand and the shutdown process takes too long to complete. Another fail-safe is a built-in shutdown timeout (currently set to 30 seconds). After the given time period all attempts to shutdown gracefully are abandoned and daemon terminates immediatelly.

SIGTERM

This is more aggresive version of daemon shutdown process. In this case the processing is terminated immediatelly, but there is a high chance some events will be left unprocessed.

SIGUSR1

Save current application runlog to JSON file.

SIGUSR2

Save current application state to JSON file. Application state is a complete dump of the whole application.

Signals are catched by the daemon engine, transformed into high priority events and these are then handled ASAP with following built-in event callbacks:

There are following built-in application actions, that can be used to send particular signal to apropriate running daemon:

These signal actions may be executed in a following way:

path/to/zendaemon.py --action signal-usr1
path/to/zendaemon.py --action=signal-usr2

There are following built-in events that may be used by daemon components to perform some additional processing:

Daemon components

The daemon components are actual workers in this paricular daemon design. The ZenDaemon class is in fact only a container for these components, that holds them all together and provides a working environment. The actual real work is being done inside these smaller components. They need to be registered inside the daemon to receive the events and the daemon is then going through the event queue and executing correct event callbacks inside these components.

Daemon components are also a great way for code reusability, because one can have a library of usefull generic components and multiple daemons can be then implemented very quickly by simply reusing them. For example one might implement component for trailing text files and many different daemons might reuse that code and add some additional functionality on top of that.

Module contents

  • pyzenkit.zendaemon.ZenDaemon.QueueEmptyException

  • pyzenkit.zendaemon.ZenDaemon.ZenDaemonComponentException

  • pyzenkit.zendaemon.ZenDaemon.ZenDaemonException

  • pyzenkit.zendaemon.ZenDaemon.ZenDaemonStopException

  • pyzenkit.zendaemon.ZenDaemon.EventQueueManager

  • pyzenkit.zendaemon.ZenDaemon.ZenDaemonComponent

  • pyzenkit.zendaemon.ZenDaemon.ZenDaemon

  • pyzenkit.zendaemon.ZenDaemon.DemoZenDaemonComponent

  • pyzenkit.zendaemon.ZenDaemon.DemoZenDaemon

Programming API

Note

Work in progress.

pyzenkit.zendaemon.DFLT_INTERVAL_RUNLOG = 60

Default time interval in seconds for saving currrent processing runlog.

pyzenkit.zendaemon.DFLT_INTERVAL_STATISTICS = 20

Default time interval in seconds for calculating processing statistics.

class pyzenkit.zendaemon.DemoDaemonComponent(**kwargs)[source]

Bases: ZenDaemonComponent

Minimalistic class for demonstration purposes.

cbk_event_default(daemon, args=None)[source]

Callback handler for default demonstration event.

cbk_event_log_statistics(daemon, args)[source]

Periodical processing statistics logging.

cbk_event_start(daemon, args)[source]

Custom start event.

cbk_event_stop(daemon, args)[source]

Custom stop event.

get_events()[source]

Get list of internal event callbacks.

class pyzenkit.zendaemon.DemoZenDaemon(name=None, description=None)[source]

Bases: ZenDaemon

Minimalistic class for demonstration purposes.

DEMO_INTERVAL_RUNLOG = 10
DEMO_INTERVAL_STATS = 5
class pyzenkit.zendaemon.EventQueueManager[source]

Bases: object

Implementation of event queue manager. This implementation supports scheduling of both sequential events and timed events (events scheduled for specific time). The actual event object, that is added into the queue may be arbitrary object, there are no restrictions for its type or interface, because the queue manager does not interacts with the event itself. Internally two separate event queues are used, one for sequentialy scheduled events and another for timed events. For best performance the sequential queue is implemented using collections.dequeue object and the timed queue is implemented using heapq module.

count()[source]

Count the total number of scheduled events.

Returns

Number of events.

Return type

int

next()[source]

Fetch next event from queue.

Raises

QueueEmptyException – If the queue is empty.

Returns

Return next scheduled event from queue along with its optional arguments.

Return type

tuple

schedule(event, args=None)[source]

Schedule new event to the end of the event queue.

Parameters
  • event – Event to be scheduled.

  • args – Optional event arguments to be stored alongside the event.

schedule_after(delay, event, args=None)[source]

Schedule new event after a given time delay.

Parameters
  • delay (float) – Time delay after which to schedule the event.

  • event – Event to be scheduled.

  • args – Optional event arguments to be stored alongside the event.

schedule_at(tstamp, event, args=None)[source]

Schedule new event for a specific time.

Parameters
  • tstamp (float) – Timestamp to which to schedule the event (compatible with time.time()).

  • event – Event to be scheduled.

  • args – Optional event arguments to be stored alongside the event.

schedule_next(event, args=None)[source]

Schedule new event to the beginning of the event queue.

Parameters
  • event – Event to be scheduled.

  • args – Optional event arguments to be stored alongside the event.

wait()[source]

Calculate the waiting period until the next event in queue is due.

Returns

Time interval for which to wait until the next event is due.

Return type

float

when()[source]

Determine the timestamp of the next scheduled event.

Returns

Unix timestamp of next scheduled event.

Return type

float

pyzenkit.zendaemon.MAX_STOP_ATTEMPTS = 3

Maximal number of daemon stop attempts using SIGINT. After this number the daemon will exit the event processing loop immediatelly.

pyzenkit.zendaemon.MAX_STOP_TIMEOUT = 30

Maximal number of seconds to wait for gracefull shutdown.

exception pyzenkit.zendaemon.QueueEmptyException(description, **params)[source]

Bases: Exception

Exception representing empty event queue. This exception will be thrown by zendaemon.EventQueueManager in the event of empty event queue.

pyzenkit.zendaemon.SIGNALS_TO_NAMES_DICT = {Signals.SIGHUP: 'SIGHUP', Signals.SIGINT: 'SIGINT', Signals.SIGQUIT: 'SIGQUIT', Signals.SIGILL: 'SIGILL', Signals.SIGTRAP: 'SIGTRAP', Signals.SIGABRT: 'SIGIOT', Signals.SIGBUS: 'SIGBUS', Signals.SIGFPE: 'SIGFPE', Signals.SIGKILL: 'SIGKILL', Signals.SIGUSR1: 'SIGUSR1', Signals.SIGSEGV: 'SIGSEGV', Signals.SIGUSR2: 'SIGUSR2', Signals.SIGPIPE: 'SIGPIPE', Signals.SIGALRM: 'SIGALRM', Signals.SIGTERM: 'SIGTERM', Signals.SIGCHLD: 'SIGCLD', Signals.SIGCONT: 'SIGCONT', Signals.SIGSTOP: 'SIGSTOP', Signals.SIGTSTP: 'SIGTSTP', Signals.SIGTTIN: 'SIGTTIN', Signals.SIGTTOU: 'SIGTTOU', Signals.SIGURG: 'SIGURG', Signals.SIGXCPU: 'SIGXCPU', Signals.SIGXFSZ: 'SIGXFSZ', Signals.SIGVTALRM: 'SIGVTALRM', Signals.SIGPROF: 'SIGPROF', Signals.SIGWINCH: 'SIGWINCH', Signals.SIGIO: 'SIGPOLL', Signals.SIGPWR: 'SIGPWR', Signals.SIGSYS: 'SIGSYS', Signals.SIGRTMIN: 'SIGRTMIN', Signals.SIGRTMAX: 'SIGRTMAX'}

Translation table to translate signal numbers to their names.

class pyzenkit.zendaemon.ZenDaemon(**kwargs)[source]

Bases: BaseApp

Base implementation of generic daemon.

CONFIG_CHROOT_DIR = 'chroot_dir'
CONFIG_COMPONENTS = 'components'
CONFIG_NODAEMON = 'no_daemon'
CONFIG_PARALEL = 'paralel'
CONFIG_RUNLOG_INTERVAL = 'runlog_interval'
CONFIG_SCHEDULE = 'schedule'
CONFIG_SCHEDULE_AFTER = 'schedule_after'
CONFIG_SCHEDULE_AT = 'schedule_at'
CONFIG_SCHEDULE_NEXT = 'schedule_next'
CONFIG_STATE_FILE = 'state_file'
CONFIG_STATS_INTERVAL = 'stats_interval'
CONFIG_UMASK = 'umask'
CONFIG_WORK_DIR = 'work_dir'
CORE_STATE = 'state'
CORE_STATE_SAVE = 'save'
EVENT_LOG_STATISTICS = 'log_statistics'
EVENT_SAVE_RUNLOG = 'save_runlog'
EVENT_SIGNAL_HUP = 'signal_hup'
EVENT_SIGNAL_USR1 = 'signal_usr1'
EVENT_SIGNAL_USR2 = 'signal_usr2'
EVENT_START = 'start'
EVENT_STOP = 'stop'
EVENT_TERMINATE = 'terminate'
FLAG_CONTINUE = 1
FLAG_STOP = 0
RLKEY_STATISTICS = 'statistics'
cbk_action_signal_alrm()[source]

Send signal ‘SIGALRM’ to currently running daemon.

cbk_action_signal_check()[source]

Send signal ‘0’ to currently running daemon.

cbk_action_signal_hup()[source]

Send signal ‘SIGHUP’ to currently running daemon.

cbk_action_signal_int()[source]

Send signal ‘SIGINT’ to currently running daemon.

cbk_action_signal_term()[source]

Send signal ‘SIGTERM’ to currently running daemon.

cbk_action_signal_usr1()[source]

Send signal ‘SIGUSR1’ to currently running daemon.

cbk_action_signal_usr2()[source]

Send signal ‘SIGUSR2’ to currently running daemon.

cbk_event_log_statistics(daemon, args)[source]

Periodical processing statistics logging.

cbk_event_save_runlog(daemon, args)[source]

Periodical runlog dumps to enable performance monitoring.

cbk_event_signal_hup(daemon, args=None)[source]

Event callback for handling signal - SIGHUP

Todo

In the future this signal should be responsible for soft restart of daemon process. Currently work in progress.

cbk_event_signal_usr1(daemon, args=None)[source]

Event callback for handling signal - SIGUSR1

This signal forces the daemon process to save the current runlog to JSON file.

cbk_event_signal_usr2(daemon, args=None)[source]

Event callback for handling signal - SIGUSR2

This signal forces the daemon process to save the current state to JSON file. State is more verbose than runlog and it contains almost all internal data.

cbk_event_start(daemon, args)[source]

Start daemon processing.

cbk_event_stop(daemon, args)[source]

Gracefully stop daemon processing. This event handler also schedules the terminate event after the pyzenkit.zendaemon.MAX_STOP_TIMEOUT seconds as a failsafe to force daemon process to quit.

cbk_event_terminate(daemon, args)[source]

Gracefully stop daemon processing.

static gen_event_handler(event_name, callback, prepend=False)
get_events()[source]

Get list of internal event callbacks.

is_done()[source]

Check if the daemon .

send_signal(sign)[source]

Send given signal to all currently running daemon(s).

set_done()[source]

Set the DONE flag to True.

wait(period)[source]

Wait/pause for given amount of seconds.

class pyzenkit.zendaemon.ZenDaemonComponent(**kwargs)[source]

Bases: object

Base implementation for all daemon components. Daemon components are building blocks of each daemon and they are responsible for the actual work to be done. This approach enables very easy reusability.

static gen_event_handler(event_name, callback, prepend=False)
get_events()[source]

Get the list of event names and their appropriate callback handlers.

get_state()[source]

Get the current internal state of component (for debugging).

get_statistics()[source]

Calculate processing statistics

inc_statistic(key, increment=1)[source]

Raise given statistic key with given increment.

setup(daemon)[source]

Perform component setup.

setup_dump(daemon)[source]

Dump component setup.

exception pyzenkit.zendaemon.ZenDaemonComponentException(description, **params)[source]

Bases: ZenAppProcessException

Describes problems specific to daemon components.

exception pyzenkit.zendaemon.ZenDaemonException(description, **params)[source]

Bases: ZenAppProcessException

Describes problems specific to daemons.

exception pyzenkit.zendaemon.ZenDaemonStopException[source]

Bases: BaseException

Exception that is raised when daemon should stop gracefully but immediatelly.

pyzenkit.zendaemon.calc_statistics(stats_cur, stats_prev, tdiff)[source]

Calculate statistics.