pyzenkit.zendaemon module¶
This module provides base implementation of daemon service represented by the
pyzenkit.zendaemon.ZenDaemon
class. It builds on top of pyzenkit.baseapp
module and adds couple of other usefull features:
- Fully automated daemonization process.
- Event driven design.
- Support for handling arbitrary signals.
- Support for modularity with daemon components.
Daemonization¶
Daemonization is a process of transforming foreground application into a background
always running service. The pyzenkit.zendaemon.ZenDaemon
class has
this feature built in and configurable with command line options, or configuration
files/directories. Please see documentation page Configuration.
Daemonization is implemented on top of the pyzenkit.daemonizer
utility
library, please refer to its documentation for more details.
Event driven design and event queue¶
The daemon application has the event driven design. The ZenDaemon._sub_stage_process()
method is implemented to perform an infinite event loop. There are events being emited
from different parts of the application, which are then being ordered into event queue.
Each of these events is then handled with appropriate callback method.
Event callback methods must be registered into daemon application to be recognized. Multiple event callbacks my be registered for certain single event. In this case those callbacks will be called in order of registration and a result of the previous one will be passed as input of the next. In other words callbacks form a pipeline and event will be pushed through that. Each callback method has the opportunity to break the pipeline/chain by returning apropriate flag.
The naming convention for event callback method is the following:
- Event callback must be method, which accepts reference to
pyzenkit.zendaemon.ZenDaemon
daemon
as first argument anddict
args
as second argument. - Event callback method name must begin with
cbk_event_
prefix. - Event name in method name after the prefix must also be snake_cased`.
Note, that event name in callback method name is not used in any way for mapping callbacks to events (like in the case of actions), the callbacks are explicitly registered to handle particular events. It is however a great best practice and it is very clear then which callback handles which event.
Following are examples of valid event callbacks:
cbk_event_test(self, daemon, args)
cbk_event_another_test(self, daemon, args)
Each daemon application has an instance of the EventQueueManager
as
public attribute, which represents the event queue. There are following methods
available for scheduling events into the queue:
- End of the queue:
pyzenkit.zendaemon.EventQueueManager.schedule()
- Beginning of the queue:
pyzenkit.zendaemon.EventQueueManager.schedule_next()
- After certain time interval:
pyzenkit.zendaemon.EventQueueManager.schedule_after()
- At specific time:
pyzenkit.zendaemon.EventQueueManager.schedule_at()
Signal handling¶
Each daemon service should be capable of receiving and handling external signals. Currently support for following signals is built-in:
- SIGHUP
Reload configuration and reconfigure the application.
Note
This feature is not yet fully implemented.
- SIGINT
- Finish up all current work and attempt to perform gracefull stop. In case you send this signal multiple times (currently the magical number is three), you will force the event loop to be exited prematurelly. This feature can be usefull in case you are shutting the daemon down by hand and the shutdown process takes too long to complete. Another fail-safe is a built-in shutdown timeout (currently set to 30 seconds). After the given time period all attempts to shutdown gracefully are abandoned and daemon terminates immediatelly.
- SIGTERM
- This is more aggresive version of daemon shutdown process. In this case the processing is terminated immediatelly, but there is a high chance some events will be left unprocessed.
- SIGUSR1
- Save current application runlog to JSON file.
- SIGUSR2
- Save current application state to JSON file. Application state is a complete dump of the whole application.
Signals are catched by the daemon engine, transformed into high priority events and these are then handled ASAP with following built-in event callbacks:
pyzenkit.zendaemon.ZenDaemon.cbk_event_signal_hup()
pyzenkit.zendaemon.ZenDaemon.cbk_event_signal_int()
pyzenkit.zendaemon.ZenDaemon.cbk_event_signal_usr1()
pyzenkit.zendaemon.ZenDaemon.cbk_event_signal_usr2()
There are following built-in application actions, that can be used to send particular signal to apropriate running daemon:
pyzenkit.zendaemon.ZenDaemon.cbk_action_signal_alrm()
pyzenkit.zendaemon.ZenDaemon.cbk_action_signal_check()
pyzenkit.zendaemon.ZenDaemon.cbk_action_signal_hup()
pyzenkit.zendaemon.ZenDaemon.cbk_action_signal_int()
pyzenkit.zendaemon.ZenDaemon.cbk_action_signal_term()
pyzenkit.zendaemon.ZenDaemon.cbk_action_signal_usr1()
pyzenkit.zendaemon.ZenDaemon.cbk_action_signal_usr2()
These signal actions may be executed in a following way:
path/to/zendaemon.py --action signal-usr1
path/to/zendaemon.py --action=signal-usr2
There are following built-in events that may be used by daemon components to perform some additional processing:
pyzenkit.zendaemon.ZenDaemon.cbk_event_signal_hup()
pyzenkit.zendaemon.ZenDaemon.cbk_event_signal_usr1()
pyzenkit.zendaemon.ZenDaemon.cbk_event_signal_usr2()
pyzenkit.zendaemon.ZenDaemon.cbk_event_log_statistics()
pyzenkit.zendaemon.ZenDaemon.cbk_event_save_runlog()
pyzenkit.zendaemon.ZenDaemon.cbk_event_stop()
pyzenkit.zendaemon.ZenDaemon.cbk_event_terminate()
Daemon components¶
The daemon components are actual workers in this paricular daemon design. The ZenDaemon
class is in fact only a container for these components, that holds them all
together and provides a working environment. The actual real work is being done inside
these smaller components. They need to be registered inside the daemon to receive
the events and the daemon is then going through the event queue and executing correct
event callbacks inside these components.
Daemon components are also a great way for code reusability, because one can have a library of usefull generic components and multiple daemons can be then implemented very quickly by simply reusing them. For example one might implement component for trailing text files and many different daemons might reuse that code and add some additional functionality on top of that.
Module contents¶
pyzenkit.zendaemon.ZenDaemon.QueueEmptyException
pyzenkit.zendaemon.ZenDaemon.ZenDaemonComponentException
pyzenkit.zendaemon.ZenDaemon.ZenDaemonException
pyzenkit.zendaemon.ZenDaemon.ZenDaemonStopException
pyzenkit.zendaemon.ZenDaemon.EventQueueManager
pyzenkit.zendaemon.ZenDaemon.ZenDaemonComponent
pyzenkit.zendaemon.ZenDaemon.ZenDaemon
pyzenkit.zendaemon.ZenDaemon.DemoZenDaemonComponent
pyzenkit.zendaemon.ZenDaemon.DemoZenDaemon
Programming API¶
Note
Work in progress.
public attributes:
pyzenkit.zendaemon.ZenDaemon.queue
- Event queue.
public methods:
pyzenkit.zendaemon.ZenDaemon.set_done()
- Attempt to gracefull application shutdown.pyzenkit.zendaemon.ZenDaemon.is_done()
- Check whether the application is in gracefull shutdown process.pyzenkit.zendaemon.ZenDaemon.wait()
- Pause the processing for given amount of time.
-
class
pyzenkit.zendaemon.
DemoDaemonComponent
(**kwargs)[source]¶ Bases:
pyzenkit.zendaemon.ZenDaemonComponent
Minimalistic class for demonstration purposes.
-
class
pyzenkit.zendaemon.
DemoZenDaemon
(name=None, description=None)[source]¶ Bases:
pyzenkit.zendaemon.ZenDaemon
Minimalistic class for demonstration purposes.
-
DEMO_INTERVAL_RUNLOG
= 10¶
-
DEMO_INTERVAL_STATS
= 5¶
-
-
class
pyzenkit.zendaemon.
EventQueueManager
[source]¶ Bases:
object
Implementation of event queue manager. This implementation supports scheduling of both sequential events and timed events (events scheduled for specific time). The actual event object, that is added into the queue may be arbitrary object, there are no restrictions for its type or interface, because the queue manager does not interacts with the event itself. Internally two separate event queues are used, one for sequentialy scheduled events and another for timed events. For best performance the sequential queue is implemented using
collections.dequeue
object and the timed queue is implemented usingheapq
module.-
count
()[source]¶ Count the total number of scheduled events.
Returns: Number of events. Return type: int
-
next
()[source]¶ Fetch next event from queue.
Raises: QueueEmptyException – If the queue is empty. Returns: Return next scheduled event from queue along with its optional arguments. Return type: tuple
-
schedule
(event, args=None)[source]¶ Schedule new event to the end of the event queue.
Parameters: - event – Event to be scheduled.
- args – Optional event arguments to be stored alongside the event.
-
schedule_after
(delay, event, args=None)[source]¶ Schedule new event after a given time delay.
Parameters: - delay (float) – Time delay after which to schedule the event.
- event – Event to be scheduled.
- args – Optional event arguments to be stored alongside the event.
-
schedule_at
(tstamp, event, args=None)[source]¶ Schedule new event for a specific time.
Parameters: - tstamp (float) – Timestamp to which to schedule the event (compatible with
time.time()
). - event – Event to be scheduled.
- args – Optional event arguments to be stored alongside the event.
- tstamp (float) – Timestamp to which to schedule the event (compatible with
-
schedule_next
(event, args=None)[source]¶ Schedule new event to the beginning of the event queue.
Parameters: - event – Event to be scheduled.
- args – Optional event arguments to be stored alongside the event.
-
-
pyzenkit.zendaemon.
MAX_STOP_ATTEMPTS
= 3¶ Maximal number of daemon stop attempts using
SIGINT
. After this number the daemon will exit the event processing loop immediatelly.
-
pyzenkit.zendaemon.
MAX_STOP_TIMEOUT
= 30¶ Maximal number of seconds to wait for gracefull shutdown.
-
exception
pyzenkit.zendaemon.
QueueEmptyException
(description, **params)[source]¶ Bases:
Exception
Exception representing empty event queue. This exception will be thrown by
zendaemon.EventQueueManager
in the event of empty event queue.
-
pyzenkit.zendaemon.
SIGNALS_TO_NAMES_DICT
= {<Signals.SIGHUP: 1>: 'SIGHUP', <Signals.SIGINT: 2>: 'SIGINT', <Signals.SIGQUIT: 3>: 'SIGQUIT', <Signals.SIGILL: 4>: 'SIGILL', <Signals.SIGTRAP: 5>: 'SIGTRAP', <Signals.SIGABRT: 6>: 'SIGIOT', <Signals.SIGBUS: 7>: 'SIGBUS', <Signals.SIGFPE: 8>: 'SIGFPE', <Signals.SIGKILL: 9>: 'SIGKILL', <Signals.SIGUSR1: 10>: 'SIGUSR1', <Signals.SIGSEGV: 11>: 'SIGSEGV', <Signals.SIGUSR2: 12>: 'SIGUSR2', <Signals.SIGPIPE: 13>: 'SIGPIPE', <Signals.SIGALRM: 14>: 'SIGALRM', <Signals.SIGTERM: 15>: 'SIGTERM', <Signals.SIGCHLD: 17>: 'SIGCLD', <Signals.SIGCONT: 18>: 'SIGCONT', <Signals.SIGSTOP: 19>: 'SIGSTOP', <Signals.SIGTSTP: 20>: 'SIGTSTP', <Signals.SIGTTIN: 21>: 'SIGTTIN', <Signals.SIGTTOU: 22>: 'SIGTTOU', <Signals.SIGURG: 23>: 'SIGURG', <Signals.SIGXCPU: 24>: 'SIGXCPU', <Signals.SIGXFSZ: 25>: 'SIGXFSZ', <Signals.SIGVTALRM: 26>: 'SIGVTALRM', <Signals.SIGPROF: 27>: 'SIGPROF', <Signals.SIGWINCH: 28>: 'SIGWINCH', <Signals.SIGIO: 29>: 'SIGPOLL', <Signals.SIGPWR: 30>: 'SIGPWR', <Signals.SIGSYS: 31>: 'SIGSYS', <Signals.SIGRTMIN: 34>: 'SIGRTMIN', <Signals.SIGRTMAX: 64>: 'SIGRTMAX'}¶ Translation table to translate signal numbers to their names.
-
class
pyzenkit.zendaemon.
ZenDaemon
(**kwargs)[source]¶ Bases:
pyzenkit.baseapp.BaseApp
Base implementation of generic daemon.
-
CONFIG_CHROOT_DIR
= 'chroot_dir'¶
-
CONFIG_COMPONENTS
= 'components'¶
-
CONFIG_NODAEMON
= 'no_daemon'¶
-
CONFIG_PARALEL
= 'paralel'¶
-
CONFIG_PID_FILE
= 'pid_file'¶
-
CONFIG_RUNLOG_INTERVAL
= 'runlog_interval'¶
-
CONFIG_STATE_FILE
= 'state_file'¶
-
CONFIG_STATS_INTERVAL
= 'stats_interval'¶
-
CONFIG_UMASK
= 'umask'¶
-
CONFIG_WORK_DIR
= 'work_dir'¶
-
CORE_STATE
= 'state'¶
-
CORE_STATE_SAVE
= 'save'¶
-
EVENT_LOG_STATISTICS
= 'log_statistics'¶
-
EVENT_SAVE_RUNLOG
= 'save_runlog'¶
-
EVENT_SIGNAL_HUP
= 'signal_hup'¶
-
EVENT_SIGNAL_USR1
= 'signal_usr1'¶
-
EVENT_SIGNAL_USR2
= 'signal_usr2'¶
-
EVENT_STOP
= 'stop'¶
-
EVENT_TERMINATE
= 'terminate'¶
-
FLAG_CONTINUE
= 1¶
-
FLAG_STOP
= 0¶
-
RLKEY_STATISTICS
= 'statistics'¶
-
cbk_event_save_runlog
(daemon, args)[source]¶ Periodical runlog dumps to enable performance monitoring.
-
cbk_event_signal_hup
(daemon, args=None)[source]¶ Event callback for handling signal - SIGHUP
Todo
In the future this signal should be responsible for soft restart of daemon process. Currently work in progress.
-
cbk_event_signal_int
(daemon, args=None)[source]¶ Event callback for handling signal - SIGINT
This signal forces the daemon process to finish all its current work and stop gracefully.
-
cbk_event_signal_usr1
(daemon, args=None)[source]¶ Event callback for handling signal - SIGUSR1
This signal forces the daemon process to save the current runlog to JSON file.
-
cbk_event_signal_usr2
(daemon, args=None)[source]¶ Event callback for handling signal - SIGUSR2
This signal forces the daemon process to save the current state to JSON file. State is more verbose than runlog and it contains almost all internal data.
-
cbk_event_stop
(daemon, args)[source]¶ Gracefully stop daemon processing. This event handler also schedules the
terminate
event after thepyzenkit.zendaemon.MAX_STOP_TIMEOUT
seconds as a failsafe to force daemon process to quit.
-
-
class
pyzenkit.zendaemon.
ZenDaemonComponent
(**kwargs)[source]¶ Bases:
object
Base implementation for all daemon components. Daemon components are building blocks of each daemon and they are responsible for the actual work to be done. This approach enables very easy reusability.
-
exception
pyzenkit.zendaemon.
ZenDaemonComponentException
(description, **params)[source]¶ Bases:
pyzenkit.baseapp.ZenAppProcessException
Describes problems specific to daemon components.
-
exception
pyzenkit.zendaemon.
ZenDaemonException
(description, **params)[source]¶ Bases:
pyzenkit.baseapp.ZenAppProcessException
Describes problems specific to daemons.
-
exception
pyzenkit.zendaemon.
ZenDaemonStopException
[source]¶ Bases:
BaseException
Exception that is raised when daemon should stop gracefully but immediatelly.