pynspect.filters module¶
This module provides tools for data filtering based on filtering and query grammar.
The filtering grammar is thoroughly described in following modules:
-
Lexical analyzer, descriptions of valid grammar tokens.
-
Grammar parser, language grammar description
-
Object representation of grammar rules, interface definition
-
The addressing language JPath.
Please refer to appropriate module for more in-depth information.
There are following main tools in this package:
-
Tool capable of filtering data structures according to given filtering rules.
Available filtering functions¶
size
Return the size/length of given list. This enables writing rules like events with more than five source addressess:
size(Source.IP4) > 5
time
Return current Unix timestamp as
float
.utcnow
Return current date and time in UTC timezone. This enables writing rules like events with detection time older than two hours:
DetectTime < (utcnow() - 02:00:00)
Example filters¶
Following is a non exhaustive list of example filtering rules:
DetectTime < (utcnow() - 02:00:00)
exists EventTime and exists DetectTime and EventTime > DetectTime
Category in ['Anomaly.Connection'] and Source.Type in ['Booter']
Category in ['Attempt.Exploit'] and (Target.Port in [3306] or Source.Proto in ['mysql'] or Target.Proto in ['mysql'])
Warning
Be carefull with the grammar function names. Currently, there is a flaw in the expression grammar that forbids using function names that begin with the same characters as grammar keywords like ‘and’, ‘le’, ‘like’, etc. For example the name ‘len’ is not a valid function name, because there is a collision with ‘le’ comparison operator.
Todo
There is quite a lot of code that needs to be written before actual filtering can take place. In the future, there should be some kind of object, that will be tailored for immediate processing and will take care of initializing uderlying parser, compiler and filter. This object will be designed later.
-
class
pynspect.filters.
DataObjectFilter
(parser=None, compiler=None)[source]¶ Bases:
pynspect.traversers.BaseFilteringTreeTraverser
Rule tree traverser implementing default object filtering logic.
Following example demonstrates DataObjectFilter usage in conjuction with PynspectFilterParser:
>>> flt = DataObjectFilter() >>> psr = PynspectFilterParser() >>> psr.build() >>> rule = psr.parse('ID like "e214d2d9"') >>> result = flt.filter(rule, test_msg)
You may use the built-in shortcuts for parsing and compiling rules:
>>> flt = DataObjectFilter( ... parser = PynspectFilterParser, ... compiler = IDEAFilterCompiler ... ) >>> rule = flt.prepare('(Source.IP4 == 188.14.166.39)') >>> result = flt.filter(rule, test_msg)
Rule tree can be created by hand/programatically:
>>> rule = ComparisonBinOpRule('OP_GT', VariableRule("ConnCount"), IntegerRule(1)) >>> result = flt.filter(rule, test_msg)
-
binary_operation_comparison
(rule, left, right, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.binary_operation_comparison()
interface.
-
binary_operation_logical
(rule, left, right, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.binary_operation_logical()
interface.
-
binary_operation_math
(rule, left, right, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.binary_operation_math()
interface.
-
constant
(rule, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.constant()
interface.
-
datetime
(rule, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.datetime()
interface.
-
filter
(rule, data)[source]¶ Apply given filtering rule to given data structure.
- Parameters
rule (pynspect.rules.Rule) – filtering rule to be checked
data (any) – data structure to check against rule, ussually dict
- Returns
True or False or expression result
- Return type
bool or any
-
float
(rule, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.float()
interface.
-
integer
(rule, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.integer()
interface.
-
ipv4
(rule, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.ipv4()
interface.
-
ipv6
(rule, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.ipv6()
interface.
-
list
(rule, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.list()
interface.
-
prepare
(rule)[source]¶ Parse and/or compile given rule into rule tree.
- Parameters
rule – Filtering grammar rule.
- Returns
Parsed and/or compiled rule.
-
timedelta
(rule, **kwargs)[source]¶ Implementation of
pynspect.traversers.RuleTreeTraverser.timedelta()
interface.
-
-
pynspect.filters.
grfcbk_size
(args)[source]¶ Grammar rule function callback: size. This function will count the size of first item in argument list.
- Parameters
args (list) – List of function arguments.
- Returns
Size of the first item in argument list.
- Return type
int
-
pynspect.filters.
grfcbk_strlen
(args)[source]¶ Grammar rule function callback: strlen. This function will measure the string length of all subitems of the first item in argument list.
- Parameters
args (list) – List of function arguments.
- Returns
Length of all subitems of the first item in argument list.
- Return type
int or list
-
pynspect.filters.
grfcbk_time
(args)[source]¶ Grammar rule function callback: time. This function will call the
time.time()
function and return the result.- Parameters
args (list) – List of function arguments. Should be empty, but
- Returns
The time in seconds since the epoch as a floating point number.
- Return type
float
-
pynspect.filters.
grfcbk_utcnow
(args)[source]¶ Grammar rule function callback: utcnow. This function will call the
datetime.datetime.utcnow()
function and return the result.- Parameters
args (list) – List of function arguments. Should be empty, but
- Returns
Current datetime in UTC timezone.
- Return type
datetime.datetime