pynspect.filters module

This module provides tools for data filtering based on filtering and query grammar.

The filtering grammar is thoroughly described in following modules:

• pynspect.lexer

  Lexical analyzer, descriptions of valid grammar tokens.

• pynspect.gparser

  Grammar parser, language grammar description

• pynspect.rules

  Object representation of grammar rules, interface definition

• pynspect.jpath

  The addressing language JPath.

Please refer to appropriate module for more in-depth information.

There are following main tools in this package:

• DataObjectFilter

  Tool capable of filtering data structures according to given filtering rules.

Available filtering functions

• size

  Return the size/length of given list. This enables writing rules like events with more than five source addressess:

  size(Source.IP4) > 5
  

• time

  Return current Unix timestamp as float.

• utcnow

  Return current date and time in UTC timezone. This enables writing rules like events with detection time older than two hours:

  DetectTime < (utcnow() - 02:00:00)
  

Example filters

Following is a non exhaustive list of example filtering rules:

DetectTime < (utcnow() - 02:00:00)
exists EventTime and exists DetectTime and EventTime > DetectTime
Category in ['Anomaly.Connection'] and Source.Type in ['Booter']
Category in ['Attempt.Exploit'] and (Target.Port in [3306] or Source.Proto in ['mysql'] or Target.Proto in ['mysql'])

Warning

Be carefull with the grammar function names. Currently, there is a flaw in the expression grammar that forbids using function names that begin with the same characters as grammar keywords like ‘and’, ‘le’, ‘like’, etc. For example the name ‘len’ is not a valid function name, because there is a collision with ‘le’ comparison operator.

Todo

There is quite a lot of code that needs to be written before actual filtering can take place. In the future, there should be some kind of object, that will be tailored for immediate processing and will take care of initializing uderlying parser, compiler and filter. This object will be designed later.

class pynspect.filters.DataObjectFilter(parser=None, compiler=None)

Bases: pynspect.traversers.BaseFilteringTreeTraverser

Rule tree traverser implementing default object filtering logic.

Following example demonstrates DataObjectFilter usage in conjuction with PynspectFilterParser:

>>> flt = DataObjectFilter()
>>> psr = PynspectFilterParser()
>>> psr.build()
>>> rule = psr.parse('ID like "e214d2d9"')
>>> result = flt.filter(rule, test_msg)

You may use the built-in shortcuts for parsing and compiling rules:

>>> flt = DataObjectFilter(
...     parser   = PynspectFilterParser,
...     compiler = IDEAFilterCompiler
... )
>>> rule   = flt.prepare('(Source.IP4 == 188.14.166.39)')
>>> result = flt.filter(rule, test_msg)

Rule tree can be created by hand/programatically:

>>> rule = ComparisonBinOpRule('OP_GT', VariableRule("ConnCount"), IntegerRule(1))
>>> result = flt.filter(rule, test_msg)

binary_operation_comparison(rule, left, right, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.binary_operation_comparison() interface.

binary_operation_logical(rule, left, right, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.binary_operation_logical() interface.

binary_operation_math(rule, left, right, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.binary_operation_math() interface.

constant(rule, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.constant() interface.

datetime(rule, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.datetime() interface.

filter(rule, data)

Apply given filtering rule to given data structure.

Parameters

• rule (pynspect.rules.Rule) – filtering rule to be checked

• data (any) – data structure to check against rule, ussually dict

Returns

True or False or expression result

Return type

bool or any

float(rule, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.float() interface.

integer(rule, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.integer() interface.

ipv4(rule, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.ipv4() interface.

ipv6(rule, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.ipv6() interface.

list(rule, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.list() interface.

prepare(rule)

Parse and/or compile given rule into rule tree.

Parameters

rule – Filtering grammar rule.

Returns

Parsed and/or compiled rule.

timedelta(rule, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.timedelta() interface.

unary_operation(rule, right, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.unary_operation() interface.

variable(rule, **kwargs)

Implementation of pynspect.traversers.RuleTreeTraverser.variable() interface.

pynspect.filters.grfcbk_size(args)

Grammar rule function callback: size. This function will count the size of first item in argument list.

Parameters

args (list) – List of function arguments.

Returns

Size of the first item in argument list.

Return type

int

pynspect.filters.grfcbk_strlen(args)

Grammar rule function callback: strlen. This function will measure the string length of all subitems of the first item in argument list.

Parameters

args (list) – List of function arguments.

Returns

Length of all subitems of the first item in argument list.

Return type

int or list

pynspect.filters.grfcbk_time(args)

Grammar rule function callback: time. This function will call the time.time() function and return the result.

Parameters

args (list) – List of function arguments. Should be empty, but

Returns

The time in seconds since the epoch as a floating point number.

Return type

float

pynspect.filters.grfcbk_utcnow(args)

Grammar rule function callback: utcnow. This function will call the datetime.datetime.utcnow() function and return the result.

Parameters

args (list) – List of function arguments. Should be empty, but

Returns

Current datetime in UTC timezone.

Return type

datetime.datetime