pynspect.lexer module

This module contains object encapsulation of PLY lexical analyzer for universal filtering and query language grammar. It is designed for working with almost arbitrary data structures and can be used in wide range of projects.

Currently recognized grammar tokens

# Mathematical operation tokens.
OP_PLUS   = r'\+'   # Addition
OP_MINUS  = r'-'    # Substraction
OP_TIMES  = r'\*'   # Multiplication
OP_DIVIDE = r'/'    # Division
OP_MODULO = r'%'    # Modulo

# Logical operation tokens.
OP_OR     = r'(or|OR)'              # Logical OR
OP_XOR    = r'(xor|XOR)'            # Logical XOR (exclusive OR)
OP_AND    = r'(and|AND)'            # Logical AND
OP_NOT    = r'(not|NOT)'            # Logical NOT
OP_EXISTS = r'(exists|EXISTS|\?)'   # Test for existence

# Priority logical operation tokens.
OP_OR_P  = r'\|\|'  # Logical OR with higher priority than tokens above
OP_XOR_P = r'\^\^'  # Logical XOR with higher priority than tokens above
OP_AND_P = r'&&'    # Logical AND with higher priority than tokens above

# Comparison operation tokens.
OP_LIKE = r'(like|LIKE|=~)'
OP_IN   = r'(in|IN|~~)'
OP_IS   = r'(is|IS)'
OP_EQ   = r'(eq|EQ|==)'
OP_NE   = r'(ne|NE|!=|<>)'
OP_GT   = r'(gt|GT|>)'
OP_GE   = r'(ge|GE|>=)'
OP_LT   = r'(lt|LT|<)'
OP_LE   = r'(le|LE|<=)'

# Special tokens.
COMMA  = r'\s*,\s*|\s*;\s*'
LPAREN = r'\('
RPAREN = r'\)'
LBRACK = r'\['
RBRACK = r'\]'

# Contant and variable tokens.
IPV4      = r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}(?:\/\d{1,2}|(?:-|..)\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})?'
IPV6      = r'[:a-fA-F0-9]+:[:a-fA-F0-9]*(?:\/\d{1,3}|(?:-|..)[:a-fA-F0-9]+:[:a-fA-F0-9]*)?'
DATETIME  = r'[0-9]{4}-[0-9]{2}-[0-9]{2}[Tt][0-9]{2}:[0-9]{2}:[0-9]{2}(?:\.[0-9]+)?(?:[Zz]|(?:[+-][0-9]{2}:[0-9]{2}))'
TIMEDELTA = r'([0-9]+[D|d])?[0-9]{2}:[0-9]{2}:[0-9]{2}'
INTEGER   = r'\d+'
FLOAT     = r'\d+\.\d+'
CONSTANT  = r'"([^"]+)"|\'([^\']+)\''
FUNCTION  = r'[_a-zA-Z][_a-zA-Z0-9]{2,}\('
VARIABLE  = r'[_a-zA-Z][-_a-zA-Z0-9]*(?:\[(?:\d+|-\d+|\#)\])?(?:\.?[a-zA-Z][-_a-zA-Z0-9]*(?:\[(?:\d+|-\d+|\#)\])?)*'

Note

Implementation of this module is very PLY specific, please read the appropriate documentation to understand it. For the same reason the pylint tool comlains a lot about code style in this module, but that is a feature.

Todo

Consider following options:

  • Support for negative integers and floats

  • Support for functions (count, max, min, first, last, time, etc.)