pynspect.jpath module¶

This module provides tools for parsing JPaths and setting or retrieving values on given JPath within data structures.

JPath is simplified version of JSONPath and can be used to addressing nodes within arbitrary data structure composed of dict-like and list-like objects. Basically it can be used for any data structure of objects implementing Python 3 list and/or dict interface.

The motivation for implementing this module were following two use cases:

Enable writing of simple rules in various filtering expressions, for example:
```
Source.IP4 in [192.168.0.0/24, 192.168.0.0/24]
```
Enable simple message modifications based on key => value rules, for example:
```
"Source[1].Type[*]" = "source type tag"
```

The obvious first choice for a solution was the jsonpath-rw library. The full JSONPath however seems to be too big of a gun for our needs and in some cases it could even enable users to cut the branch they are sitting on. For this reason we have designed this simplified version with only required set of basic features.

JPath syntax uses only dot characters . as node delimiters. Each node name may contain only one or more of the following characters:

[a-zA-Z0-9_]+

Node delimiters implicitly work with nested dictionaries and using delimiter results in appending new dictionary as a value of given key in parent disctionary. Working with lists is enabled by using indices. List indices must be enclosed in brackets ‘[‘ and ‘]’ and may contain one of the following values:

[int] - precise index (negative values not permitted, numbering starts with 1)
[#] - last (because you might not know number of nodes)
[*] - all nodes
(index omitted)

When retrieving value(s) at given JPath, use of indices will have following effects:

[int] - Return node at particular list position (starting with 1)
[#] - Return last node
[*] - Return all nodes (same as omitting)
(index omitted) - Return all nodes (same as ‘*’)

When setting value(s) to given JPath, use of indices will have following effects:

[int] - Set value to particular list node (starting with 1)
[#] - Set value to already existing last node, or append new one to an empty list
[*] - Append new value to a list
(index omitted) - This will result in a dictionary key instead of a list

Consider following examples:

>>> msg = {
   'Format': 'IDEA0',
   'ID': 'MESSAGE_ID',
   'DetectTime': 'DETECT TIME',
   'Category': ['CATEGORY'],
   'ConnCount': 633,
   'Description': 'Ping scan',
   'Source': [
      {
         'IP4': ['192.168.1.1', '192.168.1.2'],
         'Proto': ['icmp']
      },
      {
         'IP4': ['192.168.2.1', '192.168.2.2'],
         'Proto': ['icmp']
      }
   ],
   'Target': [
      {
         'Proto': ['icmp'],
         'IP4': ['192.168.3.1', '192.168.3.2'],
         'Anonymised': True
      }
   ],
   'Node': [
      {
         'SW' : ['KIPPO'],
         'Name' : 'NODE_NAME'
      }
   ]
}

>>> jpath_value(msg, 'Format')
'IDEA0'
>>> jpath_value(msg, 'Category')
'CATEGORY'
>>> jpath_value(msg, 'Node.Name')
'NODE_NAME'
>>> jpath_value(msg, 'Source.IP4')
'192.168.1.1'

>>> jpath_values(msg, 'Format')
['IDEA0']
>>> jpath_values(msg, 'Category')
['CATEGORY']
>>> jpath_values(msg, 'Node.Name')
['NODE_NAME']
>>> jpath_values(msg, 'Source.IP4')
['192.168.1.1', '192.168.1.2', '192.168.2.1', '192.168.2.2']

The current implementation has following known drawbacks:

Toplevel element must be a dict-like object
Nested list-like objects are not possible: [[1,2],[3,4]]
It is not possible to set value to multiple elements at once
It is not possible to customize type of created structure, only lists and dicts are always created

exception pynspect.jpath.JPathException(description)[source]¶

Bases: Exception

Custom JPath specific exception.

This exception will be thrown on module specific errors.

pynspect.jpath.RC_VALUE_DUPLICATE = 2¶: Status code for not-unique, returned by function jpath_set().

pynspect.jpath.RC_VALUE_EXISTS = 1¶: Status code for already-exists, returned by function jpath_set().

pynspect.jpath.RC_VALUE_SET = 0¶: Status code for success, returned by function jpath_set().

pynspect.jpath.RE_JPATH_CHUNK = re.compile('^([a-zA-Z0-9_]+)(\\[(#|\\*|\\d+)\\])?$')¶: Regular expression for single JPath chunk.

pynspect.jpath.cache_clear()[source]¶: Clear internal JPath cache.

pynspect.jpath.cache_size()[source]¶

Return the size of internal JPath cache.

Returns: Cache size
Return type: int

pynspect.jpath.jpath_exists(structure, jpath)[source]¶

Check if node at given JPath within given data structure does exist.

Parameters

structure (str) – data structure to be searched
jpath (str) – JPath to be evaluated

Returns

True or False

Return type

bool

pynspect.jpath.jpath_parse(jpath)[source]¶

Parse given JPath into chunks.

Returns list of dictionaries describing all of the JPath chunks.

Parameters: jpath (str) – JPath to be parsed into chunks
Returns: JPath chunks as list of dicts
Return type: list
Raises: JPathException – in case of invalid JPath syntax

pynspect.jpath.jpath_parse_c(jpath)[source]¶

Caching variant of jpath_parse() function. Same arguments and return value.

For performance reasons thee is no copying and all returned values are references to internal cache. Treat the returned values as read only, or suffer the consequences.

pynspect.jpath.jpath_set(structure, jpath, value, overwrite=True, unique=False)[source]¶

Set given JPath to given value within given structure.

For performance reasons this method is intentionally not written as recursive.

Parameters

structure (str) – data structure to be searched
jpath (str) – JPath to be evaluated
value (any) – value of any type to be set at given path
overwrite (bool) – enable/disable overwriting of already existing value
unique (bool) – ensure uniqueness of value, works only for lists

Returns

numerical return code, one of the (RC_VALUE_SET, RC_VALUE_EXISTS, RC_VALUE_DUPLICATE)

Return type

int

pynspect.jpath.jpath_unset(structure, jpath)[source]¶

Unset (delete) value at given JPath within given structure.

For performance reasons this method is intentionally not written as recursive.

Parameters

structure (str) – data structure to be trimmed
jpath (str) – JPath to be evaluated

pynspect.jpath.jpath_value(structure, jpath)[source]¶

Return single value or first value from list at given JPath within given data structure.

This method returns None for non-existent JPaths.

Parameters

structure (str) – data structure to be searched
jpath (str) – JPath to be evaluated

Returns

None or found value

pynspect.jpath.jpath_values(structure, jpath)[source]¶

Return all values at given JPath within given data structure.

For performance reasons this method is intentionally not written as recursive.

Parameters

structure (str) – data structure to be searched
jpath (str) – JPath to be evaluated

Returns

found values as a list

Return type

list