The ruleitem module

This module contains the implementation of the replacable rule item objects, and some functions to query and manipulate rules that are used by the Lexicon.

class Item

Bases: object

Base class for any replacable rule item.

An Item is considered to be immutable; you should never alter the attributes after instantiation. When an Item can be partly pre-evaluated a copy must be returned by calling type(self)(*args).

In some cases you can also return a different Item type when pre-evaluating partly succeeds. For example, the select type simply returns the chosen item if the index already can be evaluated, without evaluating the other items.

evaluate(ns)

Evaluate item in namespace dict ns.

pre_evaluate(ns)

Try to evaluate item in namespace dict ns.

Return a two-tuple(obj, success).

Success is a two-bit value indicating whether the result is completely evaluated and whether something has changed. Bit 0 is set when the item is completely evaluated, and bit 1 is set when there was no modification. So all possible return values are:

0: the object has changed but it is not yet completely evaluated

1: the object has changed and now it is fully evaluated

2: the object needs evaluation but is not changed

Specific items may reimplement this method to return partially evaluated items.

variations()

Yield the possible results for this item.

This is used to build a decision tree for a rule, to see which actions and targets it could bring.

The default implementation raises a RuntimeError; only RuleItem objects can yield variations.

class VariableItem(name, predicate=None)

Bases: parce.ruleitem.Item

A named variable that’s accessed in the namespace.

If a predicate is given, it is called when this item is called, with this item as first argument, and then the other arguments that were specified.

evaluate(ns)

Get the variable from the namespace dict.

class call(predicate, *arguments)

Bases: parce.ruleitem.Item

Call predicate with arguments.

evaluate(ns)

Call predicate with the arguments.

pre_evaluate(ns)

Optimize by pre-evaluating what can be pre-evaluated.

class RuleItem

Bases: parce.ruleitem.Item

Classes inheriting RuleItem are allowed in toplevel in rules.

They are evaluated by the lexicon when a rule matches.

class select(index, *items)

Bases: parce.ruleitem.RuleItem

Chooses one of the items.

If an item is a list, it is unrolled when replacing the item in a rule.

evaluate(ns)

Return items[index].

pre_evaluate(ns)

Optimize by pre-evaluating what can be pre-evaluated.

variations()

Yield all the items that could be chosen (unevaluated).

class target(value, *lexicons)

Bases: parce.ruleitem.RuleItem

Has a special handling: if the value is an integer, it is used as the result value (to push/pop contexts).

If it is a two-tuple(index, argument): The index points to the lexicon and the argument is used as lexicon argument.

evaluate(ns)

Return value if integer, otherwise lexicons[value[0]](value[1]).

pre_evaluate(ns)

Optimize by pre-evaluating what can be pre-evaluated.

variations()

Yield our possible variations.

If the value is evaluated, yield either the value or the chosen lexicon. If not, yields a_number and all lexicons items.

class PostponedItem

Bases: parce.ruleitem.RuleItem

Mixin base class for items that keep alive after the Lexicon.

When inheriting from this class, implement the evaluate_items() method, which lists all values as they were given to the __init__ method.

If this method returns values, those are evaluated, and a new PostponedItem is returned with the contents evaluated.

evaluate(ns)

Evaluate all values returned by the evaluate_items() method.

If any value changes, a copy of the Item is returned, otherwise the Item ifself. If the evaluate_items() method does not yield any value, this Item is always returned unchanged.

pre_evaluate(ns)

Pre-evaluate all values returned by the evaluate_items() method.

If any value changes, a copy of the Item is returned, otherwise the Item ifself.

evaluate_items()

Return a tuple of the values as given to the __init__ method, when they need to be evaluated inside this PostponedItem.

This method should either yield all values that were given to the __init__ method, or nothing. The default implementation yields nothing, so nothing is evaluated or pre-evaluated.

class pattern(value)

Bases: parce.ruleitem.PostponedItem

Represents a pattern.

This evaluates its value, but remains alive after building the rule.

property value

Get the pattern value.

evaluate_items()

Yield the pattern value.

variations()

If the value is evaluated, yield it, otherwise yields None and a_string.

class ActionItem

Bases: parce.ruleitem.PostponedItem

Mixin base class for dynamic actions.

class SubgroupAction(*actions)

Bases: parce.ruleitem.ActionItem

Yield actions from subgroups in a match.

A SubgroupAction looks at subgroups in the regular expression match and returns the same amount of tokens as there are subgroups, using the specified action for every subgroup.

For example, the rule:

"(0x)([0-9a-f]+)", SubgroupAction(Number.Prefix, Number.Hexadecimal)

yields two tokens in case of a match, one for “0x” and the other for the other group of the match.

There should be the same number of subgroups in the regular expression as there are action attributes given to __init__().

replace(lexer, pos, text, match)

variations()

Yield the possible actions.

class DelegateAction(lexicon)

Bases: parce.ruleitem.ActionItem

This action uses a lexicon to parse the text.

All tokens are yielded as one group, flattened, ignoring the tree structure, so this is not efficient for large portions of text, as the whole region is parsed again on every modification.

But it can be useful when you want to match a not too large text blob first that’s difficult to capture otherwise, and then lex it with a lexicon that does (almost) not enter other lexicons.

replace(lexer, pos, text, match)

Use our lexicon to parse the matched text.

evaluate_items()

Return the lexicon specified on init, used by evaluate() and pre_evaluate().

variations()

Yield our lexicon.

class SkipAction

Bases: parce.ruleitem.ActionItem

A DynamicAction that yields nothing.

A SkipAction() is stored in the module variable skip and causes the rule to silently ignore the matched text.

replace(lexer, pos, text, match)

variations()

Yield no variations.

evaluate(obj, ns)

Evaluate an object, that may or may not be an Item.

The namespace ns is a dictionary containing text, match and/or arg variables. A list or a tuple of items is also evaluated and always becomes a tuple.

evaluate_rule(rule, match)

Evaluate all RuleItem objects in the rule.

The specified match object provides the value for the TEXT and MATCH variables. Lists and tuples are unrolled.

pre_evaluate(obj, ns)

Pre-evaluate any object, that may or may not be an Item.

Returns a two-tuple(result, success). The namespace ns is a dictionary containing text, match and/or arg variables.

• If the object is an Item, returns object.pre_evaluate(ns).

• If the object is a list or tuple, evaluates the contents and returns a tuple.

• If the object is none of the above, simply returns the object unchanged.

The success value can be one of the values described in Item.pre_evaluate(), or it is 3, meaning that the object is returned unchanged and needs no evaluation.

pre_evaluate_rule(rule, arg)

Evaluate all RuleItem objects that can be evaluated in the rule.

The specified arg provides the value for the ARG variable. Rule items that depend on the match object are not yet evaluated. Lists and tuples are unrolled. Returns the rule as a tuple.

needs_evaluation(rule)

Return True if there are items in the rule that need evaluating.

variations_tree(rule)

Return a tuple with the tree structure of all possible variations.

Branches (choices) are indicated by a frozenset, which contains zero or more tuples.

variations(rule)

Yield all possible variations of the rule.

unroll(obj)

Unroll a tuple or list.

If the object is a tuple or list, yields the unrolled members recursively. Otherwise just the object itself is yielded.

a_number = a_number

sentinel denoting that a variation is any integer

a_string = a_string

sentinel denoting that a variation is any string