The treebuilderutil module

Helper functions and classes for the treebuilder module.

class BuildResult(tree, start, end, offset, lexicons)

Bases: tuple

encapsulates the return values of TreeBuilder.build_new_tree()

end

Alias for field number 2

lexicons

Alias for field number 4

offset

Alias for field number 3

start

Alias for field number 1

tree

Alias for field number 0

class ReplaceResult(start, end, lexicons)

Bases: tuple

encapsulates the return values of TreeBuilder.replace_tree()

end

Alias for field number 1

lexicons

Alias for field number 2

start

Alias for field number 0

class Changes

Bases: object

Store changes that have to be made to a tree.

This object is used by get_changes(). Calling add() merges new changes with the existing changes.

text

root_lexicon

start

removed

added

add(text, root_lexicon=False, start=0, removed=None, added=None)

Merge new change with existing changes.

If added and removed are not given, all text after start is considered to be replaced.

has_changes()

Return True when there are actually changes.

new_position(pos)

Return how the current changes would affect an older start.

get_prepared_lexer(tree, text, start, new_tree=False)

Get a prepared lexer reading from text, positioned at (or before) start.

Returns the three-tuple (lexer, events, tokens). The events stream is returned seperately because the last Event can be pushed back, so it is yielded again. The tokens are the last tokens group that remained the same.

Returns None when no position to start can be found, just start from the beginning in this case.

If new_tree is True, does not find a start position if there would no tokens remain left of it. This is useful when restarting a tree build; it avoids leaving empty contexts in the build tree that should not be there.

events_with_tokens(start_token, last_token)

Yield (Event, tokens) tuples for start_token until and including last_token.

Events are yielded together with token groups (or single tokens in a 1-length tuple).

This function is used by get_prepared_lexer() to compare an existing token structure with events originating from a lexer.

The start_token must be the first of a group, if it is a GroupToken. Additionally, it should not be the first token in a context whose lexicon has the consume flag set. But if the start_token is the very first token in a tree, it does not matter if it is not in the root context, this case is handled gracefully.

get_lexer(token)

Get a Lexer initialized at the token’s ancestry.

new_tree(token)

Return an empty context (and its root) with the same ancestry as the token’s.

find_token_before(node, pos)

A version of Context.find_token_before() that can handle empty contexts.

The new tree built inside TreeBuilder.build_new_tree() can have an empty context at the beginning and/or the end. Returns None if there is no token left from pos.

ancestors_with_index(node)

A version of Node.ancestors_with_index() that can handle empty contexts.

backward(node)

A version of Node.backward() that can handle empty contexts.

common_ancestor_with_trail(node, other)

A version of Token.common_ancestor_with_trail() that can handle empty contexts.

same_events(e1, e2)

Compare Event tuples in a robust way.

Returns True if the events are completely the same. The lexicon in a target is compared on identity instead of equality. This prevents different derived lexicons from comparing the same, which is needed when rebuilding a tree.