Lexer (xonsh.lexer
)¶
Lexer for xonsh code.
Written using a hybrid of tokenize
and PLY.
-
class
xonsh.lexer.
Lexer
[source]¶ Implements a lexer for the xonsh language.
- Attributes
- fnamestr
Filename
- lasttoken
The last token seen.
- linenoint
The last line number seen.
-
property
tokens
¶
-
xonsh.lexer.
get_tokens
(s)[source]¶ Given a string containing xonsh code, generates a stream of relevant PLY tokens using
handle_token
.
-
xonsh.lexer.
handle_error_linecont
(state, token)[source]¶ Function for handling special line continuations as whitespace characters in subprocess mode.
-
xonsh.lexer.
handle_error_space
(state, token)[source]¶ Function for handling special whitespace characters in subprocess mode
-
xonsh.lexer.
handle_ignore
(state, token)[source]¶ Function for handling tokens that should be ignored
-
xonsh.lexer.
handle_token
(state, token)[source]¶ General-purpose token handler. Makes use of
token_map
orspecial_map
to yield one or more PLY tokens from the given input.- Parameters
- state :
The current state of the lexer, including information about whether we are in Python mode or subprocess mode, which changes the lexer’s behavior. Also includes the stream of tokens yet to be considered.
- token :
The token (from
tokenize
) currently under consideration