xonsh.lexer

Lexer for xonsh code.

Written using a hybrid of tokenize and PLY.

class xonsh.lexer.Lexer(tolerant=False, pymode=True)

Implements a lexer for the xonsh language.

Attributes:

fnamestr

Filename

lasttoken

The last token seen.

linenoint

The last line number seen.

tolerantbool

Tokenize without extra checks (e.g. paren matching). When True, ERRORTOKEN contains the erroneous string instead of an error msg.

pymodebool

Start the lexer in Python mode.

build(**kwargs)

Part of the PLY lexer API.

input(s)

Calls the lexer on the string s.

reset()

split(s)

Splits a string into a list of strings which are whitespace-separated tokens.

token()

Retrieves the next token.

property tokens

property tolerant

xonsh.lexer.get_tokens(s, tolerant, pymode=True, tokenize_ioredirects=True)

Given a string containing xonsh code, generates a stream of relevant PLY tokens using handle_token.

xonsh.lexer.handle_double_amps(state, token)

xonsh.lexer.handle_double_pipe(state, token)

xonsh.lexer.handle_error_linecont(state, token)

Function for handling special line continuations as whitespace characters in subprocess mode.

xonsh.lexer.handle_error_space(state, token)

Function for handling special whitespace characters in subprocess mode

xonsh.lexer.handle_error_token(state, token)

Function for handling error tokens

xonsh.lexer.handle_ignore(state, token)

Function for handling tokens that should be ignored

xonsh.lexer.handle_name(state, token)

Function for handling name tokens

xonsh.lexer.handle_rbrace(state, token)

Function for handling }

xonsh.lexer.handle_rbracket(state, token)

Function for handling ]

xonsh.lexer.handle_redirect(state, token)

xonsh.lexer.handle_rparen(state, token)

Function for handling )

xonsh.lexer.handle_token(state, token)

General-purpose token handler. Makes use of token_map or special_map to yield one or more PLY tokens from the given input.

Parameters:

state

The current state of the lexer, including information about whether we are in Python mode or subprocess mode, which changes the lexer’s behavior. Also includes the stream of tokens yet to be considered.

token

The token (from tokenize) currently under consideration