Python Procedures as Subprocess Commands (xonsh.proc)

Interface for running Python functions as subprocess-mode commands.

Code for several helper methods in the ProcProxy class have been reproduced without modification from subprocess.py in the Python 3.4.2 standard library. The contents of subprocess.py (and, thus, the reproduced methods) are Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se> and were licensed to the Python Software foundation under a Contributor Agreement.

class xonsh.proc.BufferedFDParallelReader(fd, buffer=None, chunksize=1024)

Buffered, parallel background thread reader.

Parameters:

fd : int

File descriptor from which to read.

buffer : binary file-like or None, optional

A buffer to write bytes into. If None, a new BytesIO object is created.

chunksize : int, optional

The max size of the parallel reads, default 1 kb.

class xonsh.proc.CommandPipeline(specs)

Represents a subprocess-mode command pipeline.

Parameters:

specs : list of SubprocSpec

Process specifications

Attributes:

spec : SubprocSpec

The last specification in specs

proc : Popen-like

The process in procs

ended : bool

Boolean for if the command has stopped executing.

input : str

A string of the standard input.

output : str

Non-blocking, lazy access to output

errors : str

A string of the standard error.

lines : list of str

The output lines

starttime : floats or None

Pipeline start timestamp.

end(tee_output=True)

End the pipeline, return the controlling terminal if needed.

Main things done in self._end().

itercheck()

Iterates through the command lines and throws an error if the returncode is non-zero.

iterraw()

Iterates through the last stdout, and returns the lines exactly as found.

resume(job, tee_output=True)

stream_stderr(lines)

Streams lines to sys.stderr and the errors attribute.

tee_stdout()

Writes the process stdout to the output variable, line-by-line, and yields each line. This may optionally accept lines (in bytes) to iterate over, in which case it does not call iterraw().

alias

Alias the process used.

args

Arguments to the process.

attrnames = ('stdin', 'stdout', 'stderr', 'pid', 'returncode', 'args', 'alias', 'stdin_redirect', 'stdout_redirect', 'stderr_redirect', 'timestamps', 'executed_cmd', 'input', 'output', 'errors')

err

Error messages as a string.

executed_cmd

The resolve and executed command.

inp

Creates normalized input string from args.

nonblocking = (<class '_io.BytesIO'>, <class 'xonsh.proc.NonBlockingFDReader'>, <class 'xonsh.proc.ConsoleParallelReader'>)

out

Output value as a str.

output

Non-blocking, lazy access to output

pid

Process identifier.

returncode

Process return code, waits until command is completed.

rtn

Alias to return code.

stderr

Process stderr.

stderr_postfix

Postfix to print after stderr, as bytes.

stderr_prefix

Prefix to print in front of stderr, as bytes.

stderr_redirect

Redirection used for stderr.

stdin

Process stdin.

stdin_redirect

Redirection used for stdin.

stdout

Process stdout.

stdout_redirect

Redirection used for stdout.

timestamps

The start and end time stamps.

class xonsh.proc.ConsoleParallelReader(fd, buffer=None, chunksize=1024, timeout=None)

Parallel reader for consoles that runs in a background thread. This is only needed, available, and useful on Windows.

Parameters:

fd : int

Standard buffer file descriptor, 0 for stdin, 1 for stdout (default), and 2 for stderr.

buffer : ctypes.c_wchar_p, optional

An existing buffer to (re-)use.

chunksize : int, optional

The max size of the parallel reads, default 1 kb.

timeout : float, optional

The queue reading timeout.

class xonsh.proc.FileThreadDispatcher(default=None)

Dispatches to different file handles depending on the current thread. Useful if you want file operation to go to different places for different threads.

Parameters:

default : file-like or None, optional

The file handle to write to if a thread cannot be found in the registry. If None, a new in-memory instance.

Attributes:

registry : dict

Maps thread idents to file handles.

close()

Closes the current thread’s handle.

deregister()

Removes the current thread from the registry.

detach()

Detaches the buffer for the current thread.

fileno()

Returns the file descriptor for the current thread.

flush()

Flushes the file descriptor for the current thread.

isatty()

Returns if the file descriptor for the current thread is a tty.

read(size=None)

Reads from the handle for the current thread.

readable()

Returns if file descriptor for the current thread is readable.

readline(size=-1)

Reads a line from the handle for the current thread.

readlines(hint=-1)

Reads lines from the handle for the current thread.

register(handle)

Registers a file handle for the current thread. Returns self so that this method can be used in a with-statement.

seek(offset, whence=0)

Seeks the current file.

seekable()

Returns if file descriptor for the current thread is seekable.

tell()

Reports the current position in the handle for the current thread.

truncate(size=None)

Truncates the file for for the current thread.

writable(size=None)

Returns if file descriptor for the current thread is writable.

write(s)

Writes to this thread’s handle. This also flushes, just to be extra sure the string was written.

writelines()

Writes lines for the file descriptor for the current thread.

available

True if the thread is available in the registry.

buffer

Gets the buffer for this thread’s handle.

closed

Is the thread’s handle closed.

encoding

Gets the encoding for this thread’s handle.

errors

Gets the errors for this thread’s handle.

handle

Gets the current handle for the thread.

line_buffering

Gets if line buffering for this thread’s handle enabled.

newlines

Gets the newlines for this thread’s handle.

class xonsh.proc.Handle

Close(CloseHandle=None)

Detach()

closed = False

class xonsh.proc.HiddenCommandPipeline(specs)

Parameters:

specs : list of SubprocSpec

Process specifications

Attributes:

spec : SubprocSpec

The last specification in specs

proc : Popen-like

The process in procs

ended : bool

Boolean for if the command has stopped executing.

input : str

A string of the standard input.

output : str

Non-blocking, lazy access to output

errors : str

A string of the standard error.

lines : list of str

The output lines

starttime : floats or None

Pipeline start timestamp.

class xonsh.proc.NonBlockingFDReader(fd, timeout=None)

A class for reading characters from a file descriptor on a background thread. This has the advantages that the calling thread can close the file and that the reading does not block the calling thread.

Parameters:

fd : int

A file descriptor

timeout : float or None, optional

The queue reading timeout.

class xonsh.proc.PopenThread(*args, stdin=None, stdout=None, stderr=None, **kwargs)

A thread for running and managing subprocess. This allows reading from the stdin, stdout, and stderr streams in a non-blocking fashion.

This takes the same arguments and keyword arguments as regular Popen. This requires that the captured_stdout and captured_stderr attributes to be set following instantiation.

kill()

Dispatches to Popen.kill().

poll()

Dispatches to Popen.returncode.

run()

Runs the subprocess by performing a parallel read on stdin if allowed, and copying bytes from captured_stdout to stdout and bytes from captured_stderr to stderr.

send_signal(signal)

Dispatches to Popen.send_signal().

terminate()

Dispatches to Popen.terminate().

wait(timeout=None)

Dispatches to Popen.wait(), but also does process cleanup such as joining this thread and replacing the original window size signal handler.

returncode

Process return code.

signal

Process signal, or None.

class xonsh.proc.PrevProcCloser(pipeline)

Previous process closer thread for pipelines whose last command is itself unthreadable. This makes sure that the pipeline is driven forward and does not deadlock.

Parameters:

pipeline : CommandPipeline

The pipeline whose prev procs we should close.

run()

Runs the closing algorithm.

class xonsh.proc.ProcProxy(f, args, stdin=None, stdout=None, stderr=None, universal_newlines=False, env=None)

This is process proxy class that runs its alias functions on the same thread that it was called from, which is typically the main thread. This prevents the process from running on a background thread, but enables debugger and profiler tools (functions) be run on the same thread that they are attempting to debug.

poll()

Check if the function has completed via the returncode or None.

wait(timeout=None)

Runs the function and returns the result. Timeout argument only present for API compatibility.

class xonsh.proc.ProcProxyThread(f, args, stdin=None, stdout=None, stderr=None, universal_newlines=False, env=None)

Class representing a function to be run as a subprocess-mode command.

Parameters:

f : function

The function to be executed.

args : list

A (possibly empty) list containing the arguments that were given on the command line

stdin : file-like, optional

A file-like object representing stdin (input can be read from here). If stdin is not provided or if it is explicitly set to None, then an instance of io.StringIO representing an empty file is used.

stdout : file-like, optional

A file-like object representing stdout (normal output can be written here). If stdout is not provided or if it is explicitly set to None, then sys.stdout is used.

stderr : file-like, optional

A file-like object representing stderr (error output can be written here). If stderr is not provided or if it is explicitly set to None, then sys.stderr is used.

universal_newlines : bool, optional

Whether or not to use universal newlines.

env : Mapping, optional

Environment mapping.

poll()

Check if the function has completed.

Returns:

None if the function is still executing, and the returncode otherwise

run()

Set up input/output streams and execute the child function in a new thread. This is part of the threading.Thread interface and should not be called directly.

wait(timeout=None)

Waits for the process to finish and returns the return code.

class xonsh.proc.QueueReader(fd, timeout=None)

Provides a file-like interface to reading from a queue.

Parameters:

fd : int

A file descriptor

timeout : float or None, optional

The queue reading timeout.

close()

close the reader

fileno()

Returns the file descriptor number.

is_fully_read()

Returns whether or not the queue is fully read and the reader is closed.

iterqueue()

Iterates through all remaining chunks in a blocking fashion.

read(size=-1)

Reads bytes from the file.

read_queue()

Reads a single chunk from the queue. This is blocking if the timeout is None and non-blocking otherwise.

static readable()

Returns true, because this object is always readable.

readline(size=-1)

Reads a line, or a partial line from the file descriptor.

readlines(hint=-1)

Reads lines from the file descriptor. This is blocking for negative hints (i.e. read all the remaining lines) and non-blocking otherwise.

xonsh.proc.parse_proxy_return(r, stdout, stderr)

Proxies may return a variety of outputs. This handles them generally.

Parameters:

r : tuple, str, int, or None

Return from proxy function

stdout : file-like

Current stdout stream

stdout : file-like

Current stderr stream

Returns:

cmd_result : int

The return code of the proxy

xonsh.proc.partial_proxy(f)

Dispatches the appropriate proxy function based on the number of args.

xonsh.proc.pause_call_resume(p, f, *args, **kwargs)

For a process p, this will call a function f with the remaining args and and kwargs. If the process cannot accept signals, the function will be called.

Parameters:

p : Popen object or similar

f : callable

args : remaining arguments

kwargs : keyword arguments

xonsh.proc.populate_buffer(reader, fd, buffer, chunksize)

Reads bytes from the file descriptor and copies them into a buffer.

The reads happen in parallel using the pread() syscall; which is only available on POSIX systems. If the read fails for any reason, the reader is flagged as closed.

xonsh.proc.populate_console(reader, fd, buffer, chunksize, queue, expandsize=None)

Reads bytes from the file descriptor and puts lines into the queue. The reads happened in parallel, using xonsh.winutils.read_console_output_character(), and is thus only available on windows. If the read fails for any reason, the reader is flagged as closed.

xonsh.proc.populate_fd_queue(reader, fd, queue)

Reads 1 kb of data from a file descriptor into a queue. If this ends or fails, it flags the calling reader object as closed.

xonsh.proc.proxy_four(f, args, stdin, stdout, stderr, spec)

Calls a proxy function which takes four parameter: args, stdin, stdout, and stderr.

xonsh.proc.proxy_one(f, args, stdin, stdout, stderr, spec)

Calls a proxy function which takes one parameter: args

xonsh.proc.proxy_three(f, args, stdin, stdout, stderr, spec)

Calls a proxy function which takes three parameter: args, stdin, stdout.

xonsh.proc.proxy_two(f, args, stdin, stdout, stderr, spec)

Calls a proxy function which takes two parameter: args and stdin.

xonsh.proc.proxy_zero(f, args, stdin, stdout, stderr, spec)

Calls a proxy function which takes no parameters.

xonsh.proc.safe_fdclose(handle, cache=None)

Closes a file handle in the safest way possible, and potentially storing the result.

xonsh.proc.safe_flush(handle)

Attempts to safely flush a file handle, returns success bool.

xonsh.proc.safe_readable(handle)

Attempts to find if the handle is readable without throwing an error.

xonsh.proc.safe_readlines(handle, hint=-1)

Attempts to read lines without throwing an error.

xonsh.proc.still_writable(fd)

Determines whether a file descriptor is still writable by trying to write an empty string and seeing if it fails.

xonsh.proc.update_fg_process_group(pipeline_group, background)