Python Procedures as Subprocess Commands (xonsh.proc)

Interface for running Python functions as subprocess-mode commands.

Code for several helper methods in the ProcProxy class have been reproduced without modification from subprocess.py in the Python 3.4.2 standard library. The contents of subprocess.py (and, thus, the reproduced methods) are Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se> and were licensed to the Python Software foundation under a Contributor Agreement.

class xonsh.proc.BufferedFDParallelReader(fd, buffer=None, chunksize=1024)

Buffered, parallel background thread reader.

Parameters

fdint

File descriptor from which to read.

bufferbinary file-like or None, optional

A buffer to write bytes into. If None, a new BytesIO object is created.

chunksizeint, optional

The max size of the parallel reads, default 1 kb.

class xonsh.proc.CommandPipeline(specs)

Represents a subprocess-mode command pipeline.

Parameters

specslist of SubprocSpec

Process specifications

Attributes

specSubprocSpec

The last specification in specs

procPopen-like

The process in procs

endedbool

Boolean for if the command has stopped executing.

inputstr

A string of the standard input.

outputstr

Non-blocking, lazy access to output

errorsstr

A string of the standard error.

lineslist of str

The output lines

starttimefloats or None

Pipeline start timestamp.

end(tee_output=True)

End the pipeline, return the controlling terminal if needed.

Main things done in self._end().

itercheck()

Iterates through the command lines and throws an error if the returncode is non-zero.

iterraw()

Iterates through the last stdout, and returns the lines exactly as found.

resume(job, tee_output=True)

stream_stderr(lines)

Streams lines to sys.stderr and the errors attribute.

tee_stdout()

Writes the process stdout to the output variable, line-by-line, and yields each line. This may optionally accept lines (in bytes) to iterate over, in which case it does not call iterraw().

property alias

Alias the process used.

property args

Arguments to the process.

attrnames = ('stdin', 'stdout', 'stderr', 'pid', 'returncode', 'args', 'alias', 'stdin_redirect', 'stdout_redirect', 'stderr_redirect', 'timestamps', 'executed_cmd', 'input', 'output', 'errors')

property err

Error messages as a string.

property executed_cmd

The resolve and executed command.

property inp

Creates normalized input string from args.

nonblocking = (<class '_io.BytesIO'>, <class 'xonsh.proc.NonBlockingFDReader'>, <class 'xonsh.proc.ConsoleParallelReader'>)

property out

Output value as a str.

property output

Non-blocking, lazy access to output

property pid

Process identifier.

property returncode

Process return code, waits until command is completed.

property rtn

Alias to return code.

property stderr

Process stderr.

property stderr_postfix

Postfix to print after stderr, as bytes.

property stderr_prefix

Prefix to print in front of stderr, as bytes.

property stderr_redirect

Redirection used for stderr.

property stdin

Process stdin.

property stdin_redirect

Redirection used for stdin.

property stdout

Process stdout.

property stdout_redirect

Redirection used for stdout.

property timestamps

The start and end time stamps.

class xonsh.proc.ConsoleParallelReader(fd, buffer=None, chunksize=1024, timeout=None)

Parallel reader for consoles that runs in a background thread. This is only needed, available, and useful on Windows.

Parameters

fdint

Standard buffer file descriptor, 0 for stdin, 1 for stdout (default), and 2 for stderr.

bufferctypes.c_wchar_p, optional

An existing buffer to (re-)use.

chunksizeint, optional

The max size of the parallel reads, default 1 kb.

timeoutfloat, optional

The queue reading timeout.

class xonsh.proc.FileThreadDispatcher(default=None)

Dispatches to different file handles depending on the current thread. Useful if you want file operation to go to different places for different threads.

Parameters

defaultfile-like or None, optional

The file handle to write to if a thread cannot be found in the registry. If None, a new in-memory instance.

Attributes

registrydict

Maps thread idents to file handles.

close()

Closes the current thread’s handle.

deregister()

Removes the current thread from the registry.

detach()

Detaches the buffer for the current thread.

fileno()

Returns the file descriptor for the current thread.

flush()

Flushes the file descriptor for the current thread.

isatty()

Returns if the file descriptor for the current thread is a tty.

read(size=None)

Reads from the handle for the current thread.

readable()

Returns if file descriptor for the current thread is readable.

readline(size=-1)

Reads a line from the handle for the current thread.

readlines(hint=-1)

Reads lines from the handle for the current thread.

register(handle)

Registers a file handle for the current thread. Returns self so that this method can be used in a with-statement.

seek(offset, whence=0)

Seeks the current file.

seekable()

Returns if file descriptor for the current thread is seekable.

tell()

Reports the current position in the handle for the current thread.

truncate(size=None)

Truncates the file for for the current thread.

writable(size=None)

Returns if file descriptor for the current thread is writable.

write(s)

Writes to this thread’s handle. This also flushes, just to be extra sure the string was written.

writelines()

Writes lines for the file descriptor for the current thread.

property available

True if the thread is available in the registry.

property buffer

Gets the buffer for this thread’s handle.

property closed

Is the thread’s handle closed.

property encoding

Gets the encoding for this thread’s handle.

property errors

Gets the errors for this thread’s handle.

property handle

Gets the current handle for the thread.

property line_buffering

Gets if line buffering for this thread’s handle enabled.

property newlines

Gets the newlines for this thread’s handle.

class xonsh.proc.Handle

Close(CloseHandle=None)

Detach()

closed = False

class xonsh.proc.HiddenCommandPipeline(specs)

Parameters

specslist of SubprocSpec

Process specifications

Attributes

specSubprocSpec

The last specification in specs

procPopen-like

The process in procs

endedbool

Boolean for if the command has stopped executing.

inputstr

A string of the standard input.

outputstr

Non-blocking, lazy access to output

errorsstr

A string of the standard error.

lineslist of str

The output lines

starttimefloats or None

Pipeline start timestamp.

class xonsh.proc.NonBlockingFDReader(fd, timeout=None)

A class for reading characters from a file descriptor on a background thread. This has the advantages that the calling thread can close the file and that the reading does not block the calling thread.

Parameters

fdint

A file descriptor

timeoutfloat or None, optional

The queue reading timeout.

class xonsh.proc.PopenThread(*args, stdin=None, stdout=None, stderr=None, **kwargs)

A thread for running and managing subprocess. This allows reading from the stdin, stdout, and stderr streams in a non-blocking fashion.

This takes the same arguments and keyword arguments as regular Popen. This requires that the captured_stdout and captured_stderr attributes to be set following instantiation.

This constructor should always be called with keyword arguments. Arguments are:

group should be None; reserved for future extension when a ThreadGroup class is implemented.

target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called.

name is the thread name. By default, a unique name is constructed of the form “Thread-N” where N is a small decimal number.

args is the argument tuple for the target invocation. Defaults to ().

kwargs is a dictionary of keyword arguments for the target invocation. Defaults to {}.

If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread.

kill()

Dispatches to Popen.kill().

poll()

Dispatches to Popen.returncode.

run()

Runs the subprocess by performing a parallel read on stdin if allowed, and copying bytes from captured_stdout to stdout and bytes from captured_stderr to stderr.

send_signal(signal)

Dispatches to Popen.send_signal().

terminate()

Dispatches to Popen.terminate().

wait(timeout=None)

Dispatches to Popen.wait(), but also does process cleanup such as joining this thread and replacing the original window size signal handler.

property returncode

Process return code.

property signal

Process signal, or None.

class xonsh.proc.PrevProcCloser(pipeline)

Previous process closer thread for pipelines whose last command is itself unthreadable. This makes sure that the pipeline is driven forward and does not deadlock.

Parameters

pipelineCommandPipeline

The pipeline whose prev procs we should close.

run()

Runs the closing algorithm.

class xonsh.proc.ProcProxy(f, args, stdin=None, stdout=None, stderr=None, universal_newlines=False, close_fds=False, env=None)

This is process proxy class that runs its alias functions on the same thread that it was called from, which is typically the main thread. This prevents the process from running on a background thread, but enables debugger and profiler tools (functions) be run on the same thread that they are attempting to debug.

poll()

Check if the function has completed via the returncode or None.

wait(timeout=None)

Runs the function and returns the result. Timeout argument only present for API compatibility.

class xonsh.proc.ProcProxyThread(f, args, stdin=None, stdout=None, stderr=None, universal_newlines=False, close_fds=False, env=None)

Class representing a function to be run as a subprocess-mode command.

Parameters

ffunction

The function to be executed.

argslist

A (possibly empty) list containing the arguments that were given on the command line

stdinfile-like, optional

A file-like object representing stdin (input can be read from here). If stdin is not provided or if it is explicitly set to None, then an instance of io.StringIO representing an empty file is used.

stdoutfile-like, optional

A file-like object representing stdout (normal output can be written here). If stdout is not provided or if it is explicitly set to None, then sys.stdout is used.

stderrfile-like, optional

A file-like object representing stderr (error output can be written here). If stderr is not provided or if it is explicitly set to None, then sys.stderr is used.

universal_newlinesbool, optional

Whether or not to use universal newlines.

close_fdsbool, optional

Whether or not to close file descriptors. This is here for Popen compatability and currently does nothing.

envMapping, optional

Environment mapping.

poll()

Check if the function has completed.

Returns

None if the function is still executing, and the returncode otherwise

run()

Set up input/output streams and execute the child function in a new thread. This is part of the threading.Thread interface and should not be called directly.

wait(timeout=None)

Waits for the process to finish and returns the return code.

class xonsh.proc.QueueReader(fd, timeout=None)

Provides a file-like interface to reading from a queue.

Parameters

fdint

A file descriptor

timeoutfloat or None, optional

The queue reading timeout.

close()

close the reader

fileno()

Returns the file descriptor number.

is_fully_read()

Returns whether or not the queue is fully read and the reader is closed.

iterqueue()

Iterates through all remaining chunks in a blocking fashion.

read(size=-1)

Reads bytes from the file.

read_queue()

Reads a single chunk from the queue. This is blocking if the timeout is None and non-blocking otherwise.

static readable()

Returns true, because this object is always readable.

readline(size=-1)

Reads a line, or a partial line from the file descriptor.

readlines(hint=-1)

Reads lines from the file descriptor. This is blocking for negative hints (i.e. read all the remaining lines) and non-blocking otherwise.

xonsh.proc.parse_proxy_return(r, stdout, stderr)

Proxies may return a variety of outputs. This handles them generally.

Parameters

rtuple, str, int, or None

Return from proxy function

stdoutfile-like

Current stdout stream

stdoutfile-like

Current stderr stream

Returns

cmd_resultint

The return code of the proxy

xonsh.proc.partial_proxy(f)

Dispatches the appropriate proxy function based on the number of args.

xonsh.proc.pause_call_resume(p, f, *args, **kwargs)

For a process p, this will call a function f with the remaining args and and kwargs. If the process cannot accept signals, the function will be called.

Parameters

pPopen object or similar

fcallable

argsremaining arguments

kwargskeyword arguments

xonsh.proc.populate_buffer(reader, fd, buffer, chunksize)

Reads bytes from the file descriptor and copies them into a buffer.

The reads happen in parallel using the pread() syscall; which is only available on POSIX systems. If the read fails for any reason, the reader is flagged as closed.

xonsh.proc.populate_console(reader, fd, buffer, chunksize, queue, expandsize=None)

Reads bytes from the file descriptor and puts lines into the queue. The reads happened in parallel, using xonsh.winutils.read_console_output_character(), and is thus only available on windows. If the read fails for any reason, the reader is flagged as closed.

xonsh.proc.populate_fd_queue(reader, fd, queue)

Reads 1 kb of data from a file descriptor into a queue. If this ends or fails, it flags the calling reader object as closed.

xonsh.proc.proxy_five(f, args, stdin, stdout, stderr, spec, stack)

Calls a proxy function which takes four parameter: args, stdin, stdout, stderr, and spec.

xonsh.proc.proxy_four(f, args, stdin, stdout, stderr, spec, stack)

Calls a proxy function which takes four parameter: args, stdin, stdout, and stderr.

xonsh.proc.proxy_one(f, args, stdin, stdout, stderr, spec, stack)

Calls a proxy function which takes one parameter: args

xonsh.proc.proxy_three(f, args, stdin, stdout, stderr, spec, stack)

Calls a proxy function which takes three parameter: args, stdin, stdout.

xonsh.proc.proxy_two(f, args, stdin, stdout, stderr, spec, stack)

Calls a proxy function which takes two parameter: args and stdin.

xonsh.proc.proxy_zero(f, args, stdin, stdout, stderr, spec, stack)

Calls a proxy function which takes no parameters.

xonsh.proc.safe_fdclose(handle, cache=None)

Closes a file handle in the safest way possible, and potentially storing the result.

xonsh.proc.safe_flush(handle)

Attempts to safely flush a file handle, returns success bool.

xonsh.proc.safe_readable(handle)

Attempts to find if the handle is readable without throwing an error.

xonsh.proc.safe_readlines(handle, hint=-1)

Attempts to read lines without throwing an error.

xonsh.proc.still_writable(fd)

Determines whether a file descriptor is still writable by trying to write an empty string and seeing if it fails.

xonsh.proc.update_fg_process_group(pipeline_group, background)