Skip to content

Recipe subprocess

Jump to bottom Edit New page
Lucretius Biah edited this page Dec 10, 2020 · 15 revisions

Windows workarounds

Invoking

Calling subprocess.Popen or one of its variants requires care when using Pyinstaller; issue 1339 gives an example of these difficulties. The following function provides workarounds for several common problems in Windows:

  • subprocess.Popen will pop up a command window by default when run from Pyinstaller with the --noconsole option.
  • Windows doesn't search the path by default.
  • Running this from the binary produced by Pyinstaller with the --noconsole option requires redirecting everything (stdin, stdout, stderr) to avoid an OSError exception: "[Error 6] the handle is invalid."

This code was based on an Enki library.

import subprocess
import os, os.path
import sys

# Create a set of arguments which make a ``subprocess.Popen`` (and
# variants) call work with or without Pyinstaller, ``--noconsole`` or
# not, on Windows and Linux. Typical use::
#
#   subprocess.call(['program_to_run', 'arg_1'], **subprocess_args())
#
# When calling ``check_output``::
#
#   subprocess.check_output(['program_to_run', 'arg_1'],
#                           **subprocess_args(False))
def subprocess_args(include_stdout=True):
    # The following is true only on Windows.
    if hasattr(subprocess, 'STARTUPINFO'):
        # On Windows, subprocess calls will pop up a command window by default
        # when run from Pyinstaller with the ``--noconsole`` option. Avoid this
        # distraction.
        si = subprocess.STARTUPINFO()
        si.dwFlags |= subprocess.STARTF_USESHOWWINDOW
        # Windows doesn't search the path by default. Pass it an environment so
        # it will.
        env = os.environ
    else:
        si = None
        env = None

    # ``subprocess.check_output`` doesn't allow specifying ``stdout``::
    #
    #   Traceback (most recent call last):
    #     File "test_subprocess.py", line 58, in <module>
    #       **subprocess_args(stdout=None))
    #     File "C:\Python27\lib\subprocess.py", line 567, in check_output
    #       raise ValueError('stdout argument not allowed, it will be overridden.')
    #   ValueError: stdout argument not allowed, it will be overridden.
    #
    # So, add it only if it's needed.
    if include_stdout:
        ret = {'stdout': subprocess.PIPE}
    else:
        ret = {}

    # On Windows, running this from the binary produced by Pyinstaller
    # with the ``--noconsole`` option requires redirecting everything
    # (stdin, stdout, stderr) to avoid an OSError exception
    # "[Error 6] the handle is invalid."
    ret.update({'stdin': subprocess.PIPE,
                'stderr': subprocess.PIPE,
                'startupinfo': si,
                'env': env })
    return ret

# A simple test routine. Compare this output when run by Python, Pyinstaller,
# and Pyinstaller ``--noconsole``.
if __name__ == '__main__':
    # Save the output from invoking Python to a text file. This is the only
    # option when running with ``--noconsole``.
    with open('out.txt', 'w') as f:
        try:
            txt = subprocess.check_output(['python', '--help'],
                                          **subprocess_args(False))
            f.write(txt)
        except OSError as e:
            f.write('Failed: ' + str(e))

Dangling file handles

On Windows under PyInstaller, the subprocess launched by subprocess.Popen inherits open filehandles from the parent, and this includes the filehandle opened for the parent's own executable file. Therefore, if your Python code (the parent) creates a new process (the child), then the parent terminates, the parent doesn't disappear completely: there is still a lock remaining on the .exe of the parent. One of the consequences is that you can't delete/modify the parent .exe from the child's process (like, you can't use a child .exe to update the first one).

On Linux and Darwin you can get away with a simple call to subprocess.Popen. On Windows you need to add the option close_fds=True in your subprocess.Popen call:

import subprocess

subprocess.Popen("myexecutable.exe", close_fds=True)
"""Now my Python code can exit, and none of its file handles will remain open."""

Windows DLL loading order

In the bootloader, PyInstaller calls SetDllDirectory so that some ctypes modules will work as expected. This behavior is Windows specific and does not respect the PATH loading order; the PyInstaller DLLs will be loaded first, regardless of PATH. See issue 3795 for additional details.

If you don't need this ctypes support, the following code will restore the standard Windows DLL loading order.

import sys
if sys.platform == "win32":
    import ctypes
    ctypes.windll.kernel32.SetDllDirectoryA(None)

https://github.com/pyinstaller/pyinstaller.wiki.git

https://github.com/pyinstaller/pyinstaller.wiki.git

Clone this wiki locally