ci_exec

Contents

• ci_exec

  • About

• Installation

• License

  • Intended Audience

  • Full Documentation

About

A wrapper package designed for running continuous integration (CI) build steps using Python 3.6+.

Managing cross platform build scripts for CI can become tedious at times when you need to e.g., maintain two nearly identical scripts install_deps.sh and install_deps.bat due to incompatible syntaxes. ci_exec enables a single file to manage this using Python.

The ci_exec package provides a set of wrappers / utility functions designed specifically for running build steps on CI providers. It is

Logging by Default

Commands executed, including their full command-line arguments, are logged. This includes any output on stdout / stderr from the commands. The logging resembles what set -x would give you in a shell script. For commands that will take a long time, as long as output is being produced, this will additionally prevent timeouts on the build.

Failing by Default

Any command that does not succeed will fail the entire build. An attempt to exit with the same exit code as the command that failed will be performed. Meaning the CI provider will correctly report a failed build.

Convenient

ci_exec affords users the ability to write shell-like scripts that will work on any platform that Python can run on. A simple example:

from ci_exec import cd, which

cmake = which("cmake")
ninja = which("ninja")
with cd("build", create=True):
    cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release")
    ninja("-j", "2", "test")

Installation

ci_exec is available on PyPI. It can be installed using your python package manager of choice:

$ pip install ci-exec

Note

The PyPI package has a -: ci-exec, not ci_exec.

There is also a setup.py here, so you can also install it from source:

$ pip install git+https://github.com/svenevs/ci_exec.git@master

License

This software is licensed under the Apache 2.0 license.

Intended Audience

Note

ci_exec can be used for anything related to writing build steps, but it was originally written to manage C++ projects. The documentation will often have examples using cmake and ninja, users do not need to understand what these commands are for.

ci_exec utilizes some “advanced” features of Python that pertain to how the library itself is consumed. It may not be appropriate for users who do not know any Python at all. The main features a user should be aware of:

• *args and **kwargs are used liberally. ci_exec mostly consists of wrapper classes / functions around the python standard library, and in most cases *args and **kwargs are the “pass-through” parameters.

• Keyword-only arguments. Many library signatures look something like:

  def foo(a: str, *, b: int = 2):
      pass
  
  foo("hi")       # OK: b = 2
  foo("hi", 3)    # ERROR: b is keyword only
  foo("hi", b=3)  # OK: b = 3
  

  Anything after the *, must be named explicitly.

• Operator overloading, particularly what __call__ means and how it works. A quick overview:

  from ci_exec import Executable
  
  # Typically: prefer ci_exec.which instead, which returns a ci_exec.Executable.
  cmake = Executable("/usr/bin/cmake")
  
  # cmake.__call__ invoked with args = [".."], kwargs = {}
  cmake("..")
  
  # cmake.__call__ invoked with args = [".."], kwargs = {"check": False}
  cmake("..", check=False)
  

None of these features are altogether that special, but it must be stated clearly and plainly: this library is designed for users who already know Python.

Note

C++ users are encouraged to look at conan as an alternative. ci_exec has zero intention of becoming a package manager, and was written to help manage projects that are not well suited to conan for various reasons.

Full Documentation

Quick reference:

Ansi()	Wrapper class for defining the escape character and clear sequence.
Colors()	The core ANSI color codes.
Styles()	A non-exhaustive list of ANSI style formats.
colorize(message, *, color)	Return message colorized with specified style.
log_stage(stage, *[, …])	Print a terminal width block with stage message in the middle.
Executable(exe_path, *[, …])	Represent a reusable executable.
fail(why, *[, exit_code, no_prefix])	Write a failure message to sys.stderr and exit.
mkdir_p(path[, mode, parents, …])	Permissive wrapper around pathlib.Path.mkdir().
rm_rf(path[, ignore_errors, …])	Permissive wrapper around shutil.rmtree() bypassing FileNotFoundError and NotADirectoryError.
which(cmd, *[, mode, path])	Restrictive wrapper around shutil.which() that will fail() if not found.
CMakeParser(*)	A CMake focused argument parser.
filter_file(path, pattern, repl)	Filter the contents of a file.
unified_diff(from_path, to_path)	Return the unified_diff between two files.
Provider()	Check if code is executing on a continuous integration (CI) service.
cd(dest, *[, create])	Context manager / decorator that can be used to change directories.
merge_kwargs(defaults, kwargs)	Merge defaults into kwargs and return kwargs.
set_env(**kwargs)	Context manager / decorator that can be used to set environment variables.
unset_env(*args)	Context manager / decorator that can be used to unset environment variables.

ci_exec

The ci_exec package top-level namespace.

Quick Reference:

Ansi()	Wrapper class for defining the escape character and clear sequence.
Colors()	The core ANSI color codes.
Styles()	A non-exhaustive list of ANSI style formats.
colorize(message, *, color)	Return message colorized with specified style.
log_stage(stage, *[, …])	Print a terminal width block with stage message in the middle.
Executable(exe_path, *[, …])	Represent a reusable executable.
fail(why, *[, exit_code, no_prefix])	Write a failure message to sys.stderr and exit.
mkdir_p(path[, mode, parents, …])	Permissive wrapper around pathlib.Path.mkdir().
rm_rf(path[, ignore_errors, …])	Permissive wrapper around shutil.rmtree() bypassing FileNotFoundError and NotADirectoryError.
which(cmd, *[, mode, path])	Restrictive wrapper around shutil.which() that will fail() if not found.
CMakeParser(*)	A CMake focused argument parser.
filter_file(path, pattern, repl)	Filter the contents of a file.
unified_diff(from_path, to_path)	Return the unified_diff between two files.
Provider()	Check if code is executing on a continuous integration (CI) service.
cd(dest, *[, create])	Context manager / decorator that can be used to change directories.
merge_kwargs(defaults, kwargs)	Merge defaults into kwargs and return kwargs.
set_env(**kwargs)	Context manager / decorator that can be used to set environment variables.
unset_env(*args)	Context manager / decorator that can be used to unset environment variables.

ci_exec.core

The core functionality of the ci_exec package.

fail(why, *[, exit_code, no_prefix])	Write a failure message to sys.stderr and exit.
Executable(exe_path, *[, log_calls, …])	Represent a reusable executable.
mkdir_p(path[, mode, parents, exist_ok])	Permissive wrapper around pathlib.Path.mkdir().
rm_rf(path[, ignore_errors, onerror])	Permissive wrapper around shutil.rmtree() bypassing FileNotFoundError and NotADirectoryError.
which(cmd, *[, mode, path])	Restrictive wrapper around shutil.which() that will fail() if not found.

fail(why, *, exit_code=1, no_prefix=False)

Write a failure message to sys.stderr and exit.

Parameters

• why (str) – The message explaining why the program is being failed out.

• exit_code (int) – The exit code to use. Default: 1.

• no_prefix (bool) – Whether to prefix a bold red "[X] " before why. Default: False, the bold red "[X] " prefix is included unless set to True.

class Executable(exe_path, *, log_calls=True, log_prefix='$ ', log_color='36', log_style='1')

Represent a reusable executable.

Each executable is:

1. Failing by default: unless called with check=False, any execution that fails (has a non-zero exit code) will result in a call to fail(), terminating the entire application.

2. Logging by default: every call executed will print what will be run in color and then dump the output of the command. In the event of failure, this makes finding the last call issued much simpler.

Consider the following simple script:

from ci_exec import Executable
git = Executable("/usr/bin/git")
git("remote")
# Oops! --pretty not --petty ;)
git("log", "-1", "--petty=%B")
git("status")  # will not execute (previous failed)

When we execute python simple.py and check the exit code with echo $?:

> python simple.py
$ /usr/bin/git remote
origin
$ /usr/bin/git log -1 --petty=%B
fatal: unrecognized argument: --petty=%B
[X] Command '('/usr/bin/git', 'log', '-1', "--petty=%B")' returned
    non-zero exit status 128.
> echo $?
128

See __call__() for more information.

Tip

Hard-coded paths in these examples were for demonstrative purposes. In practice this should not be done, use which() instead.

exe_path

The path to the executable that will be run when called.

Type

str

log_calls

Whether or not every invocation of __call__() should print what will execute before executing it. Default: True.

Type

bool

log_prefix

The prefix to use when printing a given invocation of __call__(). Default: "$ " to simulate a console lexer. Set to the empty string "" to have no prefix.

Type

str

log_color

The color code to use when calling colorize() to display the next invocation of __call__(). Set to None to disable colorizing each log of __call__(). Default: Colors.Cyan.

Type

str

log_style

The style code to use when calling colorize() to display the next invocation of __call__(). If no colors are desired, set log_color to None. Default: Styles.Bold.

Type

str

Raises

ValueError – If exe_path is not a file, or if it is not executable.

PATH_EXTENSIONS = {}

The set of valid file extensions that can be executed on Windows.

On *nix systems this will be the empty set, and takes no meaning. On Windows, it is controlled by the user. These are stored in lower case, and comparisons should be lower case for consistency. The typical default value on Windows would be:

PATH_EXTENSIONS = {".com", ".exe", ".bat", ".cmd"}

__call__(*args, **kwargs)

Run exe_path with the specified command-line *args.

The usage of the parameters is best summarized in code:

popen_args = (self.exe_path, *args)
# ... some potential logging ...
return subprocess.run(popen_args, **kwargs)

For example, sending multiple arguments to the executable is as easy as:

cmake = Executable("/usr/bin/cmake")
cmake("..", "-G", "Ninja", "-DBUILD_SHARED_LIBS=ON")

and any overrides to subprocess.run() you wish to include should be done with **kwargs, which are forwarded directly.

Warning

Any exceptions generated result in a call to fail(), which will terminate the application.

Parameters

• *args – The positional arguments will be forwarded along with exe_path to subprocess.run().

• **kwargs –

  The key-value arguments are all forwarded to subprocess.run(). If check is not provided, this is an implicit check=True. That is, if you do not want the application to exit (via fail()), you must specify check=False:

  >>> from ci_exec import Executable
  >>> git = Executable("/usr/bin/git")
  >>> proc = git("not-a-command", check=False)
  $ /usr/bin/git not-a-command
  git: 'not-a-command' is not a git command. See 'git --help'.
  >>> proc.returncode
  1
  >>> git("not-a-command")
  $ /usr/bin/git not-a-command
  git: 'not-a-command' is not a git command. See 'git --help'.
  [X] Command '('/usr/bin/git', 'not-a-command')' returned non-zero exit
      status 1.
  

  The final git("not-a-command") exited the shell (this is what is meant by “failing by default”).

Returns

The result of calling subprocess.run() as outlined above.

Note

Unless you are are calling with check=False, you generally don’t need to store the return type.

Return type

subprocess.CompletedProcess

mkdir_p(path, mode=511, parents=True, exist_ok=True)

Permissive wrapper around pathlib.Path.mkdir().

The intention is to behave like mkdir -p, meaning the only real difference is that parents and exist_ok default to True for this method (rather than False for pathlib).

Parameters

• path (pathlib.Path or str) – The directory path to make.

• mode (int) – Access mask for directory permissions. See pathlib.Path.mkdir().

• parents (bool) – Whether or not parent directories may be created. Default: True.

• exist_ok (bool) –

  Whether or not the command should be considered successful if the specified path already exists. Default: True.

  Note

  If the path exists and is a directory with exist_ok=True, the command will succeed. If the path exists and is a file, even with exist_ok=True the command will fail().

rm_rf(path, ignore_errors=False, onerror=None)

Permissive wrapper around shutil.rmtree() bypassing FileNotFoundError and NotADirectoryError.

This function simply checks if path exists first before calling shutil.rmtree(). If the path does not exist, nothing is done. If the path exists but is a file, pathlib.Path.unlink() is called instead.

Essentially, this function tries to behave like rm -rf, but in the event that removal is not possible (e.g., due to insufficient permissions), the function will still fail().

Parameters

• path (pathlib.Path or str) – The directory path to delete (including all children).

• ignore_errors (bool) – Whether or not errors should be ignored. Default: False, to ensure that permission errors are still caught.

• onerror – See shutil.rmtree() for more information on the callback.

which(cmd, *, mode=1, path=None, **kwargs)

Restrictive wrapper around shutil.which() that will fail() if not found.

The primary difference is that when cmd is not found, shutil.which() will return None whereas this function will fail(). If you need to conditionally check for a command, do not use this function, use shutil.which() instead.

Parameters

• cmd (str) – The name of the command to search for. E.g., "cmake".

• mode (int) – The flag permission mask. Default: (os.F_OK | os.X_OK), see: os.F_OK, os.X_OK, shutil.which().

• path (str or None) – Default: None. See shutil.which().

• **kwargs –

  Included as a convenience bypass, forwards directly to Executable constructor. Suppose a non-logging Executable is desired. One option:

  git = which("git")
  git.log_calls = False
  

  Or alternatively:

  git = which("git", log_calls=False)
  

  This is in recognition that for continuous integration users will likely have many different preferences. Users can provide their own which to always use this default, or say, change the logging color:

  from ci_exec import which as ci_which
  from ci_exec import Colors, Styles
  
  def which(cmd: str):
      return ci_which(cmd, log_color=Colors.Magenta, log_style=Styles.Regular)
  

Returns

An executable created with the full path to the found cmd.

Return type

Executable

Tests

Tests for the ci_exec.core module.

test_fail(capsys, why, exit_code, no_prefix)

Validate fail() exits as expected.

test_executable_construction_failures()

Validate that non-absolute and non-(executable)file constructions will raise.

test_executable_relative()

Validate Executable accepts relative paths.

test_executable_logging(capsys)

Validate Executable runs and logs as expected.

test_executable_failures(capsys)

Validate failing executables error as expected.

test_mkdir_p(capsys)

Validate that mkdir_p() creates directories as expected.

test_rm_rf(capsys)

Validate rm_rf() deletes files / directories as expected.

test_which(capsys)

Validate that which() finds or does not find executables.

ci_exec.colorize

Various utilities for colorizing terminal output.

Ansi()	Wrapper class for defining the escape character and clear sequence.
Colors()	The core ANSI color codes.
Styles()	A non-exhaustive list of ANSI style formats.
colorize(message, *, color[, style])	Return message colorized with specified style.
dump_predefined_color_styles()	Dump all predefined Colors in every Styles to the console.
log_stage(stage, *[, fill_char, pad, l_pad, …])	Print a terminal width block with stage message in the middle.

class Ansi

Wrapper class for defining the escape character and clear sequence.

Escape = '\x1b['

The opening escape sequence to use before inserting color / style.

Clear = '\x1b[0m'

Convenience definition used to clear ANSI formatting.

class Colors

The core ANSI color codes.

Black = '30'

The black ANSI color.

Red = '31'

The red ANSI color.

Green = '32'

The green ANSI color.

Yellow = '33'

The yellow ANSI color.

Blue = '34'

The blue ANSI color.

Magenta = '35'

The magenta ANSI color.

Cyan = '36'

The cyan ANSI color.

White = '37'

The white ANSI color.

classmethod all_colors()

Return a tuple of all string colors available (used in tests).

class Styles

A non-exhaustive list of ANSI style formats.

The styles included here are reliable across many terminals, more exotic styles such as ‘Blinking’ are not included as they often are not supported.

Regular = ''

The regular ANSI format.

Bold = '1'

The bold ANSI format.

Dim = '2'

The dim ANSI format.

Underline = '4'

The underline ANSI format.

Inverted = '7'

The inverted ANSI format.

BoldUnderline = '1;4'

Bold and underlined ANSI format.

BoldInverted = '1;7'

Bold and inverted ANSI format.

BoldUnderlineInverted = '1;4;7'

Bold, underlined, and inverted ANSI format.

DimUnderline = '2;4'

Dim and underlined ANSI format.

DimInverted = '2;7'

Dim and inverted ANSI format.

DimUnderlineInverted = '2;4;7'

Dim, underlined, and inverted ANSI format.

classmethod all_styles()

Return a tuple of all style strings available (used in tests).

colorize(message, *, color, style='')

Return message colorized with specified style.

Warning

For both the color and style parameters, these are not supposed to have the m after. For example, a color="32m" is invalid, it should just be "32". Similarly, a style="1m" is invalid, it should just be "1".

Parameters

• message (str) – The message to insert an Ansi.Escape sequence with the specified color before, and Ansi.Clear sequence after.

• color (str) – A string describing the ANSI color code to use, e.g., Colors.Red.

• style (str) – The ANSI style to use. Default: Styles.Regular. Note that any number of ANSI style specifiers may be used, but it is assumed that the user has already formed the semicolon delineated list. For multiple ANSI specifiers, see for example Styles.BoldUnderline. Semicolons should be on the interior separating each style.

Returns

The original message with the specified color escape sequence.

Return type

str

dump_predefined_color_styles()

Dump all predefined Colors in every Styles to the console.

log_stage(stage, *, fill_char='=', pad=' ', l_pad=None, r_pad=None, color='32', style='1', width=None, **kwargs)

Print a terminal width block with stage message in the middle.

Similar to the output of tox, a bar of === {stage} === will be printed, adjusted to the width of the terminal. For example:

>>> log_stage("CMake.Configure")
======================== CMake.Configure ========================

By default, this will be printed using ANSI bold green to make it stick out. If the terminal size cannot be obtained, a width of 80 is assumed. Specify width if fixed width is desired.

Note

If the length of the stage parameter is too long (cannot pad with at least one fill_char and the specified padding both sides), the message with any coloring is printed as is. Prefer shorter stage messages when possible.

Parameters

• stage (str) – The description of the build stage to print to the console. This is the only required argument.

• fill_char (str) –

  A length 1 string to use as the fill character. Default: "=".

  Warning

  No checks on the input are performed, but any non-length-1 string will produce unattractive results.

• pad (str) –

  A padding to insert both before and after stage. Default: " ". This value can be any length, but may not be None. If no padding is desired, use the empty string "". Some examples:

  >>> log_stage("CMake.Configure")
  ============================= CMake.Configure ==============================
  >>> log_stage("CMake.Configure", fill_char="_", pad="")
  ______________________________CMake.Configure_______________________________
  

  See also: l_pad and r_pad if asymmetrical patterns are desired.

• l_pad (str or None) – A padding to insert before the stage (on the left). Default: None (implies use value from pad parameter). See examples in r_pad below.

• r_pad (str or None) –

  A padding to insert after the stage (on the right). Default: None (implies use value from pad parameter). Some examples:

  >>> log_stage("CMake.Configure", fill_char="-", l_pad="+ ", r_pad=" +")
  ----------------------------+ CMake.Configure +-----------------------------
  # Without specifying r_pad, pad is used (default: " ")
  >>> log_stage("CMake.Configure", fill_char="-", l_pad="+ ")
  -----------------------------+ CMake.Configure -----------------------------
  

• color (str or None) – The ANSI color code to use with colorize(). If no coloring is desired, call this function with color=None to disable.

• style (str) – The ANSI style specification to use with colorize(). If no coloring is desired, leave this parameter as is and specify color=None.

• width (int) –

  If specified, the terminal size will be ignored and a message formatted to this positive valued parameter will be used instead. If the value is less than the length of the stage message, this parameter is ignored.

  Note

  The specified width here does not necessarily equal the length of the string printed. The ANSI escape sequences added / trailing newline character will make the printed string longer than width, but the perceived width printed to the terminal will be correct.

  That is, if logging to a file, you may also desire to set color=None to remove the ANSI escape sequences / achieve the actual desired width.

• **kwargs – If provided, **kwargs is forwarded to the print(). E.g., to specify file=some_log_file_object or file=sys.stderr rather than printing to sys.stdout.

Tests

Tests for the ci_exec.colorize module.

test_all_colors()

Validate Colors.all_colors returns all available colors.

test_all_styles()

Validate Styles.all_styles returns all available styles.

test_colorize(color, style)

Test colorize() colors as expected for each platform.

test_dump_predefined_color_styles(capsys)

Validate dump_predefined_color_styles() dumps all.

test_log_stage(capsys, stage, fill_char_, pad_, l_pad_, r_pad_, color_, style_, width_)

Test log_stage() prints the expected messages.

ci_exec.patch

Various utilities for patching files.

filter_file(path, pattern, repl[, count, …])	Filter the contents of a file.
unified_diff(from_path, to_path[, n, …])	Return the unified_diff between two files.

filter_file(path, pattern, repl, count=0, flags=0, backup_extension='.orig', line_based=False, demand_different=True, encoding=None)

Filter the contents of a file.

1. Backup path to {path} + {backup_extension}. Typically, this would mean copying e.g., file.txt to file.txt.orig.

2. Perform filtering using re.sub().

3. If demand_different=True (default), verify that replacements were actually made. If not, fail().

The only required arguments are path, pattern, and repl. If any errors occur, including invalid input, this function will fail().

Parameters

• path (pathlib.Path or str) – The file that needs to be filtered.

• pattern (str) – The pattern to replace. Pass-through parameter to re.sub().

• repl (Callable[[Match], str] or str) – The replacement to be made. Pass-through parameter to re.sub().

• count (int) – The number of replacements to make (default 0 means replace all). Pass-through parameter to re.sub().

• flags (int) – Any flags such as re.IGNORECASE or re.MULTILINE (default 0 means no special flags). Pass-through parameter to re.sub().

• backup_extension (str) – The name to tack onto the back of path to make a backup with. Must be a non-empty string. Default: ".orig".

• line_based (bool) – Whether or not replacements should be made on the entirety of the file, or on a per-line basis. Default: False, do re.sub() on the entire contents. Setting line_based=True can make for simpler or more restrictive regular expressions depending on the replacement needed.

• demand_different (bool) – Whether or not this function should fail() if no changes were actually made. Default: True, fail() if no filtering was performed.

• encoding (str or None) – The encoding to open files with. Default: None implies default. Pass-through parameter to open().

Returns

The path to the backup file that was created with the original contents.

Return type

pathlib.Path

unified_diff(from_path, to_path, n=3, lineterm='\n', encoding=None, no_pygments=False)

Return the unified_diff between two files.

Any errors, such as not being able to read a file, will fail() the application abruptly.

Parameters

• from_path (pathlib.Path or str) – The file to diff from (the “original” file).

• to_path (pathlib.Path or str) – The file to diff to (the “changed” file).

• n (int) – Number of context lines. Default: 3. Pass-through parameter to difflib.unified_diff().

• lineterm (str) – Default: "\n". Pass-through parameter to difflib.unified_diff().

• encoding (str or None) – The encoding to open files with. Default: None implies default. Pass-through parameter to open().

• no_pygments (bool) –

  Whether or not an attempt to colorize the output using Pygments using the console formatter. If Pygments is not installed, no errors will ensue.

  Default: False, always try and make pretty output. Set to True if you need to enforce that the returned string does not have colors.

Returns

A string ready to be printed to the console.

Return type

str

Tests

Tests for the ci_exec.patch module.

test_filter_file(capsys)

Validate that filter_file() patches / errors as expected.

test_unified_diff(capsys)

Validate that unified_diff() diffs / errors as expected.

ci_exec.parsers

argparse derivatives tailored to specific build systems.

If you would like to contribute a generic parser for a build system, the pull request shall add the following:

1. Add file ci_exec/parsers/{build_system}_parser.py with the implementation.

2. Add file tests/parsers/{build_system}_parser.py with the tests.

3. Add a top-level import to ci_exec/parsers/__init__.py (keep alphabetical).

4. Update the top-level imports for parsers in ci_exec/__init__.py.

5. Add docs/source/api/parsers/{build_system}.rst and update the toctree directive in docs/source/api/parsers.rst (keep alphabetical).

When contributing a custom parser, please do your best to make it as generic as possible to support as wide an audience as is reasonably possible. If “arbitrary” decisions need to be made, please document them clearly and allow them to be overriden.

Lastly, by contributing a build system parser, you agree to being CC’d via a GitHub @ mention for any issues that may arise with the parser. In other words, please help me maintain the new parser being added 🙂

ci_exec.parsers.cmake_parser

Module for CMake focused argument parser CMakeParser.

class CMakeParser(*, add_extra_args=True, shared_or_static_required=False, **kwargs)

A CMake focused argument parser.

The goal of this parser is to fold some of the more common flags used by many CMake projects and parse them into cmake_configure_args and cmake_build_args automatically. Expected workflow is:

from ci_exec import CMakeParser, cd, which

parser = CMakeParser(description="Mylib CI Builder")
# ... add any extra arguments specific to project ...
# parser.add_argument("--foo", ...)

# Parse arguments.  cmake_{configure,build}_args are parsed for you, and are a
# list of strings (possibly empty).
args = parser.parse_args()
configure_args = args.cmake_configure_args
build_args = args.cmake_build_args

# Use the configure / build args created for you by CMakeParser.
cmake = which("cmake")
with cd("build", create=True):
    cmake("..", *configure_args)
    cmake("--build", ".", *build_args)

This is helpful for situations where how users are expected to interact with CMake changes depending on the generator chosen. A reputable example being the build type for single-config vs multi-config generators:

• Single-config: configure with -DCMAKE_BUILD_TYPE=<build_type>

• Multi-config: build with --config <build_type>

The command line arguments added here should be appropriate for most projects that obey typical CMake practices, but is as customizable as possible. Command-line argument values such as the help string or default value can be changed with set_argument(). Arguments added by this parser that are not desired can be removed using remove().

Note

The documentation makes a distinction between registered and unregistered arguments.

Registered

A flag that this parser adds in the constructor, and will later use to populate cmake_configure_args and cmake_build_args.

Unregistered

Any arguments that the user adds. This parser does not keep track of them (parser.add_argument("--foo", ...) in example above).

The registered command-line arguments added are as follows:

-G (args.generator) – default: Ninja

The CMake generator to use. Pass-through configure argument cmake -G. Generator choices are validated, however:

1. Extra Generators are not supported.

2. The now deprecated Visual Studio XX YYYY Win64 format with Win64 is parsed as an error. Users should take advantage of -A.

-A (args.architecture) – default: None

The CMake architecture to build. Pass-through configure argument cmake -A. Not validated, invalid arguments (e.g., -A provided when generator does not support it) will result in the cmake configure step failing.

-T (args.toolset) – default: None

The CMake toolset to use. Pass-through configure argument cmake -T. Not validated, invalid arguments (e.g., -T provided when generator does not support it) will result in the cmake configure step failing.

--shared (args.shared) – default: False

Add -DBUILD_SHARED_LIBS=ON to configure arguments? Conflicts with --static flag.

See also: shared_or_static_required constructor parameter.

--static (args.static) – default: False

Add -DBUILD_SHARED_LIBS=OFF to configure arguments? Conflicts with --shared flag.

See also: shared_or_static_required constructor parameter.

--cc (args.cc) – default: platform dependent

The C compiler to use. If the generator requested is a single-config generator, then the -DCMAKE_C_COMPILER={args.cc} configure argument. Multi-config generators such as Visual Studio or Xcode will ignore this flag.

Default: $CC environment variable if set, otherwise:

Platform	Default
Windows	cl.exe
Darwin	clang
Other	gcc

--cxx (args.cxx) – default: platform dependent

The C++ compiler to use. Like --cc, adds -DCMAKE_CXX_COMPILER={args.cxx} configure argument for single-config generators.

Default: $CXX environment variable if set, otherwise:

Platform	Default
Windows	cl.exe
Darwin	clang++
Other	g++

--build-type (args.build_type) – default: Release

For single-config generators, this will result in a configure argument of -DCMAKE_BUILD_TYPE={args.build_type}. For multi-config generators, results in ["--config", "{args.build_type}"] in the cmake_build_args.

Choices: Release, Debug, RelWithDebInfo, and MinSizeRel.

[extra_args] (args.extra_args) – default: []

By default, a positional argument with nargs="*" (meaning 0 or more) will be added. These are parsed as anything after the -- sequence, and will be added directly to cmake_configure_args. This supports users doing something like:

$ python .ci/build.py --shared
# args.extra_args = []

$ python .ci/build.py --shared -- -Werror=dev -DMYLIB_DEV=ON
# args.extra_args = ["-Werror=dev", "-DMYLIB_DEV=ON"]

$ python .ci/build.py --shared -- -DMYLIB_DEV=OFF
# args.extra_args = ["-DMYLIB_DEV=OFF"]

Note

This positional argument can be disabled two ways:

# Option 1: at construction.
parser = CMakeParser(add_extra_args=False)

# Option 2: set attribute to False *BEFORE* calling parse_args()
parser = CMakeParser()
parser.add_extra_args = False
args = parser.parse_args()

Since this is a positional argument consuming nargs="*", it must be added last in order for users to add their own positional arguments. The way this is implemented is by having parse_args() actually add the argument, which means:

1. It must be disabled before parser.parse_args() is called to prevent.

2. Unlike other arguments, it’s attributes such as default value of [] or help string cannot be changed.

Parameters

• add_extra_args (bool) – Default: True, support [extra_args] CMake configure arguments at the end of the command-line, after the -- sequence (see above).

• shared_or_static_required (bool) –

  Default: False. The --shared and --static flags are added using add_mutually_exclusive_group(), this is a pass-through parameter:

  shared_or_static = self.add_mutually_exclusive_group(
      required=shared_or_static_required
  )
  

  • When False, if neither --shared nor --static are supplied, then args.cmake_configure_args will not contain any -DBUILD_SHARED_LIBS=[val] entries.

  • When True, one of --shared or --static must be provided, meaning that args.cmake_configure_args will always contain either -DBUILD_SHARED_LIBS=ON or -DBUILD_SHARED_LIBS=OFF.

  Typically CMake projects will option(BUILD_SHARED_LIBS "Build shared libraries?" OFF), meaning that if not specified --static is implied. This is because the default behavior of add_library with no explicit SHARED|STATIC is STATIC. However, if a project defaults BUILD_SHARED_LIBS to ON, requiring --shared or --static be explicitly provided can help ensure that dependencies etc will all receive the same BUILD_SHARED_LIBS arguments.

• **kwargs – All other parameters are forwarded to argparse.ArgumentParser. Note that every parameter to the CMakeParser class must be specified as a keyword-only argument. Positional arguments are disabled.

add_extra_args

Whether or not CMake configure arguments after -- sequence will be added.

Type

bool

flag_map

Mapping of string flag keys (e.g., "-G", or "--build-type") to the actual Action of all registered arguments. Direct usage discouraged by users, use get_argument() or set_argument() instead.

Type

dict

dest_map

Mapping of string dest keys (e.g., "generator" or "build_type") to the actual Action of all registered arguments. Direct usage discouraged by users, use get_argument() or set_argument() instead.

Type

dict

makefile_generators = {'Borland Makefiles', 'MSYS Makefiles', 'MinGW Makefiles', 'NMake Makefiles', 'NMake Makefiles JOM', 'Unix Makefiles', 'Watcom WMake'}

The Makefile Generators.

ninja_generator = {'Ninja'}

The Ninja Generator.

ninja_multi_generator = {'Ninja Multi-Config'}

The Ninja Multi-Config Generator.

visual_studio_generators = {'Visual Studio 10 2010', 'Visual Studio 11 2012', 'Visual Studio 12 2013', 'Visual Studio 14 2015', 'Visual Studio 15 2017', 'Visual Studio 16 2019', 'Visual Studio 17 2022', 'Visual Studio 9 2008'}

The Visual Studio Generators.

other_generators = {'Green Hills MULTI', 'Xcode'}

The Other Generators.

classmethod is_multi_config_generator(generator)

Whether or not string generator is a multi-config generator.

classmethod is_single_config_generator(generator)

Whether or not string generator is a single-config generator.

add_argument(*args, **kwargs)

Add an argument to the parser.

Parameters

• *args – Positional arguments to pass directly to add_argument().

• **kwargs – Keyword arguments to pass directly to add_argument().

Returns

The return value of add_argument() (return value often not needed).

Return type

argparse.Action

Raises

• ValueError – If cmake_configure_args or cmake_build_args are in the positional *args. These are reserved attribute names that get populated after parsing the arguments.

• ValueError – If add_extra_args is True, then extra_args is also reserved and a value error will be raised if it is found in the positional *args.

get_argument(arg)

Get the Action instance for the specified argument.

Parameters

arg (str) – The command-line flag (e.g., "-G", "--shared") or the dest (e.g., "generator", "shared") to look for.

Returns

The argument action instance (created from add_argument()). If arg does not describe a command-line flag or dest, None is returned.

Return type

argparse.Action or None

parse_args(args=None, namespace=None)

Parse the command-line arguments.

Typically, no arguments are needed:

parser = CMakeParser()
# ... add your own arguments ...
args = parser.parse_args()  # uses sys.argv

Parameters

• args – See parse_args().

• namespace – See parse_args().

Returns

The parsed command-line arguments in a wrapper struct. Will also have cmake_configure_args and cmake_build_args (both will be lists of strings) attributes populated.

Return type

argparse.Namespace

remove(*args)

Remove any registered argument(s).

This method may be used to remove any arguments not desired. Only arguments that have been created by instantiating a CMakeParser can be removed.

Example:

parser = CMakeParser()
parser.remove("--shared", "--static")  # Remove by flags, or
parser.remove("shared", "static")      # remove by dest.

Parameters

*args (str) – Arguments to remove, listed by either flag or dest names. See CMakeParser docs for all flags / dest names added.

Raises

• ValueError – If "-G" or "generator" in args. The generator argument may not be removed.

• ValueError – If "extra_args" in args. This is to be prevented from being added, see parse_args().

• ValueError – If any arguments requested to be removed have not been found. This should only happen if (a) there was a typo, or (b) a user tries to remove an argument that was not registered.

set_argument(arg, **attrs)

Set attributes for arg argument.

Example:

parser = CMakeParser()

# Change default generator from Ninja to Unix Makefiles.
parser.set_argument("generator", default="Unix Makefiles")

# Change default build type from Release to Debug, only allow Release and
# Debug builds (only as demonstration...not useful in practice).
parser.set_argument("build_type", choices={"Release", "Debug"},
                    default="Debug")

Parameters

• arg (str) – May either be the command-line flag (e.g., "--shared" or "-G"), or the dest of the argument (e.g., "shared" or "generator").

• **attrs –

  The attributes to set. Only the following attributes are allowed to be changed via this method:

  • default: value returned if not specified on command-line.

  • choices: the list of valid values to validate against.

  • required: whether user must specify.

  • help: the help string for the argument.

  • metavar: how the argument is displayed in usage.

  Note

  Other values such as dest or nargs are disallowed from being changed as doing so will break all functionality this class provides.

Raises

• ValueError – If the argument described by arg cannot be found.

• ValueError – The keys of **attrs are not supported to be changed. E.g., dest may not be changed.

• ValueError – arg in {"-G", "generator"} and "choices" in attrs. The generator choices may not be changed (detection of single vs multi config generators will not be reliable).

Tests

Tests for the ci_exec.parsers.cmake_parser module.

default_cc_cxx()

Return the default (cc, cxx) for the current platform.

test_cmake_parser_is_x_config_generator()

Validate is_single_config_generator() and is_multi_config_generator().

test_cmake_parser_defaults()

Validate the CMakeParser defaults are as expected.

test_cmake_parser_add_argument_failues()

Validate add_argument() fails with expected names.

test_cmake_parser_get_argument()

Validate get_argument() finds both flag and dest names.

test_cmake_parser_remove()

Validate remove() can remove registered arguments (except for generator).

test_cmake_parser_set_argument()

Validate set_argument() can set supported attributes.

test_cmake_parser_extra_args()

Validate add_extra_args works as described.

test_cmake_parser_shared_or_static(capsys)

Validate --shared and --static CMakeParser options.

test_cmake_parser_parse_args_cmake_configure_args()

Validate parse_args works as expected.

test_cmake_parser_single_vs_multi_configure_build_args()

Validate that single vs multi config generators affect configure / build args.

ci_exec.parsers.utils

Helper routines for any custom ci_exec.parsers.

In a nested module to avoid any circular import problems.

env_or_platform_default(*, env, windows, darwin, other)

Return either the environment variable or the specified platform default.

Convenience routine to check os.getenv() for env variable, and if not found check platform.system() and return the default. Example:

env_or_platform_default(env="CC", windows="cl.exe", darwin="clang", other="gcc")

Only used to avoid writing the same conditional structure repeatedly.

Parameters

• env (str) – The environment variable to check for first. If it is set in the environment, it will be returned.

• windows (str) – Returned if env not set, and platform.system() returns "Windows".

• darwin (str) – Returned if env not set and platform.system() returns "Darwin".

• other (str) – Returned if env not set and platform.system() is neither "Windows" nor "Darwin".

Tests

Tests for the ci_exec.parsers.utils module.

test_env_or_platform_default()

Validate env_or_platform_default() returns expected values for each platform.

ci_exec.provider

Mechanisms to detect a given CI provider.

Provider()

Check if code is executing on a continuous integration (CI) service.

provider(func)

Mark a function as a CI provider.

Not intended for use outside of the Provider class.

Parameters

func – The function to decorate.

Returns

A static method that has an attribute register_provider=True.

Return type

staticmethod()

class ProviderMeta(name, bases, attrs)

Metaclass for Provider.

This metaclass populates Provider._all_provider_functions by coordinating with the provider() decorator.

Not intended to be used as a metaclass for any other classes.

class Provider

Check if code is executing on a continuous integration (CI) service.

Every now and then it is useful to know

1. If you are running on any CI service, or

2. If you are running on a specific CI service.

The static methods in this class provide a way of checking for pre-defined (by the CI service provider) environment variables:

from ci_exec import Provider, which

def build():
    # ... run cmake etc ...
    ninja = which("ninja")
    if Provider.is_travis():
        # Ninja uses too much memory during link phase.  See:
        # "My build script is killed without any error"
        # https://docs.travis-ci.com/user/common-build-problems/
        ninja("-j", "2", "install")
    else:
        ninja("install")

Available Providers:

is_ci()	Whether or not the code is executing on a CI service.
is_appveyor()	Whether or not the code is executing on AppVeyor.
is_azure_pipelines()	Whether or not the code is executing on Azure Pipelines.
is_circle_ci()	Whether or not the code is executing on CircleCI.
is_github_actions()	Whether or not the code is executing on GitHub Actions.
is_jenkins()	Whether or not the code is executing on Jenkins.
is_travis()	Whether or not the code is executing on Travis.

Adding a New Provider:

Pull requests are welcome. Alternatively, simply raise an issue with a link to the provider’s main homepage as well as a link to the documentation certifying the environment variables we can rely on.

1. Add a new is_{new_provider} method to this class, decorated with @provider. Keep these alphabetically sorted (except for is_ci, which should always be first).

2. Document any environment variable(s) involved in a table, including hyperlinks to the provider’s main homepage as well as documentation describing the environment variables in question.

3. Add to the _specific_providers list of environment variables in the tests/provider.py file (near provider_sum()).

4. Add a “pseudo-test” in tests/provider.py in the appropriate location.

_all_provider_functions

Not intended for external usage. The list of all known (implemented) CI provider functions in this class. For example, it will contain Provider.is_appveyor(), …, Provider.is_travis(), etc. This is a class attribute, the Provider class is not intended to be instantiated.

Type

list

static is_ci()

Whether or not the code is executing on a CI service.

Environment variables considered:

Environment Variable	Environment Value
CI	true (case insensitive)
CONTINUOUS_INTEGRATION	true (case insensitive)

If neither of these are true, this function will query every provider directly. For example, it will end up checking if any([Provider.is_appveyor(), ..., Provider.is_travis(), ...]).

static is_appveyor()

Whether or not the code is executing on AppVeyor.

Environment variables considered:

Environment Variable	Environment Value
APPVEYOR	true (case insensitive)

static is_azure_pipelines()

Whether or not the code is executing on Azure Pipelines.

Environment variables considered:

Environment Variable	Environment Value
AZURE_HTTP_USER_AGENT	Existence checked, value ignored.
AGENT_NAME	Existence checked, value ignored.
BUILD_REASON	Existence checked, value ignored.

Note

All three must be set for this function to return True.

static is_circle_ci()

Whether or not the code is executing on CircleCI.

Environment variables considered:

Environment Variable	Environment Value
CIRCLECI	true (case insensitive)

static is_github_actions()

Whether or not the code is executing on GitHub Actions.

Environment variables considered:

Environment Variable	Environment Value
GITHUB_ACTIONS	true (case insensitive)

static is_jenkins()

Whether or not the code is executing on Jenkins.

Environment variables considered:

Environment Variable	Environment Value
JENKINS_URL	Existence checked, value ignored.
BUILD_NUMBER	Existence checked, value ignored.

Note

Both must be set for this function to return True.

static is_travis()

Whether or not the code is executing on Travis.

Environment variables considered:

Environment Variable	Environment Value
TRAVIS	true (case insensitive)

Tests

Tests for the ci_exec.provider module.

provider_sum()

Return number of Provider’s that return True.

test_provider_is_ci()

Validate Provider.is_ci() reports as expected.

test_provider_is_appveyor()

Validate Provider.is_appveyor() reports as expected.

test_provider_is_azure_pipelines()

Validate Provider.is_azure_pipelines() reports as expected.

test_provider_is_circle_ci()

Validate Provider.is_circle_ci() reports as expected.

test_provider_is_github_actions()

Validate Provider.is_github_actions reports as expected.

test_provider_is_jenkins()

Validate Provider.is_jenkins() reports as expected.

test_provider_is_travis()

Validate Provider.is_travis() reports as expected.

ci_exec.utils

Assorted utility functions.

This module aims to house any utility functions that may facilitate easier consumption of the ci_exec package.

cd(dest, *[, create])	Context manager / decorator that can be used to change directories.
merge_kwargs(defaults, kwargs)	Merge defaults into kwargs and return kwargs.
set_env(**kwargs)	Context manager / decorator that can be used to set environment variables.
unset_env(*args)	Context manager / decorator that can be used to unset environment variables.

class cd(dest, *, create=False)

Context manager / decorator that can be used to change directories.

This context manager will change directories to dest, and after its scope expires (outside of the with statement, or after the decorated function) it will change directories back to the original current working directory.

As a context manager:

from ci_exec import cd, which

if __name__ == "__main__":
    # Get the build tools setup.
    cmake = which("cmake")
    ninja = which("ninja")

    # Suppose current directory here is "/source"
    with cd("build", create=True):
        # Current directory is now "/source/build"
        cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release")
        ninja()

    # Any code out-dented (not under the `with`): current directory is "/source"

As a decorator:

from ci_exec import cd, which

@cd("build", create=True)
def build():
    # Inside the function: current directory is "/source/build"
    cmake = which("cmake")
    ninja = which("ninja")
    cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release")
    ninja()

if __name__ == "__main__":
    # Suppose current directory here is "/source"
    build()  # Function executes in "/source/build"
    # After the function current directory is "/source"

Parameters

• dest (pathlib.Path or str) – The destination to change directories to.

• create (bool) – Whether or not the dest is allowed to be created. Default: False, the dest must exist already (will fail() if it does not). If True, mkdir_p() will be called with dest.

merge_kwargs(defaults, kwargs)

Merge defaults into kwargs and return kwargs.

Intended usage is for setting defaults to **kwargs when the caller did not provide a given argument, but making sure not to overwrite the caller’s explicit argument when specified.

For example:

>>> merge_kwargs({"a": 1, "b": 2}, {})
{'a': 1, 'b': 2}
>>> merge_kwargs({"a": 1, "b": 2}, {"a": 3})
{'a': 3, 'b': 2}

Entries in the defaults parameter only get included of not present in the kwargs argument. This is to facilitate something like this:

from ci_exec import merge_kwargs

# The function we want to customize the defaults for.
def func(alpha=1, beta=2):
    return alpha + beta

# Example: default to alpha=2, leave beta alone.
def custom(**kwargs):
    return func(**merge_kwargs({"alpha": 2}, kwargs))

# custom()                == 4
# custom(alpha=0)         == 2
# custom(beta=0)          == 2
# custom(alpha=0, beta=0) == 0

Parameters

• defaults (dict) – The dictionary of defaults to add to kwargs if not present.

• kwargs (dict) – The dictionary to merge defaults into.

Returns

The kwargs dictionary, possibly with values from defaults injected.

Return type

dict

class set_env(**kwargs)

Context manager / decorator that can be used to set environment variables.

Usage example:

from ci_exec import set_env

@set_env(CC="clang", CXX="clang++")
def build_clang():
    # CC="clang" and CXX="clang++" inside function.

# ... or ...

with set_env(CC="clang", CXX="clang++"):
    # CC="clang" and CXX="clang++" in `with` block

Prior environment variable state will be recorded and later restored when a decorated function / with block’s scope ends.

1. An environment variable was already set. Its value is saved before overwriting, and then later restored:

   # Example: CC=gcc was already set.
   with set_env(CC="clang"):
       # Inside block: CC="clang"
   # Out-dented: CC=gcc again.
   

2. An environment variable was not already set. Its value is unset again:

   # Example: CC was _not_ set in environment.
   with set_env(CC="clang"):
       # Inside block: CC="clang"
   # Out-dented: CC _not_ set in environment.
   

Note

See note in unset_env for more information on removing environment variables.

Parameters

**kwargs – Keyword argument parameter pack. Keys are the environment variable to set, and values are the desired value of the environment variable. All keys and all values must be strings.

Raises

ValueError – If no arguments are provided (len(kwargs) == 0), or if any keys / values are not a str.

class unset_env(*args)

Context manager / decorator that can be used to unset environment variables.

Usage example:

from ci_exec import unset_env

@unset_env("CC", "CXX")
def build():
    # Neither CC nor CXX are set in the environment during this function call.

# ... or ...

with unset_env("CC", "CXX"):
    # Neither CC nor CXX are set in the environment inside this block.

Prior environment variable state will be recorded and later restored when a decorated function / with block’s scope ends. So if an environment variable was already set, its value is saved before deletion, and then later restored:

# Example: CC=gcc was already set.
with unset_env("CC"):
    # Inside block: CC not set in environment.
# Out-dented: CC=gcc again.

Note

Removing the environment variable is done via del os.environ[env_var]. This may or may not affect child processes in the manner you expect, depending on whether your platform supports os.unsetenv(). See the end of the description of os.environ for more information.

Parameters

*args – Argument parameter pack. Each argument is an environment variable to unset. Each argument must be a string. If a specified argument is not currently set in the environment, it will effectively be skipped.

Raises

ValueError – If no arguments are provided (len(args) == 0), or if any arguments are not a str.

Tests

Tests for the ci_exec.utils module.

test_cd(capsys)

Validate cd behaves as expected.

test_merge_kwargs()

Validate merge_kwargs() merges as expected.

test_set_env()

Validate set_env sets environment variables.

test_unset_env()

Validate unset_env unsets environment variables.

Demos

More demos, particularly related to building C++, will be added when possible.

custom_log_stage

Simple demo for how to modify the ci_exec defaults to suit user preferences. Go to demo ↱

custom_log_stage

Simple demo for how to modify the ci_exec defaults to suit user preferences.

Run this demo by cloning the repository:

$ git clone https://github.com/svenevs/ci_exec.git
$ cd ci_exec
$ python -m demos custom_log_stage

By default log_stage() will log in bold green, using "=" as a separator. This makes stages stick out / easy to spot during CI builds, but users may not prefer this style. Instead of manually calling with explicit arguments each time:

from ci_exec import Colors, Styles, log_stage
# ... some time later ...
log_stage("CMake.Configure", color=Colors.Cyan, style=Styles.Regular, fill_char="-")

it is preferable to just create your own wrapper function. If you want a "-" fill character in regular cyan each time, why force yourself to type that out every time? It is much cleaner to just define your own wrapper with your preferred defaults. The most simple wrapper you can create:

import ci_exec
from ci_exec import Colors, Styles

def log_stage(stage: str):
    ci_exec.log_stage(stage, color=Colors.Cyan, style=Styles.Regular, fill_char="-")

Chances are, this will satisfy 95% of use-cases. The code in this file demonstrates how you can enable keyword arguments on your wrapper function so that if you have a scenario where you want to override your custom log_stage function defaults for just one or two use cases you can.

You could of course just set ci_exec.log_stage.__kwdefaults__, but changing this can lead to some surprising behavior if you don’t know what you are doing. Additionally, other readers will have a harder time following what your build script is doing.

log_stage(stage, **kwargs)

Sample wrapper #1: provide custom behavior of log_stage.

log_sub_stage(sub_stage, **kwargs)

Sample wrapper #2: enable sub-stages to be printed (e.g., for big scripts).

bold_green(msg)

Sample wrapper #3: return msg in bold green text.

do_work(n, width=80)

Ignore this function, pretend this is the real work you need to do.

main()

Execute all build stages and log progress.

Demos Program Execution

Wrapper module for executing each respective location from the repository root.

Each individual demo is executable on its own. However, users may also run a demo from repository root doing python demos/ <demo_name>.

By default the programs are not run in “animated” mode. The --animated flag is what is used to record the videos hosted on each individual demo page, which utilizes clear, PAUSE and a delay in calls to type_message(). See mock_shell() for more information.

clear()	Clear the console screen.
pause([amount])	Pause by amount using time.sleep().
type_message(message, *, delay)	Flush message to sys.stdout, sleep by delay after character.
mock_shell(program, *, cwd, delay, animated)	Run a “shell” program from the specified working directory.
run_demo(program, cwd, animated)	Run the specified demo program.

CI_EXEC_DEMOS_COVERAGE = False

Whether or not this is a coverage run of the demo files.

Warning

This should not be set unless invoking from tox. See notes in [testenv:docs] section of tox.ini at repository root.

windows_cmd_builtin(builtin)

Return a function that runs the specified builtin.

Note

There is a reason this is not in the main library. To deal with shell builtins requires a significant amount of extra work for little to no benefit. The demos just need "cls" to clear and "type" to cat.

The return is a function that can support:

*args

Any command line arguments to provide to the builtin.

**kwargs

Any keyword arguments to provide to subprocess.run(). This function will add check=True and shell=True unless these keys are already explicitly specified.

Parameters

builtin (str) – Any of the CMD builtins, such as "type" or "cls". No checking is performed!

clear()

Clear the console screen. Uses cls on Windows, and clear otherwise.

pause(amount=3.0)

Pause by amount using time.sleep().

This function exists so that it can be used for PAUSE statements in mock_shell().

Parameters

amount (float) – The amount of time to time.sleep() for. Default: 3.0 seconds.

type_message(message, *, delay)

Flush message to sys.stdout, sleep by delay after character.

Parameters

• message (str) – What to type. A trailing newline "\n" will be written at the end.

• delay (float) – The positive amount to time.sleep() after each character in message is written. Suggested value for simulating typing to the console: 0.05. Use 0.0 to avoid delays.

mock_shell(program, *, cwd, delay, animated)

Run a “shell” program from the specified working directory.

• Lines starting with # are “comment” lines, they will be printed to the screen using type_message().

• Lines starting with "$ " are a command to execute.

  • Commands executed will be printed to the screen using type_message().

There is also a notion of “animated” mode. When animated=True, the following special behavior is enabled:

• $ clear: calls clear(). In non-animated mode, clear is skipped.

• PAUSE or PAUSE=X.Y: call pause(), X.Y should be parseable as a float e.g., PAUSE=7.0 would pause for 7 seconds. In non-animated mode this is skipped.

• Calls to type_message() will have a delay=0.05. In non-animated mode the delay=0.0.

These scripts are not intended to be robust. Features are implemented as needed… this is not intended to be a real shell programming language! See demos/__main__.py for example program’s that can execute.

Parameters

• program (str) – The “shell” program to execute.

• cwd (str) – The directory to execute program from.

• delay (float) – Pass-through parameter for type_message().

• animated (bool) – Whether or not this is an “animated” shell, meaning commands such as clear or PAUSE should be executed.

run_demo(program, cwd, animated)

Run the specified demo program.

When animated=True, clear() the screen and sleep for 2 seconds to allow recording to begin. The delay parameter passed-through to type_message() will be set to 0.05. In non-animated mode the screen will not be cleared, and the delay will be 0.0.

Parameters

• program (str) – The “shell” program to execute with mock_shell().

• cwd (str) – Pass-through parameter to mock_shell().

• animated (bool) – Pass-through parameter to mock_shell().

main()

Create the argument parser and run the specified demo.

Important Usage Notes

Namespace Pollution

The ci_exec package namespace is polluted intentionally. Always import from the polluted namespace, never import from the module originally defining something:

from ci_exec import which       # Yes
from ci_exec.core import which  # NO!

In practice it shouldn’t matter that much, but

1. Any functions moved to different modules will not be considered ABI breaking changes. So if ci_exec.core.which moved to a different ci_exec.other_module, this would be done in a patch release (ci_exec uses semantic versioning).

2. Anything that is not exposed at the top-level is more than likely something that should not be used.

Beware of Commands Expecting Input

The Executable.__call__() method internally invokes subprocess.run(), which does some fancy manipulations of stdout, stderr, and stdin. In particular, the subprocess by default will allow communication on stdin for commands that are expecting input. In the context of CI, this is problematic, and users need to be aware that the subprocess will indeed pause and solicit user input. Consider a scenario where say filter_file() was used to make some significant changes on the CI provider. If you ran:

from ci_exec import which

git = which("git")
git("diff")

The build may fail from a timeout rather than just displaying the changes, because for a large enough change to report git will want to use your $PAGER to let you scroll through it. Most commands that allow input also provide a command-line flag to disable this behavior and in this scenario, it would be:

git("--no-pager", "diff")

In situations where you know that input is required, subprocess.run() enables this via the input parameter. Consider the following toy script that is expecting two pieces of user input at different stages:

name = input("What is your name? ")
print(f"Hi {name}, nice to meet you.")

age = input("How old are you? ")
print(f"Wow!  {age} is such a great age :)")

Then you could call it using ci_exec like this:

import sys
from ci_exec import Executable

if __name__ == "__main__":
    python = Executable(sys.executable)
    python("./multi_input.py", input=b"Bilbo\n111")

where

1. The input parameter needs a bytes object (the b prefix in b"...").

2. You use \n to send a newline. From the docs: For stdin, line ending characters '\n' in the input will be converted to the default line separator os.linesep. So "Bilbo\n" will end up being the answer to the first name = input(...), and "111" will be sent to the age = input(...).

Build Logs out of Order (Azure Pipelines)

If the results of call logging appear out of order, then your CI provider is affected by this. This is a known problem on Azure Pipelines, it seems unlikely to affect many other providers due to the nature of how the Azure Pipelines build logs are served. Example “script”:

cmake("..")
ninja()

Produces:

-- The C compiler identification is GNU 8.3.1
-- The CXX compiler identification is GNU 8.3.1
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
...
[12/74] Building C object ...joystick.c.o
$ /usr/bin/cmake ..
$ /usr/bin/ninja

The log is out of order, all cmake / ninja output appeared before the call logging ($ /usr/bin/cmake .. and $ /usr/bin/ninja). There are two possible solutions:

1. Invoke your build script using python -u: python -u ./.ci/build.py

2. Set the environment variable PYTHONUNBUFFERED=true.

Changelog

v0.1.2

• Drop python 3.5 support (#27): python 3.6 or later required.

• Add Visual Studio 17 2022 and Ninja Multi-Config to CMakeParser (#29).

v0.1.1

• Added CMakeParser (#16).

v0.1.0.post0

• Fix support for early 3.5.x by conditionally importing NoReturn and TYPE_CHECKING (#12).

v0.1.0

Initial release. Broken for python 3.5.x where typing module does not house NoReturn and TYPE_CHECKING. Definitely broken on 3.5.2 and earlier.