ci_exec

About

A wrapper package designed for running continuous integration (CI) build steps using Python 3.6+.

Managing cross platform build scripts for CI can become tedious at times when you need to e.g., maintain two nearly identical scripts install_deps.sh and install_deps.bat due to incompatible syntaxes. ci_exec enables a single file to manage this using Python.

The ci_exec package provides a set of wrappers / utility functions designed specifically for running build steps on CI providers. It is

Logging by Default

Commands executed, including their full command-line arguments, are logged. This includes any output on stdout / stderr from the commands. The logging resembles what set -x would give you in a shell script. For commands that will take a long time, as long as output is being produced, this will additionally prevent timeouts on the build.

Failing by Default

Any command that does not succeed will fail the entire build. An attempt to exit with the same exit code as the command that failed will be performed. Meaning the CI provider will correctly report a failed build.

Convenient

ci_exec affords users the ability to write shell-like scripts that will work on any platform that Python can run on. A simple example:

from ci_exec import cd, which

cmake = which("cmake")
ninja = which("ninja")
with cd("build", create=True):
    cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release")
    ninja("-j", "2", "test")

Installation

ci_exec is available on PyPI. It can be installed using your python package manager of choice:

$ pip install ci-exec

Note

The PyPI package has a -: ci-exec, not ci_exec.

There is also a setup.py here, so you can also install it from source:

$ pip install git+https://github.com/svenevs/ci_exec.git@master

License

This software is licensed under the Apache 2.0 license.

Intended Audience

Note

ci_exec can be used for anything related to writing build steps, but it was originally written to manage C++ projects. The documentation will often have examples using cmake and ninja, users do not need to understand what these commands are for.

ci_exec utilizes some “advanced” features of Python that pertain to how the library itself is consumed. It may not be appropriate for users who do not know any Python at all. The main features a user should be aware of:

  • *args and **kwargs are used liberally. ci_exec mostly consists of wrapper classes / functions around the python standard library, and in most cases *args and **kwargs are the “pass-through” parameters.

  • Keyword-only arguments. Many library signatures look something like:

    def foo(a: str, *, b: int = 2):
    pass

foo("hi")       # OK: b = 2
foo("hi", 3)    # ERROR: b is keyword only
foo("hi", b=3)  # OK: b = 3

    Anything after the *, must be named explicitly.

  • Operator overloading, particularly what __call__ means and how it works. A quick overview:

    from ci_exec import Executable

# Typically: prefer ci_exec.which instead, which returns a ci_exec.Executable.
cmake = Executable("/usr/bin/cmake")

# cmake.__call__ invoked with args = [".."], kwargs = {}
cmake("..")

# cmake.__call__ invoked with args = [".."], kwargs = {"check": False}
cmake("..", check=False)

None of these features are altogether that special, but it must be stated clearly and plainly: this library is designed for users who already know Python.

Note

C++ users are encouraged to look at conan as an alternative. ci_exec has zero intention of becoming a package manager, and was written to help manage projects that are not well suited to conan for various reasons.

Full Documentation

Quick reference:

Ansi()

Wrapper class for defining the escape character and clear sequence.

Colors()

The core ANSI color codes.

Styles()

A non-exhaustive list of ANSI style formats.

colorize(message, *, color)

Return message colorized with specified style.

log_stage(stage, *[, …])

Print a terminal width block with stage message in the middle.

Executable(exe_path, *[, …])

Represent a reusable executable.

fail(why, *[, exit_code, no_prefix])

Write a failure message to sys.stderr and exit.

mkdir_p(path[, mode, parents, …])

Permissive wrapper around pathlib.Path.mkdir().

rm_rf(path[, ignore_errors, …])

Permissive wrapper around shutil.rmtree() bypassing FileNotFoundError and NotADirectoryError.

which(cmd, *[, mode, path])

Restrictive wrapper around shutil.which() that will fail() if not found.

CMakeParser(*)

A CMake focused argument parser.

filter_file(path, pattern, repl)

Filter the contents of a file.

unified_diff(from_path, to_path)

Return the unified_diff between two files.

Provider()

Check if code is executing on a continuous integration (CI) service.

cd(dest, *[, create])

Context manager / decorator that can be used to change directories.

merge_kwargs(defaults, kwargs)

Merge defaults into kwargs and return kwargs.

set_env(**kwargs)

Context manager / decorator that can be used to set environment variables.

unset_env(*args)

Context manager / decorator that can be used to unset environment variables.