ci_exec¶
About¶
A wrapper package designed for running continuous integration (CI) build steps using Python 3.6+.
Managing cross platform build scripts for CI can become tedious at times when you need
to e.g., maintain two nearly identical scripts install_deps.sh
and
install_deps.bat
due to incompatible syntaxes. ci_exec
enables a single file
to manage this using Python.
The ci_exec
package provides a set of wrappers / utility functions designed
specifically for running build steps on CI providers. It is
- Logging by Default
Commands executed, including their full command-line arguments, are logged. This includes any output on
stdout
/stderr
from the commands. The logging resembles whatset -x
would give you in a shell script. For commands that will take a long time, as long as output is being produced, this will additionally prevent timeouts on the build.- Failing by Default
Any command that does not succeed will fail the entire build. An attempt to exit with the same exit code as the command that failed will be performed. Meaning the CI provider will correctly report a failed build.
- Convenient
ci_exec
affords users the ability to write shell-like scripts that will work on any platform that Python can run on. A simple example:from ci_exec import cd, which cmake = which("cmake") ninja = which("ninja") with cd("build", create=True): cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release") ninja("-j", "2", "test")
Installation¶
ci_exec
is available on PyPI. It can be
installed using your python package manager of choice:
$ pip install ci-exec
Note
The PyPI package has a -
: ci-exec
, not ci_exec
.
There is also a setup.py
here, so you can also install it from source:
$ pip install git+https://github.com/svenevs/ci_exec.git@master
License¶
This software is licensed under the Apache 2.0 license.
Intended Audience¶
Note
ci_exec
can be used for anything related to writing build steps, but it was
originally written to manage C++ projects. The documentation will often have
examples using cmake
and ninja
, users do not need to understand what these
commands are for.
ci_exec
utilizes some “advanced” features of Python that pertain to how the library
itself is consumed. It may not be appropriate for users who do not know any Python at
all. The main features a user should be aware of:
*args
and**kwargs
are used liberally.ci_exec
mostly consists of wrapper classes / functions around the python standard library, and in most cases*args
and**kwargs
are the “pass-through” parameters.Keyword-only arguments. Many library signatures look something like:
def foo(a: str, *, b: int = 2): pass foo("hi") # OK: b = 2 foo("hi", 3) # ERROR: b is keyword only foo("hi", b=3) # OK: b = 3
Anything after the
*,
must be named explicitly.Operator overloading, particularly what
__call__
means and how it works. A quick overview:from ci_exec import Executable # Typically: prefer ci_exec.which instead, which returns a ci_exec.Executable. cmake = Executable("/usr/bin/cmake") # cmake.__call__ invoked with args = [".."], kwargs = {} cmake("..") # cmake.__call__ invoked with args = [".."], kwargs = {"check": False} cmake("..", check=False)
None of these features are altogether that special, but it must be stated clearly and plainly: this library is designed for users who already know Python.
Note
C++ users are encouraged to look at conan as an alternative.
ci_exec
has zero intention of becoming a package manager, and was written to
help manage projects that are not well suited to conan for various reasons.
Full Documentation¶
Quick reference:
|
Wrapper class for defining the escape character and clear sequence. |
|
The core ANSI color codes. |
|
A non-exhaustive list of ANSI style formats. |
|
Return |
|
Print a terminal width block with |
|
Represent a reusable executable. |
|
Write a failure message to |
|
Permissive wrapper around |
|
Permissive wrapper around |
|
Restrictive wrapper around |
|
A CMake focused argument parser. |
|
Filter the contents of a file. |
|
Return the |
|
Check if code is executing on a continuous integration (CI) service. |
|
Context manager / decorator that can be used to change directories. |
|
Merge |
|
Context manager / decorator that can be used to set environment variables. |
|
Context manager / decorator that can be used to unset environment variables. |
ci_exec¶
The ci_exec
package top-level namespace.
Quick Reference:
|
Wrapper class for defining the escape character and clear sequence. |
|
The core ANSI color codes. |
|
A non-exhaustive list of ANSI style formats. |
|
Return |
|
Print a terminal width block with |
|
Represent a reusable executable. |
|
Write a failure message to |
|
Permissive wrapper around |
|
Permissive wrapper around |
|
Restrictive wrapper around |
|
A CMake focused argument parser. |
|
Filter the contents of a file. |
|
Return the |
|
Check if code is executing on a continuous integration (CI) service. |
|
Context manager / decorator that can be used to change directories. |
|
Merge |
|
Context manager / decorator that can be used to set environment variables. |
|
Context manager / decorator that can be used to unset environment variables. |
ci_exec.core¶
The core functionality of the ci_exec
package.
fail
(why, *[, exit_code, no_prefix])Write a failure message to
sys.stderr
and exit.
Executable
(exe_path, *[, log_calls, …])Represent a reusable executable.
mkdir_p
(path[, mode, parents, exist_ok])Permissive wrapper around
pathlib.Path.mkdir()
.
rm_rf
(path[, ignore_errors, onerror])Permissive wrapper around
shutil.rmtree()
bypassingFileNotFoundError
andNotADirectoryError
.
which
(cmd, *[, mode, path])Restrictive wrapper around
shutil.which()
that willfail()
if not found.
- fail(why, *, exit_code=1, no_prefix=False)[source]¶
Write a failure message to
sys.stderr
and exit.
- class Executable(exe_path, *, log_calls=True, log_prefix='$ ', log_color='36', log_style='1')[source]¶
Represent a reusable executable.
Each executable is:
Failing by default: unless called with
check=False
, any execution that fails (has a non-zero exit code) will result in a call tofail()
, terminating the entire application.Logging by default: every call executed will print what will be run in color and then dump the output of the command. In the event of failure, this makes finding the last call issued much simpler.
Consider the following simple script:
from ci_exec import Executable git = Executable("/usr/bin/git") git("remote") # Oops! --pretty not --petty ;) git("log", "-1", "--petty=%B") git("status") # will not execute (previous failed)
When we execute
python simple.py
and check the exit code withecho $?
:> python simple.py $ /usr/bin/git remote origin $ /usr/bin/git log -1 --petty=%B fatal: unrecognized argument: --petty=%B [X] Command '('/usr/bin/git', 'log', '-1', "--petty=%B")' returned non-zero exit status 128. > echo $? 128
See
__call__()
for more information.Tip
Hard-coded paths in these examples were for demonstrative purposes. In practice this should not be done, use
which()
instead.- log_calls¶
Whether or not every invocation of
__call__()
should print what will execute before executing it. Default:True
.- Type
- log_prefix¶
The prefix to use when printing a given invocation of
__call__()
. Default:"$ "
to simulate aconsole
lexer. Set to the empty string""
to have no prefix.- Type
- log_color¶
The
color
code to use when callingcolorize()
to display the next invocation of__call__()
. Set toNone
to disable colorizing each log of__call__()
. Default:Colors.Cyan
.- Type
- log_style¶
The
style
code to use when callingcolorize()
to display the next invocation of__call__()
. If no colors are desired, setlog_color
toNone
. Default:Styles.Bold
.- Type
- Raises
ValueError – If
exe_path
is not a file, or if it is not executable.
- PATH_EXTENSIONS = {}¶
The set of valid file extensions that can be executed on Windows.
On *nix systems this will be the empty set, and takes no meaning. On Windows, it is controlled by the user. These are stored in lower case, and comparisons should be lower case for consistency. The typical default value on Windows would be:
PATH_EXTENSIONS = {".com", ".exe", ".bat", ".cmd"}
- __call__(*args, **kwargs)[source]¶
Run
exe_path
with the specified command-line*args
.The usage of the parameters is best summarized in code:
popen_args = (self.exe_path, *args) # ... some potential logging ... return subprocess.run(popen_args, **kwargs)
For example, sending multiple arguments to the executable is as easy as:
cmake = Executable("/usr/bin/cmake") cmake("..", "-G", "Ninja", "-DBUILD_SHARED_LIBS=ON")
and any overrides to
subprocess.run()
you wish to include should be done with**kwargs
, which are forwarded directly.Warning
Any exceptions generated result in a call to
fail()
, which will terminate the application.- Parameters
*args – The positional arguments will be forwarded along with
exe_path
tosubprocess.run()
.**kwargs –
The key-value arguments are all forwarded to
subprocess.run()
. Ifcheck
is not provided, this is an implicitcheck=True
. That is, if you do not want the application to exit (viafail()
), you must specifycheck=False
:>>> from ci_exec import Executable >>> git = Executable("/usr/bin/git") >>> proc = git("not-a-command", check=False) $ /usr/bin/git not-a-command git: 'not-a-command' is not a git command. See 'git --help'. >>> proc.returncode 1 >>> git("not-a-command") $ /usr/bin/git not-a-command git: 'not-a-command' is not a git command. See 'git --help'. [X] Command '('/usr/bin/git', 'not-a-command')' returned non-zero exit status 1.
The final
git("not-a-command")
exited the shell (this is what is meant by “failing by default”).
- Returns
The result of calling
subprocess.run()
as outlined above.Note
Unless you are are calling with
check=False
, you generally don’t need to store the return type.- Return type
- mkdir_p(path, mode=511, parents=True, exist_ok=True)[source]¶
Permissive wrapper around
pathlib.Path.mkdir()
.The intention is to behave like
mkdir -p
, meaning the only real difference is thatparents
andexist_ok
default toTrue
for this method (rather thanFalse
forpathlib
).- Parameters
path (pathlib.Path or str) – The directory path to make.
mode (int) – Access mask for directory permissions. See
pathlib.Path.mkdir()
.parents (bool) – Whether or not parent directories may be created. Default:
True
.exist_ok (bool) –
Whether or not the command should be considered successful if the specified path already exists. Default:
True
.Note
If the path exists and is a directory with
exist_ok=True
, the command will succeed. If the path exists and is a file, even withexist_ok=True
the command willfail()
.
- rm_rf(path, ignore_errors=False, onerror=None)[source]¶
Permissive wrapper around
shutil.rmtree()
bypassingFileNotFoundError
andNotADirectoryError
.This function simply checks if
path
exists first before callingshutil.rmtree()
. If thepath
does not exist, nothing is done. If the path exists but is a file,pathlib.Path.unlink()
is called instead.Essentially, this function tries to behave like
rm -rf
, but in the event that removal is not possible (e.g., due to insufficient permissions), the function will stillfail()
.- Parameters
path (pathlib.Path or str) – The directory path to delete (including all children).
ignore_errors (bool) – Whether or not errors should be ignored. Default:
False
, to ensure that permission errors are still caught.onerror – See
shutil.rmtree()
for more information on the callback.
- which(cmd, *, mode=1, path=None, **kwargs)[source]¶
Restrictive wrapper around
shutil.which()
that willfail()
if not found.The primary difference is that when
cmd
is not found,shutil.which()
will returnNone
whereas this function willfail()
. If you need to conditionally check for a command, do not use this function, useshutil.which()
instead.- Parameters
cmd (str) – The name of the command to search for. E.g.,
"cmake"
.mode (int) – The flag permission mask. Default:
(os.F_OK | os.X_OK)
, see:os.F_OK
,os.X_OK
,shutil.which()
.path (str or None) – Default:
None
. Seeshutil.which()
.**kwargs –
Included as a convenience bypass, forwards directly to
Executable
constructor. Suppose a non-loggingExecutable
is desired. One option:git = which("git") git.log_calls = False
Or alternatively:
git = which("git", log_calls=False)
This is in recognition that for continuous integration users will likely have many different preferences. Users can provide their own
which
to always use this default, or say, change the logging color:from ci_exec import which as ci_which from ci_exec import Colors, Styles def which(cmd: str): return ci_which(cmd, log_color=Colors.Magenta, log_style=Styles.Regular)
- Returns
An executable created with the full path to the found
cmd
.- Return type
Tests¶
Tests for the ci_exec.core
module.
- test_executable_construction_failures()[source]¶
Validate that non-absolute and non-(executable)file constructions will raise.
- test_executable_relative()[source]¶
Validate
Executable
accepts relative paths.
- test_executable_logging(capsys)[source]¶
Validate
Executable
runs and logs as expected.
ci_exec.colorize¶
Various utilities for colorizing terminal output.
Ansi
()Wrapper class for defining the escape character and clear sequence.
Colors
()The core ANSI color codes.
Styles
()A non-exhaustive list of ANSI style formats.
colorize
(message, *, color[, style])Return
message
colorized with specified style.
log_stage
(stage, *[, fill_char, pad, l_pad, …])Print a terminal width block with
stage
message in the middle.
- class Ansi[source]¶
Wrapper class for defining the escape character and clear sequence.
- Escape = '\x1b['¶
The opening escape sequence to use before inserting color / style.
- Clear = '\x1b[0m'¶
Convenience definition used to clear ANSI formatting.
- class Colors[source]¶
The core ANSI color codes.
- Black = '30'¶
The black ANSI color.
- Red = '31'¶
The red ANSI color.
- Green = '32'¶
The green ANSI color.
- Yellow = '33'¶
The yellow ANSI color.
- Blue = '34'¶
The blue ANSI color.
- Magenta = '35'¶
The magenta ANSI color.
- Cyan = '36'¶
The cyan ANSI color.
- White = '37'¶
The white ANSI color.
- class Styles[source]¶
A non-exhaustive list of ANSI style formats.
The styles included here are reliable across many terminals, more exotic styles such as ‘Blinking’ are not included as they often are not supported.
- Regular = ''¶
The regular ANSI format.
- Bold = '1'¶
The bold ANSI format.
- Dim = '2'¶
The dim ANSI format.
- Underline = '4'¶
The underline ANSI format.
- Inverted = '7'¶
The inverted ANSI format.
- BoldUnderline = '1;4'¶
Bold and underlined ANSI format.
- BoldInverted = '1;7'¶
Bold and inverted ANSI format.
- BoldUnderlineInverted = '1;4;7'¶
Bold, underlined, and inverted ANSI format.
- DimUnderline = '2;4'¶
Dim and underlined ANSI format.
- DimInverted = '2;7'¶
Dim and inverted ANSI format.
- DimUnderlineInverted = '2;4;7'¶
Dim, underlined, and inverted ANSI format.
- colorize(message, *, color, style='')[source]¶
Return
message
colorized with specified style.Warning
For both the
color
andstyle
parameters, these are not supposed to have them
after. For example, acolor="32m"
is invalid, it should just be"32"
. Similarly, astyle="1m"
is invalid, it should just be"1"
.- Parameters
message (str) – The message to insert an
Ansi.Escape
sequence with the specified color before, andAnsi.Clear
sequence after.color (str) – A string describing the ANSI color code to use, e.g.,
Colors.Red
.style (str) – The ANSI style to use. Default:
Styles.Regular
. Note that any number of ANSI style specifiers may be used, but it is assumed that the user has already formed the semicolon delineated list. For multiple ANSI specifiers, see for exampleStyles.BoldUnderline
. Semicolons should be on the interior separating each style.
- Returns
The original message with the specified color escape sequence.
- Return type
- log_stage(stage, *, fill_char='=', pad=' ', l_pad=None, r_pad=None, color='32', style='1', width=None, **kwargs)[source]¶
Print a terminal width block with
stage
message in the middle.Similar to the output of
tox
, a bar of=== {stage} ===
will be printed, adjusted to the width of the terminal. For example:>>> log_stage("CMake.Configure") ======================== CMake.Configure ========================
By default, this will be printed using ANSI bold green to make it stick out. If the terminal size cannot be obtained, a width of
80
is assumed. Specifywidth
if fixed width is desired.Note
If the length of the
stage
parameter is too long (cannot pad with at least onefill_char
and the specified padding both sides), the message with any coloring is printed as is. Prefer shorter stage messages when possible.- Parameters
stage (str) – The description of the build stage to print to the console. This is the only required argument.
fill_char (str) –
A length 1 string to use as the fill character. Default:
"="
.Warning
No checks on the input are performed, but any non-length-1 string will produce unattractive results.
pad (str) –
A padding to insert both before and after
stage
. Default:" "
. This value can be any length, but may not beNone
. If no padding is desired, use the empty string""
. Some examples:>>> log_stage("CMake.Configure") ============================= CMake.Configure ============================== >>> log_stage("CMake.Configure", fill_char="_", pad="") ______________________________CMake.Configure_______________________________
See also:
l_pad
andr_pad
if asymmetrical patterns are desired.l_pad (str or None) – A padding to insert before the
stage
(on the left). Default:None
(implies use value frompad
parameter). See examples inr_pad
below.A padding to insert after the
stage
(on the right). Default:None
(implies use value frompad
parameter). Some examples:>>> log_stage("CMake.Configure", fill_char="-", l_pad="+ ", r_pad=" +") ----------------------------+ CMake.Configure +----------------------------- # Without specifying r_pad, pad is used (default: " ") >>> log_stage("CMake.Configure", fill_char="-", l_pad="+ ") -----------------------------+ CMake.Configure -----------------------------
color (str or None) – The ANSI color code to use with
colorize()
. If no coloring is desired, call this function withcolor=None
to disable.style (str) – The ANSI style specification to use with
colorize()
. If no coloring is desired, leave this parameter as is and specifycolor=None
.width (int) –
If specified, the terminal size will be ignored and a message formatted to this positive valued parameter will be used instead. If the value is less than the length of the
stage
message, this parameter is ignored.Note
The specified width here does not necessarily equal the length of the string printed. The ANSI escape sequences added / trailing newline character will make the printed string longer than
width
, but the perceived width printed to the terminal will be correct.That is, if logging to a file, you may also desire to set
color=None
to remove the ANSI escape sequences / achieve the actual desired width.**kwargs – If provided,
**kwargs
is forwarded to theprint()
. E.g., to specifyfile=some_log_file_object
orfile=sys.stderr
rather than printing tosys.stdout
.
Tests¶
Tests for the ci_exec.colorize
module.
- test_all_colors()[source]¶
Validate
Colors.all_colors
returns all available colors.
- test_all_styles()[source]¶
Validate
Styles.all_styles
returns all available styles.
- test_colorize(color, style)[source]¶
Test
colorize()
colors as expected for each platform.
- test_dump_predefined_color_styles(capsys)[source]¶
Validate
dump_predefined_color_styles()
dumps all.
- test_log_stage(capsys, stage, fill_char_, pad_, l_pad_, r_pad_, color_, style_, width_)[source]¶
Test
log_stage()
prints the expected messages.
ci_exec.patch¶
Various utilities for patching files.
filter_file
(path, pattern, repl[, count, …])Filter the contents of a file.
unified_diff
(from_path, to_path[, n, …])Return the
unified_diff
between two files.
- filter_file(path, pattern, repl, count=0, flags=0, backup_extension='.orig', line_based=False, demand_different=True, encoding=None)[source]¶
Filter the contents of a file.
Backup
path
to{path} + {backup_extension}
. Typically, this would mean copying e.g.,file.txt
tofile.txt.orig
.Perform filtering using
re.sub()
.If
demand_different=True
(default), verify that replacements were actually made. If not,fail()
.
The only required arguments are
path
,pattern
, andrepl
. If any errors occur, including invalid input, this function willfail()
.- Parameters
path (pathlib.Path or str) – The file that needs to be filtered.
pattern (str) – The pattern to replace. Pass-through parameter to
re.sub()
.repl (
Callable[[Match], str]
orstr
) – The replacement to be made. Pass-through parameter tore.sub()
.count (int) – The number of replacements to make (default
0
means replace all). Pass-through parameter tore.sub()
.flags (int) – Any flags such as
re.IGNORECASE
orre.MULTILINE
(default0
means no special flags). Pass-through parameter tore.sub()
.backup_extension (str) – The name to tack onto the back of
path
to make a backup with. Must be a non-empty string. Default:".orig"
.line_based (bool) – Whether or not replacements should be made on the entirety of the file, or on a per-line basis. Default:
False
, dore.sub()
on the entire contents. Settingline_based=True
can make for simpler or more restrictive regular expressions depending on the replacement needed.demand_different (bool) – Whether or not this function should
fail()
if no changes were actually made. Default:True
,fail()
if no filtering was performed.encoding (str or None) – The encoding to open files with. Default:
None
implies default. Pass-through parameter toopen()
.
- Returns
The path to the backup file that was created with the original contents.
- Return type
- unified_diff(from_path, to_path, n=3, lineterm='\n', encoding=None, no_pygments=False)[source]¶
Return the
unified_diff
between two files.Any errors, such as not being able to read a file, will
fail()
the application abruptly.- Parameters
from_path (pathlib.Path or str) – The file to diff from (the “original” file).
to_path (pathlib.Path or str) – The file to diff to (the “changed” file).
n (int) – Number of context lines. Default:
3
. Pass-through parameter todifflib.unified_diff()
.lineterm (str) – Default:
"\n"
. Pass-through parameter todifflib.unified_diff()
.encoding (str or None) – The encoding to open files with. Default:
None
implies default. Pass-through parameter toopen()
.no_pygments (bool) –
Whether or not an attempt to colorize the output using Pygments using the
console
formatter. If Pygments is not installed, no errors will ensue.Default:
False
, always try and make pretty output. Set toTrue
if you need to enforce that the returned string does not have colors.
- Returns
A string ready to be printed to the console.
- Return type
Tests¶
Tests for the ci_exec.patch
module.
- test_filter_file(capsys)[source]¶
Validate that
filter_file()
patches / errors as expected.
- test_unified_diff(capsys)[source]¶
Validate that
unified_diff()
diffs / errors as expected.
ci_exec.parsers¶
argparse
derivatives tailored to specific build systems.
If you would like to contribute a generic parser for a build system, the pull request shall add the following:
Add file
ci_exec/parsers/{build_system}_parser.py
with the implementation.Add file
tests/parsers/{build_system}_parser.py
with the tests.Add a top-level import to
ci_exec/parsers/__init__.py
(keep alphabetical).Update the top-level imports for
parsers
inci_exec/__init__.py
.Add
docs/source/api/parsers/{build_system}.rst
and update thetoctree
directive indocs/source/api/parsers.rst
(keep alphabetical).
When contributing a custom parser, please do your best to make it as generic as possible to support as wide an audience as is reasonably possible. If “arbitrary” decisions need to be made, please document them clearly and allow them to be overriden.
Lastly, by contributing a build system parser, you agree to being CC’d via a GitHub
@
mention for any issues that may arise with the parser. In other words, please
help me maintain the new parser being added 🙂
ci_exec.parsers.cmake_parser¶
Module for CMake focused argument parser CMakeParser
.
- class CMakeParser(*, add_extra_args=True, shared_or_static_required=False, **kwargs)[source]¶
A CMake focused argument parser.
The goal of this parser is to fold some of the more common flags used by many CMake projects and parse them into
cmake_configure_args
andcmake_build_args
automatically. Expected workflow is:from ci_exec import CMakeParser, cd, which parser = CMakeParser(description="Mylib CI Builder") # ... add any extra arguments specific to project ... # parser.add_argument("--foo", ...) # Parse arguments. cmake_{configure,build}_args are parsed for you, and are a # list of strings (possibly empty). args = parser.parse_args() configure_args = args.cmake_configure_args build_args = args.cmake_build_args # Use the configure / build args created for you by CMakeParser. cmake = which("cmake") with cd("build", create=True): cmake("..", *configure_args) cmake("--build", ".", *build_args)
This is helpful for situations where how users are expected to interact with CMake changes depending on the generator chosen. A reputable example being the build type for single-config vs multi-config generators:
Single-config: configure with
-DCMAKE_BUILD_TYPE=<build_type>
Multi-config: build with
--config <build_type>
The command line arguments added here should be appropriate for most projects that obey typical CMake practices, but is as customizable as possible. Command-line argument values such as the
help
string ordefault
value can be changed withset_argument()
. Arguments added by this parser that are not desired can be removed usingremove()
.Note
The documentation makes a distinction between registered and unregistered arguments.
- Registered
A flag that this parser adds in the constructor, and will later use to populate
cmake_configure_args
andcmake_build_args
.- Unregistered
Any arguments that the user adds. This parser does not keep track of them (
parser.add_argument("--foo", ...)
in example above).
The registered command-line arguments added are as follows:
-G
(args.generator
) – default:Ninja
The CMake generator to use. Pass-through configure argument
cmake -G
. Generator choices are validated, however:Extra Generators are not supported.
The now deprecated
Visual Studio XX YYYY Win64
format withWin64
is parsed as an error. Users should take advantage of-A
.
-A
(args.architecture
) – default:None
The CMake architecture to build. Pass-through configure argument
cmake -A
. Not validated, invalid arguments (e.g.,-A
provided when generator does not support it) will result in the cmake configure step failing.-T
(args.toolset
) – default:None
The CMake toolset to use. Pass-through configure argument
cmake -T
. Not validated, invalid arguments (e.g.,-T
provided when generator does not support it) will result in the cmake configure step failing.--shared
(args.shared
) – default:False
Add
-DBUILD_SHARED_LIBS=ON
to configure arguments? Conflicts with--static
flag.See also:
shared_or_static_required
constructor parameter.--static
(args.static
) – default:False
Add
-DBUILD_SHARED_LIBS=OFF
to configure arguments? Conflicts with--shared
flag.See also:
shared_or_static_required
constructor parameter.--cc
(args.cc
) – default: platform dependentThe C compiler to use. If the generator requested is a single-config generator, then the
-DCMAKE_C_COMPILER={args.cc}
configure argument. Multi-config generators such as Visual Studio or Xcode will ignore this flag.Default:
$CC
environment variable if set, otherwise:Platform
Default
Windows
cl.exe
Darwin
clang
Other
gcc
--cxx
(args.cxx
) – default: platform dependentThe C++ compiler to use. Like
--cc
, adds-DCMAKE_CXX_COMPILER={args.cxx}
configure argument for single-config generators.Default:
$CXX
environment variable if set, otherwise:Platform
Default
Windows
cl.exe
Darwin
clang++
Other
g++
--build-type
(args.build_type
) – default:Release
For single-config generators, this will result in a configure argument of
-DCMAKE_BUILD_TYPE={args.build_type}
. For multi-config generators, results in["--config", "{args.build_type}"]
in thecmake_build_args
.Choices:
Release
,Debug
,RelWithDebInfo
, andMinSizeRel
.[extra_args]
(args.extra_args
) – default:[]
By default, a positional argument with
nargs="*"
(meaning 0 or more) will be added. These are parsed as anything after the--
sequence, and will be added directly tocmake_configure_args
. This supports users doing something like:$ python .ci/build.py --shared # args.extra_args = [] $ python .ci/build.py --shared -- -Werror=dev -DMYLIB_DEV=ON # args.extra_args = ["-Werror=dev", "-DMYLIB_DEV=ON"] $ python .ci/build.py --shared -- -DMYLIB_DEV=OFF # args.extra_args = ["-DMYLIB_DEV=OFF"]
Note
This positional argument can be disabled two ways:
# Option 1: at construction. parser = CMakeParser(add_extra_args=False) # Option 2: set attribute to False *BEFORE* calling parse_args() parser = CMakeParser() parser.add_extra_args = False args = parser.parse_args()
Since this is a positional argument consuming
nargs="*"
, it must be added last in order for users to add their own positional arguments. The way this is implemented is by havingparse_args()
actually add the argument, which means:It must be disabled before
parser.parse_args()
is called to prevent.Unlike other arguments, it’s attributes such as default value of
[]
or help string cannot be changed.
- Parameters
add_extra_args (bool) – Default:
True
, support[extra_args]
CMake configure arguments at the end of the command-line, after the--
sequence (see above).shared_or_static_required (bool) –
Default:
False
. The--shared
and--static
flags are added usingadd_mutually_exclusive_group()
, this is a pass-through parameter:shared_or_static = self.add_mutually_exclusive_group( required=shared_or_static_required )
When
False
, if neither--shared
nor--static
are supplied, thenargs.cmake_configure_args
will not contain any-DBUILD_SHARED_LIBS=[val]
entries.When
True
, one of--shared
or--static
must be provided, meaning thatargs.cmake_configure_args
will always contain either-DBUILD_SHARED_LIBS=ON
or-DBUILD_SHARED_LIBS=OFF
.
Typically CMake projects will
option(BUILD_SHARED_LIBS "Build shared libraries?" OFF)
, meaning that if not specified--static
is implied. This is because the default behavior ofadd_library
with no explicitSHARED|STATIC
isSTATIC
. However, if a project defaultsBUILD_SHARED_LIBS
toON
, requiring--shared
or--static
be explicitly provided can help ensure that dependencies etc will all receive the sameBUILD_SHARED_LIBS
arguments.**kwargs – All other parameters are forwarded to
argparse.ArgumentParser
. Note that every parameter to theCMakeParser
class must be specified as a keyword-only argument. Positional arguments are disabled.
- flag_map¶
Mapping of string flag keys (e.g.,
"-G"
, or"--build-type"
) to the actualAction
of all registered arguments. Direct usage discouraged by users, useget_argument()
orset_argument()
instead.- Type
- dest_map¶
Mapping of string
dest
keys (e.g.,"generator"
or"build_type"
) to the actualAction
of all registered arguments. Direct usage discouraged by users, useget_argument()
orset_argument()
instead.- Type
- makefile_generators = {'Borland Makefiles', 'MSYS Makefiles', 'MinGW Makefiles', 'NMake Makefiles', 'NMake Makefiles JOM', 'Unix Makefiles', 'Watcom WMake'}¶
The Makefile Generators.
- ninja_generator = {'Ninja'}¶
The Ninja Generator.
- ninja_multi_generator = {'Ninja Multi-Config'}¶
- visual_studio_generators = {'Visual Studio 10 2010', 'Visual Studio 11 2012', 'Visual Studio 12 2013', 'Visual Studio 14 2015', 'Visual Studio 15 2017', 'Visual Studio 16 2019', 'Visual Studio 17 2022', 'Visual Studio 9 2008'}¶
- other_generators = {'Green Hills MULTI', 'Xcode'}¶
The Other Generators.
- classmethod is_multi_config_generator(generator)[source]¶
Whether or not string
generator
is a multi-config generator.
- classmethod is_single_config_generator(generator)[source]¶
Whether or not string
generator
is a single-config generator.
- add_argument(*args, **kwargs)[source]¶
Add an argument to the parser.
- Parameters
*args – Positional arguments to pass directly to
add_argument()
.**kwargs – Keyword arguments to pass directly to
add_argument()
.
- Returns
The return value of
add_argument()
(return value often not needed).- Return type
- Raises
ValueError – If
cmake_configure_args
orcmake_build_args
are in the positional*args
. These are reserved attribute names that get populated after parsing the arguments.ValueError – If
add_extra_args
isTrue
, thenextra_args
is also reserved and a value error will be raised if it is found in the positional*args
.
- get_argument(arg)[source]¶
Get the
Action
instance for the specified argument.- Parameters
arg (str) – The command-line flag (e.g.,
"-G"
,"--shared"
) or thedest
(e.g.,"generator"
,"shared"
) to look for.- Returns
The argument action instance (created from
add_argument()
). Ifarg
does not describe a command-line flag ordest
,None
is returned.- Return type
- parse_args(args=None, namespace=None)[source]¶
Parse the command-line arguments.
Typically, no arguments are needed:
parser = CMakeParser() # ... add your own arguments ... args = parser.parse_args() # uses sys.argv
- Parameters
args – See
parse_args()
.namespace – See
parse_args()
.
- Returns
The parsed command-line arguments in a wrapper struct. Will also have
cmake_configure_args
andcmake_build_args
(both will be lists of strings) attributes populated.- Return type
- remove(*args)[source]¶
Remove any registered argument(s).
This method may be used to remove any arguments not desired. Only arguments that have been created by instantiating a
CMakeParser
can be removed.Example:
parser = CMakeParser() parser.remove("--shared", "--static") # Remove by flags, or parser.remove("shared", "static") # remove by dest.
- Parameters
*args (str) – Arguments to remove, listed by either flag or dest names. See
CMakeParser
docs for all flags / dest names added.- Raises
ValueError – If
"-G"
or"generator"
inargs
. The generator argument may not be removed.ValueError – If
"extra_args"
inargs
. This is to be prevented from being added, seeparse_args()
.ValueError – If any arguments requested to be removed have not been found. This should only happen if (a) there was a typo, or (b) a user tries to remove an argument that was not registered.
- set_argument(arg, **attrs)[source]¶
Set attributes for
arg
argument.Example:
parser = CMakeParser() # Change default generator from Ninja to Unix Makefiles. parser.set_argument("generator", default="Unix Makefiles") # Change default build type from Release to Debug, only allow Release and # Debug builds (only as demonstration...not useful in practice). parser.set_argument("build_type", choices={"Release", "Debug"}, default="Debug")
- Parameters
arg (str) – May either be the command-line flag (e.g.,
"--shared"
or"-G"
), or thedest
of the argument (e.g.,"shared"
or"generator"
).**attrs –
The attributes to set. Only the following attributes are allowed to be changed via this method:
default: value returned if not specified on command-line.
choices: the list of valid values to validate against.
required: whether user must specify.
help: the help string for the argument.
metavar: how the argument is displayed in usage.
Note
Other values such as
dest
ornargs
are disallowed from being changed as doing so will break all functionality this class provides.
- Raises
ValueError – If the argument described by
arg
cannot be found.ValueError – The keys of
**attrs
are not supported to be changed. E.g.,dest
may not be changed.ValueError –
arg in {"-G", "generator"}
and"choices" in attrs
. The generator choices may not be changed (detection of single vs multi config generators will not be reliable).
Tests¶
Tests for the ci_exec.parsers.cmake_parser
module.
- test_cmake_parser_is_x_config_generator()[source]¶
Validate
is_single_config_generator()
andis_multi_config_generator()
.
- test_cmake_parser_defaults()[source]¶
Validate the
CMakeParser
defaults are as expected.
- test_cmake_parser_add_argument_failues()[source]¶
Validate
add_argument()
fails with expected names.
- test_cmake_parser_get_argument()[source]¶
Validate
get_argument()
finds both flag and dest names.
- test_cmake_parser_remove()[source]¶
Validate
remove()
can remove registered arguments (except for generator).
- test_cmake_parser_set_argument()[source]¶
Validate
set_argument()
can set supported attributes.
- test_cmake_parser_extra_args()[source]¶
Validate
add_extra_args
works as described.
Validate
--shared
and--static
CMakeParser
options.
- test_cmake_parser_parse_args_cmake_configure_args()[source]¶
Validate
parse_args
works as expected.
ci_exec.parsers.utils¶
Helper routines for any custom ci_exec.parsers
.
In a nested module to avoid any circular import problems.
- env_or_platform_default(*, env, windows, darwin, other)[source]¶
Return either the environment variable or the specified platform default.
Convenience routine to check
os.getenv()
forenv
variable, and if not found checkplatform.system()
and return the default. Example:env_or_platform_default(env="CC", windows="cl.exe", darwin="clang", other="gcc")
Only used to avoid writing the same conditional structure repeatedly.
- Parameters
env (str) – The environment variable to check for first. If it is set in the environment, it will be returned.
windows (str) – Returned if
env
not set, andplatform.system()
returns"Windows"
.darwin (str) – Returned if
env
not set andplatform.system()
returns"Darwin"
.other (str) – Returned if
env
not set andplatform.system()
is neither"Windows"
nor"Darwin"
.
Tests¶
Tests for the ci_exec.parsers.utils
module.
- test_env_or_platform_default()[source]¶
Validate
env_or_platform_default()
returns expected values for each platform.
ci_exec.provider¶
Mechanisms to detect a given CI provider.
Provider
()Check if code is executing on a continuous integration (CI) service.
- provider(func)[source]¶
Mark a function as a CI provider.
Not intended for use outside of the
Provider
class.- Parameters
func – The function to decorate.
- Returns
A static method that has an attribute
register_provider=True
.- Return type
- class ProviderMeta(name, bases, attrs)[source]¶
Metaclass for
Provider
.This metaclass populates
Provider._all_provider_functions
by coordinating with theprovider()
decorator.Not intended to be used as a metaclass for any other classes.
- class Provider[source]¶
Check if code is executing on a continuous integration (CI) service.
Every now and then it is useful to know
If you are running on any CI service, or
If you are running on a specific CI service.
The static methods in this class provide a way of checking for pre-defined (by the CI service provider) environment variables:
from ci_exec import Provider, which def build(): # ... run cmake etc ... ninja = which("ninja") if Provider.is_travis(): # Ninja uses too much memory during link phase. See: # "My build script is killed without any error" # https://docs.travis-ci.com/user/common-build-problems/ ninja("-j", "2", "install") else: ninja("install")
Available Providers:
is_ci
()Whether or not the code is executing on a CI service.
Whether or not the code is executing on AppVeyor.
Whether or not the code is executing on Azure Pipelines.
Whether or not the code is executing on CircleCI.
Whether or not the code is executing on GitHub Actions.
Whether or not the code is executing on Jenkins.
Whether or not the code is executing on Travis.
Adding a New Provider:
Pull requests are welcome. Alternatively, simply raise an issue with a link to the provider’s main homepage as well as a link to the documentation certifying the environment variables we can rely on.
Add a new
is_{new_provider}
method to this class, decorated with@provider
. Keep these alphabetically sorted (except foris_ci
, which should always be first).Document any environment variable(s) involved in a table, including hyperlinks to the provider’s main homepage as well as documentation describing the environment variables in question.
Add to the
_specific_providers
list of environment variables in thetests/provider.py
file (nearprovider_sum()
).Add a “pseudo-test” in
tests/provider.py
in the appropriate location.
- _all_provider_functions¶
Not intended for external usage. The list of all known (implemented) CI provider functions in this class. For example, it will contain
Provider.is_appveyor()
, …,Provider.is_travis()
, etc. This is a class attribute, theProvider
class is not intended to be instantiated.- Type
- static is_ci()[source]¶
Whether or not the code is executing on a CI service.
Environment variables considered:
Environment Variable
Environment Value
CI
true
(case insensitive)CONTINUOUS_INTEGRATION
true
(case insensitive)If neither of these are
true
, this function will query every provider directly. For example, it will end up checking ifany([Provider.is_appveyor(), ..., Provider.is_travis(), ...])
.
- static is_appveyor()[source]¶
Whether or not the code is executing on AppVeyor.
Environment variables considered:
Environment Variable
Environment Value
APPVEYOR
true
(case insensitive)
- static is_azure_pipelines()[source]¶
Whether or not the code is executing on Azure Pipelines.
Environment variables considered:
Environment Variable
Environment Value
AZURE_HTTP_USER_AGENT
Existence checked, value ignored.
AGENT_NAME
Existence checked, value ignored.
BUILD_REASON
Existence checked, value ignored.
Note
All three must be set for this function to return
True
.
- static is_circle_ci()[source]¶
Whether or not the code is executing on CircleCI.
Environment variables considered:
Environment Variable
Environment Value
CIRCLECI
true
(case insensitive)
- static is_github_actions()[source]¶
Whether or not the code is executing on GitHub Actions.
Environment variables considered:
Environment Variable
Environment Value
GITHUB_ACTIONS
true
(case insensitive)
- static is_jenkins()[source]¶
Whether or not the code is executing on Jenkins.
Environment variables considered:
Environment Variable
Environment Value
JENKINS_URL
Existence checked, value ignored.
BUILD_NUMBER
Existence checked, value ignored.
Note
Both must be set for this function to return
True
.
- static is_travis()[source]¶
Whether or not the code is executing on Travis.
Environment variables considered:
Environment Variable
Environment Value
TRAVIS
true
(case insensitive)
Tests¶
Tests for the ci_exec.provider
module.
- test_provider_is_ci()[source]¶
Validate
Provider.is_ci()
reports as expected.
- test_provider_is_appveyor()[source]¶
Validate
Provider.is_appveyor()
reports as expected.
- test_provider_is_azure_pipelines()[source]¶
Validate
Provider.is_azure_pipelines()
reports as expected.
- test_provider_is_circle_ci()[source]¶
Validate
Provider.is_circle_ci()
reports as expected.
- test_provider_is_github_actions()[source]¶
Validate
Provider.is_github_actions
reports as expected.
- test_provider_is_jenkins()[source]¶
Validate
Provider.is_jenkins()
reports as expected.
- test_provider_is_travis()[source]¶
Validate
Provider.is_travis()
reports as expected.
ci_exec.utils¶
Assorted utility functions.
This module aims to house any utility functions that may facilitate easier consumption
of the ci_exec
package.
cd
(dest, *[, create])Context manager / decorator that can be used to change directories.
merge_kwargs
(defaults, kwargs)Merge
defaults
intokwargs
and returnkwargs
.
set_env
(**kwargs)Context manager / decorator that can be used to set environment variables.
unset_env
(*args)Context manager / decorator that can be used to unset environment variables.
- class cd(dest, *, create=False)[source]¶
Context manager / decorator that can be used to change directories.
This context manager will change directories to
dest
, and after its scope expires (outside of thewith
statement, or after the decorated function) it will change directories back to the original current working directory.As a context manager:
from ci_exec import cd, which if __name__ == "__main__": # Get the build tools setup. cmake = which("cmake") ninja = which("ninja") # Suppose current directory here is "/source" with cd("build", create=True): # Current directory is now "/source/build" cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release") ninja() # Any code out-dented (not under the `with`): current directory is "/source"
As a decorator:
from ci_exec import cd, which @cd("build", create=True) def build(): # Inside the function: current directory is "/source/build" cmake = which("cmake") ninja = which("ninja") cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release") ninja() if __name__ == "__main__": # Suppose current directory here is "/source" build() # Function executes in "/source/build" # After the function current directory is "/source"
- Parameters
dest (pathlib.Path or str) – The destination to change directories to.
create (bool) – Whether or not the
dest
is allowed to be created. Default:False
, thedest
must exist already (willfail()
if it does not). IfTrue
,mkdir_p()
will be called withdest
.
- merge_kwargs(defaults, kwargs)[source]¶
Merge
defaults
intokwargs
and returnkwargs
.Intended usage is for setting defaults to
**kwargs
when the caller did not provide a given argument, but making sure not to overwrite the caller’s explicit argument when specified.For example:
>>> merge_kwargs({"a": 1, "b": 2}, {}) {'a': 1, 'b': 2} >>> merge_kwargs({"a": 1, "b": 2}, {"a": 3}) {'a': 3, 'b': 2}
Entries in the
defaults
parameter only get included of not present in thekwargs
argument. This is to facilitate something like this:from ci_exec import merge_kwargs # The function we want to customize the defaults for. def func(alpha=1, beta=2): return alpha + beta # Example: default to alpha=2, leave beta alone. def custom(**kwargs): return func(**merge_kwargs({"alpha": 2}, kwargs)) # custom() == 4 # custom(alpha=0) == 2 # custom(beta=0) == 2 # custom(alpha=0, beta=0) == 0
- class set_env(**kwargs)[source]¶
Context manager / decorator that can be used to set environment variables.
Usage example:
from ci_exec import set_env @set_env(CC="clang", CXX="clang++") def build_clang(): # CC="clang" and CXX="clang++" inside function. # ... or ... with set_env(CC="clang", CXX="clang++"): # CC="clang" and CXX="clang++" in `with` block
Prior environment variable state will be recorded and later restored when a decorated function /
with
block’s scope ends.An environment variable was already set. Its value is saved before overwriting, and then later restored:
# Example: CC=gcc was already set. with set_env(CC="clang"): # Inside block: CC="clang" # Out-dented: CC=gcc again.
An environment variable was not already set. Its value is unset again:
# Example: CC was _not_ set in environment. with set_env(CC="clang"): # Inside block: CC="clang" # Out-dented: CC _not_ set in environment.
Note
See note in unset_env for more information on removing environment variables.
- Parameters
**kwargs – Keyword argument parameter pack. Keys are the environment variable to set, and values are the desired value of the environment variable. All keys and all values must be strings.
- Raises
ValueError – If no arguments are provided (
len(kwargs) == 0
), or if any keys / values are not astr
.
- class unset_env(*args)[source]¶
Context manager / decorator that can be used to unset environment variables.
Usage example:
from ci_exec import unset_env @unset_env("CC", "CXX") def build(): # Neither CC nor CXX are set in the environment during this function call. # ... or ... with unset_env("CC", "CXX"): # Neither CC nor CXX are set in the environment inside this block.
Prior environment variable state will be recorded and later restored when a decorated function /
with
block’s scope ends. So if an environment variable was already set, its value is saved before deletion, and then later restored:# Example: CC=gcc was already set. with unset_env("CC"): # Inside block: CC not set in environment. # Out-dented: CC=gcc again.
Note
Removing the environment variable is done via
del os.environ[env_var]
. This may or may not affect child processes in the manner you expect, depending on whether your platform supportsos.unsetenv()
. See the end of the description ofos.environ
for more information.- Parameters
*args – Argument parameter pack. Each argument is an environment variable to unset. Each argument must be a string. If a specified argument is not currently set in the environment, it will effectively be skipped.
- Raises
ValueError – If no arguments are provided (
len(args) == 0
), or if any arguments are not astr
.
Tests¶
Tests for the ci_exec.utils
module.
- test_merge_kwargs()[source]¶
Validate
merge_kwargs()
merges as expected.
Demos¶
More demos, particularly related to building C++, will be added when possible.
- custom_log_stage
Simple demo for how to modify the
ci_exec
defaults to suit user preferences.Go to demo ↱
custom_log_stage¶
Simple demo for how to modify the ci_exec
defaults to suit user preferences.
Run this demo by cloning the repository:
$ git clone https://github.com/svenevs/ci_exec.git
$ cd ci_exec
$ python -m demos custom_log_stage
By default log_stage()
will log in bold green, using "="
as
a separator. This makes stages stick out / easy to spot during CI builds, but users may
not prefer this style. Instead of manually calling with explicit arguments each time:
from ci_exec import Colors, Styles, log_stage
# ... some time later ...
log_stage("CMake.Configure", color=Colors.Cyan, style=Styles.Regular, fill_char="-")
it is preferable to just create your own wrapper function. If you want a "-"
fill
character in regular cyan each time, why force yourself to type that out every time? It
is much cleaner to just define your own wrapper with your preferred defaults. The most
simple wrapper you can create:
import ci_exec
from ci_exec import Colors, Styles
def log_stage(stage: str):
ci_exec.log_stage(stage, color=Colors.Cyan, style=Styles.Regular, fill_char="-")
Chances are, this will satisfy 95% of use-cases. The code in this file demonstrates how
you can enable keyword arguments on your wrapper function so that if you have a scenario
where you want to override your custom log_stage
function defaults for just one or
two use cases you can.
You could of course just set ci_exec.log_stage.__kwdefaults__
, but changing this can
lead to some surprising behavior if you don’t know what you are doing. Additionally,
other readers will have a harder time following what your build script is doing.
Demos Program Execution¶
Wrapper module for executing each respective location from the repository root.
Each individual demo is executable on its own. However, users may also run a demo from
repository root doing python demos/ <demo_name>
.
By default the programs are not run in “animated” mode. The --animated
flag is what
is used to record the videos hosted on each individual demo page, which utilizes
clear
, PAUSE
and a delay in calls to type_message()
. See
mock_shell()
for more information.
clear
()Clear the console screen.
pause
([amount])Pause by
amount
usingtime.sleep()
.
type_message
(message, *, delay)Flush
message
tosys.stdout
, sleep bydelay
after character.
mock_shell
(program, *, cwd, delay, animated)Run a “shell” program from the specified working directory.
run_demo
(program, cwd, animated)Run the specified demo program.
- CI_EXEC_DEMOS_COVERAGE = False¶
Whether or not this is a coverage run of the demo files.
Warning
This should not be set unless invoking from
tox
. See notes in[testenv:docs]
section oftox.ini
at repository root.
- windows_cmd_builtin(builtin)[source]¶
Return a function that runs the specified
builtin
.Note
There is a reason this is not in the main library. To deal with shell builtins requires a significant amount of extra work for little to no benefit. The demos just need
"cls"
to clear and"type"
tocat
.The return is a function that can support:
*args
Any command line arguments to provide to the builtin.
**kwargs
Any keyword arguments to provide to
subprocess.run()
. This function will addcheck=True
andshell=True
unless these keys are already explicitly specified.
- Parameters
builtin (str) – Any of the CMD builtins, such as
"type"
or"cls"
. No checking is performed!
- pause(amount=3.0)[source]¶
Pause by
amount
usingtime.sleep()
.This function exists so that it can be used for
PAUSE
statements inmock_shell()
.- Parameters
amount (float) – The amount of time to
time.sleep()
for. Default:3.0
seconds.
- type_message(message, *, delay)[source]¶
Flush
message
tosys.stdout
, sleep bydelay
after character.- Parameters
message (str) – What to type. A trailing newline
"\n"
will be written at the end.delay (float) – The positive amount to
time.sleep()
after each character inmessage
is written. Suggested value for simulating typing to the console:0.05
. Use0.0
to avoid delays.
- mock_shell(program, *, cwd, delay, animated)[source]¶
Run a “shell” program from the specified working directory.
Lines starting with
#
are “comment” lines, they will be printed to the screen usingtype_message()
.Lines starting with
"$ "
are a command to execute.Commands executed will be printed to the screen using
type_message()
.
There is also a notion of “animated” mode. When
animated=True
, the following special behavior is enabled:$ clear
: callsclear()
. In non-animated mode,clear
is skipped.PAUSE
orPAUSE=X.Y
: callpause()
,X.Y
should be parseable as afloat
e.g.,PAUSE=7.0
would pause for 7 seconds. In non-animated mode this is skipped.Calls to
type_message()
will have adelay=0.05
. In non-animated mode thedelay=0.0
.
These scripts are not intended to be robust. Features are implemented as needed… this is not intended to be a real shell programming language! See
demos/__main__.py
for exampleprogram
’s that can execute.- Parameters
program (str) – The “shell” program to execute.
cwd (str) – The directory to execute
program
from.delay (float) – Pass-through parameter for
type_message()
.animated (bool) – Whether or not this is an “animated” shell, meaning commands such as
clear
orPAUSE
should be executed.
- run_demo(program, cwd, animated)[source]¶
Run the specified demo program.
When
animated=True
,clear()
the screen and sleep for 2 seconds to allow recording to begin. Thedelay
parameter passed-through totype_message()
will be set to0.05
. In non-animated mode the screen will not be cleared, and the delay will be0.0
.- Parameters
program (str) – The “shell” program to execute with
mock_shell()
.cwd (str) – Pass-through parameter to
mock_shell()
.animated (bool) – Pass-through parameter to
mock_shell()
.
Important Usage Notes¶
Namespace Pollution¶
The ci_exec
package namespace is polluted intentionally. Always import from the
polluted namespace, never import from the module originally defining something:
from ci_exec import which # Yes
from ci_exec.core import which # NO!
In practice it shouldn’t matter that much, but
Any functions moved to different modules will not be considered ABI breaking changes. So if
ci_exec.core.which
moved to a differentci_exec.other_module
, this would be done in a patch release (ci_exec
uses semantic versioning).Anything that is not exposed at the top-level is more than likely something that should not be used.
Beware of Commands Expecting Input¶
The Executable.__call__()
method internally
invokes subprocess.run()
, which does some fancy manipulations of
stdout
, stderr
, and stdin
. In particular, the subprocess by default
will allow communication on stdin
for commands that are expecting input. In the
context of CI, this is problematic, and users need to be aware that the subprocess will
indeed pause and solicit user input. Consider a scenario where say filter_file()
was
used to make some significant changes on the CI provider. If you ran:
from ci_exec import which
git = which("git")
git("diff")
The build may fail from a timeout rather than just displaying the changes, because for a
large enough change to report git
will want to use your $PAGER
to let you
scroll through it. Most commands that allow input also provide a command-line flag to
disable this behavior and in this scenario, it would be:
git("--no-pager", "diff")
In situations where you know that input is required, subprocess.run()
enables this via the input
parameter. Consider the following toy script that is
expecting two pieces of user input at different stages:
name = input("What is your name? ")
print(f"Hi {name}, nice to meet you.")
age = input("How old are you? ")
print(f"Wow! {age} is such a great age :)")
Then you could call it using ci_exec
like this:
import sys
from ci_exec import Executable
if __name__ == "__main__":
python = Executable(sys.executable)
python("./multi_input.py", input=b"Bilbo\n111")
where
The
input
parameter needs abytes
object (theb
prefix inb"..."
).You use
\n
to send a newline. From the docs: For stdin, line ending characters'\n'
in the input will be converted to the default line separatoros.linesep
. So"Bilbo\n"
will end up being the answer to the firstname = input(...)
, and"111"
will be sent to theage = input(...)
.
Build Logs out of Order (Azure Pipelines)¶
If the results of call logging appear out of order, then your CI provider is affected by this. This is a known problem on Azure Pipelines, it seems unlikely to affect many other providers due to the nature of how the Azure Pipelines build logs are served. Example “script”:
cmake("..")
ninja()
Produces:
-- The C compiler identification is GNU 8.3.1
-- The CXX compiler identification is GNU 8.3.1
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
...
[12/74] Building C object ...joystick.c.o
$ /usr/bin/cmake ..
$ /usr/bin/ninja
The log is out of order, all cmake / ninja output appeared before the call logging
($ /usr/bin/cmake ..
and $ /usr/bin/ninja
). There are two possible solutions:
Invoke your build script using python -u:
python -u ./.ci/build.py
Set the environment variable PYTHONUNBUFFERED=true.
Changelog¶
v0.1.2¶
Drop python 3.5 support (#27): python 3.6 or later required.
Add
Visual Studio 17 2022
andNinja Multi-Config
toCMakeParser
(#29).
v0.1.1¶
Added
CMakeParser
(#16).
v0.1.0.post0¶
Fix support for early 3.5.x by conditionally importing
NoReturn
andTYPE_CHECKING
(#12).
v0.1.0¶
Initial release. Broken for python 3.5.x where typing
module does not house
NoReturn
and TYPE_CHECKING
. Definitely broken on 3.5.2 and earlier.