Ambient user guide

By: Lars Wirzenius

Version:

Table of Contents

1 Introduction

Ambient does automated continuous integration for programmers in a safe and secure way. That means it builds the software and runs its automated tests, but prevents the software under test from harming the host it runs on. Ambient achieves this by only running code from the software under test, or any of its dependencies, in a virtual machine without network access, and only the resources configured by the Ambient user.

2 Installation

See INSTALL.md in the source tree.

3 Getting started

3.1 Built-in help

The ambient command has extensive builtin help: give the -h or --help option to the command, or any subcommand, or use help subcommand:

ambient -h
ambient --help
ambient run --help
ambient help
ambient help run

3.2 Projects file

To run CI on a project using Ambient, you need to create a "projects file".

projects:
  dummy:
    image: /scratch/ambient-images/ambient-boot.qcow2
    source: ~/pers/ambient-ci/ambient-ci
    pre_plan:
      - action: cargo_fetch
    plan:
      - action: cargo_clippy
      - action: cargo_build
      - action: cargo_doc
      - action: cargo_test

The projects file, in YAML, specifies the projects Ambient should know about. For each project:

  • image is the virtual machine image to use
  • source is the path to the source directory of the project
    • it can be, but does not need to be, a Git checkout
  • pre_plan, plan, and post_plan are lists of actions to execute for a CI run
    • pre_plan actions are executed before the VM starts
    • plan actions are executed inside the running VM
    • post_plan actions are executed after the VM finishes

In each kind of plan, only specific actions are allowed. For pre_plan, the following are allowed:

  • cargo_fetch - download Rust crate dependencies for the software under test
  • http_get - download specific files from URLs

For plan:

  • cargo_fmt, cargo_clippy, cargo_deny, cargo_doc, cargo_build, cargo_test cargo_install - run the corresponding cargo command
  • deb - build a deb package
  • custom - run an executable action from .ambient in the root of the source tree

For post_plan:

  • dput - upload built deb packages to an APT repository
  • rsync - upload built artifact files to a server using rsync

For post-plan actions, the credentials of the user running ambient are used.

See the actions chapter for a complete list.

3.3 Configuration

You should configure Ambient by creating ~/.config/ambient/config.yaml (the location obeys the XDG base directory spec):

tmpdir: /scratch/tmp
projects: ~/liw-dot-files/ambient.yaml
target: "_ewww@webby:/srv/http"
dput_target: apt.liw.fi
executor: /scratch/cargo-cache/x86_64-unknown-linux-musl/debug/ambient-execute-plan
artifacts_max_size: 1G
cache_max_size: 50G
state: /scratch/ambient-state
qemu:
  cpus: 8
  memory: 16G

The fields are:

  • tmpdir - location where Ambient temporary files are put
    • these can get quite large, so it can be necessary to override the default $TMPDIR
  • projects - path to the projects file
  • target - where the rsync action uploads files over SSH
  • dput_target - where deb packages are upload using dput
  • executor - the ambient-execute-plan program suitable for the VM used
  • artifacts_max_size, cache_max_size - how big the artifacts and cache directories can grow
  • state - location where Ambient keeps pre-project state
  • qemu.cpus, qemu.memory - number of CPUs and memory to give to the VM

Some of those are optional, as they have defaults. To see the actual run-time configuration used by ambient, run:

ambient config

You can specify the configuration file to use with the --config option.

3.4 Run Ambient

Given an image and a projects file, run Ambient using a command like this:

ambient run

This runs CI for every project in the projects file, if the project source code has changed. Add the --force option to force CI to run.

Any output to the standard output and error from any of the actions is written to the standard output and gathered in a "run log". The run log can be viewed later with

ambient log dummy

The run log is, for now, quite messy.

4 Actions for use in Ambient CI plans

An action is an atomic task executed during a CI run. Actions are specified in the CI plan. The plan is divided into a pre-plan, actual plan, and post-plan. The pre- and post-plan actions are executed with network access and the actual plan is executed in an isolated virtual machine with no network access. The actions allowed in pre- and post-plan are carefully designed and vetted so that they are safe and secure to run. Most importantly, they do not execute any code from the software under test.

4.1 The workspace in the virtual machine

Ambient sets up a workspace where CI is run. Until version 0.10, this was /workspace, but after that, it is also known as /ci, for brevity. The old name also works to avoid breaking existing CI plans.

The workspace contains several directories:

  • /ci/src - the source code; this is the contents of the source directory in the project specification; changes in the source directory are not persisted and vanish when the VM shuts down
  • /ci/dependencies - dependencies downloaded by pre-plan actions; changes in the dependencies directory are not persisted and vanish when the VM shuts down
  • /ci/cache - a cache that persists between CI runs
  • /ci/artifacts - files built by CI, for publication by post-plan actions

The cache and artifacts directories have a maximum size. When the CI run ends, the contents of those directories are written as tar archives to the corresponding virtual block devices set up by Ambient. This is what will fail if the directories are too large. See artifacts_max_size and cache_max_size in the configuration file.

You mostly only need to care about the workspace and its layout in the shell action. All other actions know what's where, but your shell script snippets need to use the right locations. Also any custom action executables need to do that.

Ambient adds actions to unpack the source and other directories at the start of a CI run, and exporting the cache and artifacts directories at the end. You don't need to do that manually.

4.2 Pre-plan actions

The following actions are allowed in the pre-plan.

4.2.1 dummy

- action: dummy

Do nothing, except write a message to the standard output. This action is meant for trouble-shooting pre-plans.

4.2.2 pwd

- action: pwd

Write out the path to the current working directory. This action is meant for trouble-shooting.

4.2.3 cargo_fetch

- action: cargo_fetch

Download Rust crate dependencies using cargo fetch. The downloaded crates will be available in the VM in a location where cargo will find them. The downloaded crates will be cached so that they do not need to be downloaded again for the next CI run.

For safety and security, cargo fetch is run on a stripped down copy of the source tree. This prevents the software under test from reconfiguring cargo using files in the source tree. This does not affect what crates are downloaded.

The crates will be in /ci/deps and the CARGO_HOME environment variable is set to that location.

4.2.4 http_get

- action: http_get
  items:
  - url: https://files.example.com/big.xz
    filename: data.xz
  - url: https://files.example.com/cat.jpg
    filename: cat.jpg

Download files over HTTP if they are missing locally or have changed on the server. The downloaded files are available in the dependencies directory in the VM, /ci/deps. If the file exists locally, the download uses the HTTP header If-Modified-Since to avoid downloading it again, unless the file on the server has a newer modification time.

The downloads are done using HTTP GET. There is no support for authentication. There is no automatic cleaning of downloaded files.

Both the url and filename fields are required. The filename may not contain a directory.

4.3 Actual plan actions

These are also known as "unsafe actions". They're allowed to be dangerous and to run code from the software under test, because they get executed in a virtual machine that isolates and constrains the code so it can't harm the host running Ambient, or any other hosts.

4.3.1 mkdir

- action: mkdir
  pathname: /ci/src

Ensure a directory exists. If it already does, do nothing, otherwise create it. This is meant for Ambient internal use, to create the CI workspace when a CI run starts. User-provided plans can also use this action.

4.3.2 tar_create

- action: tar_create
  archive: /ci/artifacts/rsync/source.tar
  directory: /ci/src

Create a tar archive with the contents of a directory. This is meant for Ambient internal use to export data from the virtual machine via virtual block devices, i.e., the cache and artifacts directories. User-provided plans can also use this action.

4.3.3 tar_extract

- action: tar_extract
  archive: /dev/vdx
  directory: /ci/yummy

Extract a tar archive to a directory. This is meant for Ambient internal use to import data into the virtual machine via virtual block devices, e.g., the source, dependencies, and cache directories. User-provided plans can also use this action.

4.3.4 shell

- action: shell
  shell: |
    echo hello, world
    make

Execute a shell script snippet using bash. The Bash setting set -xeuo pipefail will be in effect. This means that if the snippet uses a variable that hasn't been set, or a command fails, the action fails.

4.3.5 cargo_fmt

- action: cargo_fmt

Check that Rust source code is formatted in the idiomatic style, by running cargo fmt --check.

4.3.6 cargo_clippy

- action: cargo_clippy

Check that Rust code is correct and idiomatic by running cargo clippy. Any warnings are treated as errors.

4.3.7 cargo_deny

- action: cargo_deny

Check that Rust code only has dependencies, licenses, and known security problems that are explicitly allowed, by running cargo deny.

4.3.8 cargo_doc

- action: cargo_doc

Format documentation from Rust code, using cargo doc. The formatted documentation is put in the cargo target directory, CARGO_TARGET, which is in the cache directory. It is not automatically put into the artifacts directory. You have to do that yourself, if you want it.

4.3.9 cargo_build

- action: cargo_build

Build a Rust project, including binaries, tests, and examples ("all targets"). This runs cargo build with suitable options.

4.3.10 cargo_test

- action: cargo_test

Run tests in a Rust project, using cargo test.

4.3.11 cargo_install

- action: cargo_install

Install a Rust project into the artifacts directory, /ci/artifacts.

4.3.12 deb

- action: deb

Build a Debian deb package. This first creates an "upstream tar" from the source tree, using git archive, and then running dpkg-buildpackage, the standard tool for building Debian packages. The built files, including the .changes file, get put into the artifacts directory, /ci/artifacts.

Note that this assumes the source is in Git, that Debian packaging tools are installed in the VM, and that the debian directory includes the necessary files for Debian packaging.

Also note that this action does not invent a version number for CI builds.

This action is meant to be used together with the dput post-plan action.

This action differs from the deb2 action only in the location of where built packages are put.

4.3.13 deb2

- action: deb2

Build a Debian deb package. This first creates an "upstream tar" from the source tree, using git archive, and then running dpkg-buildpackage, the standard tool for building Debian packages. The built files, including the .changes file, get put into debian directory in the artifacts directory, /ci/artifacts/debian.

Note that this assumes the source is in Git, that Debian packaging tools are installed in the VM, and that the debian directory includes the necessary files for Debian packaging.

Also note that this action does not invent a version number for CI builds.

This action is meant to be used together with the dput2 post-plan action.

This action differs from the deb action only in the location of where built packages are put.

4.3.14 custom

- action: custom
  name: dch
  args:
    debemail: liw@liw.fi
    debfullname: "Lars Wirzenius"

Execute a "custom action". Custom actions are executables (usually shell scripts, but any executable is OK), located in the .ambient directory at the root of the source tree. The custom action runs the executable and passes it arguments as specified in the args field. The whole args field is encoded as JSON and given to the executable via its standard input. In addition, each argument name is separately encoded as JSON and an environment variable AMBIENT_CI_name is set to contain the JSON. The executable can use either, whichever is more convenient to it.

Note that Ambient does not try to retrieve the custom action executables from anywhere. The project that uses the custom action has to include the executable in their source tree. This avoids making Ambient do package management for the executables, but make using them more tedious. The custom action is best considered a compromise while the Ambient project explores the needs and wants for custom actions. The Ambient project maintains a small repository of custom actions as part of the compromise.

Example custom action script to add a Debian package changelog entry with a new version suitable for a CI build:

#!/usr/bin/env bash

set -euo pipefail

export DEBEMAIL="$AMBIENT_CI_debemail"
export DEBFULLNAME="$AMBIENT_CI_debfullname"
export CARGO_TARGET_DIR=/workspace/cache
export CARGO_HOME=/workspace/deps
export HOME=/root
export PATH="/root/.cargo/bin:$PATH"

git reset --hard
git clean -fdx

V="$(dpkg-parsechangelog -SVersion | sed 's/-[^-]*$//')"
T="$(date -u "+%Y%m%dT%H%M%S")"
version="$V.ci$T-1"
dch -v "$version" "CI build under Ambient."
dch -r ''

(The example comes from the repository of custom actions, script dch.)

4.3.15 setenv

- action: setenv
  set:
    foo: bar

Set environment variables for later actions.

4.4 Post-plan actions

4.4.1 dummy

This is the same action as in the pre-plan.

4.4.2 pwd

This is the same action as in the pre-plan.

4.4.3 rsync

- action: rsync

Publish the contents of the entire artifacts directory (/ci/artifacts in the VM) using the rsync program. The sync target is specified in Ambient configuration (see rsync_target); the software under test cannot affect that. The SSH credentials for the host running Ambient are used by rsync. Any files on the target that are not in the artifacts directory are deleted.

4.4.4 rsync2

- action: rsync2

Publish the contents of the rsync subdirectory of the artifacts directory (/ci/artifacts/rsync in the VM) using the rsync program. The sync target is specified in Ambient configuration (see rsync_target); the software under test cannot affect that. The SSH credentials for the host running Ambient are used by rsync. Any files on the target that are not in the artifacts directory are deleted.

The difference between the rsync and rsync2 actions is the location of the files to be published. The rsync action publishes the entire artifacts directory, which may also contain packages built by the deb action. The rsync2 action only publishes the subdirectory, which contains no packages.

4.4.5 dput

- action: dput

Upload Debian deb packages from the artifacts directory to an APT repository, using the dput program on the host and the SSH credentials of the host. The packages are found by looking for .changes files anywhere inside /ci/artifacts, including subdirectories. The deb action puts them in /ci/artifacts.

4.4.6 dput2

- action: dput2

Upload Debian deb packages from the artifacts directory to an APT repository, using the dput program on the host and the SSH credentials of the host. The packages are found by looking for .changes files anywhere inside /ci/artifacts/debian, including subdirectories. The deb2 action puts them there.