This document gives an overview how to run tests with autopkgtest. It does not cover each detail, please consult the individual manpages like adt-run(1), adt-virt-schroot(1), etc. for all available options.
Ingredients for Debian packages:
A source package which defines tests in debian/tests/. See README.package-tests for how to define them.
There are plenty of existing package tests in Debian/Ubuntu which you can use as examples and inspiration, just look for a source package with a Testsuite: autopkgtest header, or the automatic test running services in Debian and Ubuntu (e. g. for 14.04 LTS).
A location for the source/tests: This can be a local source tree, a local .dsc, or "download with apt-get source".
Optionally some pre-built binary packages which should be tested.
Ingredients for Click packages:
Finally you need a virtualization server, which creates the environment in which the test runs. Depending on how intrusive the test is this can provide various degrees of isolation, from "run on my local system" (fastest, but unsafe) to "run in a temporary virtual machine" (slowest, but highest possible isolation). These are described in detail below.
The adt-run program is the main program to run tests which gets all these ingredients as arguments, in the following form:
adt-run [options] <source package> --- <virt-server> [<virt-server options>]
All possible options are explained in the adt-run(1) manpage. This section shows the most common scenarios, with using "mysrc" as source package name. Note that specifying the virtualization server and its options is handled in the following section, and it is independent of specifying tests and packages, so it is merely abbreviated as virt-server here.
Run tests from the Debian source package in the distribution. This calls apt-get source mysrc in the virt-server, thus will use whichever distribution/release that /etc/apt/sources.list configures:
adt-run mysrc --- virt-server
Run tests from a local unbuilt source tree, using the binary packages from the distribution. This is useful if you are creating or fixing tests, but the actual binary packages are fine:
adt-run -B --unbuilt-tree=packages/mysrc --- virt-server
Note that you can abbreviate --unbuilt-tree= with just the directory with two(!) trailing slashes, i. e.
adt-run -B packages/mysrc// --- virt-server
Run tests from a local built source tree, using the binary packages from the distribution:
adt-run -B --built-tree=packages/mysrc --- virt-server
Note that you can abbreviate --built-tree= with just the directory with one(!) trailing slash, i. e.
adt-run -B packages/mysrc/ --- virt-server
Built vs. unbuilt only makes a difference for tests which declare the build-needed restriction (see README.package-tests), in which case --built-tree avoids having to re-build the source in the virt-server.
Build a local source package in the virt-server, then run its tests against the built binaries. This is useful if you need to fix a bug in the actual packages to make the tests succeed:
adt-run packages/mysrc// --- virt-server
Same as above, but with specifying a built source package instead of a source tree:
adt-run packages/mysrc_*.dsc --- virt-server
Test new built binaries with a new source and avoid rebuilding them in virt-server. This is useful if you e. g. update a package to a new version and now want to check whether its tests still succeed:
adt-run -B packages/*.deb packages/mysrc_*.dsc --- virt-server
The previous case can be simplified if you have a binary .changes from a previous build:
adt-run packages/*.changes --- virt-server
Run tests for a locally built click package:
adt-run myclickapp/ myclickapp_0.1_all.click --- virt-server
Note that for this you need to specify a virt-server which has "click" itself and the click app's required framework already installed. To run this on an actual Ubuntu phone, you can use the SSH testbed server:
adt-run ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click --- ssh -s adb
(This is using the shipped adb setup script in /usr/share/autopkgtest/ssh-setup/.)
If you aren't running the test on an Ubuntu touch device, you can approximate the environment in a normal schroot/LXC/QEMU testbed with the --setup-commands scripts that autopkgtest provides (note that you do not need to specify the full /usr/share/autopkgtest/setup-commands/ path for shipped scripts):
adt-run --setup-commands ubuntu-touch-session \
--setup-commands apt-ro \
myclickapp/ myclickapp_0.1_all.click --- lxc adt-utopic
See the comments in the setup-commands scripts for details.
Run tests for an already installed click package:
adt-run --click=com.example.myapp --- virt-server
This will work for click apps which have an x-source/vcs-bzr entry in their manifest. If that's not the case, you will need to explicitly specify the click source directory as above.
Unless you specify some options, adt-run just writes the logging, test outputs, and test results to stdout/stderr and exits with code 0 on success, or some non-zero code if there were skipped or failed tests or problems with the virt-server. (See adt-run(1) for defined codes).
For getting output files you have three choices:
If you just want the "testname: PASS/FAIL" results, use
--summary-file=/path/to/summary.txt.
If you want the complete output of adt-run in a file, use
-l /path/to/test.log (or the long option --log-file)
If you want the log file, the individual test stdout and stderr output, and built binaries (if any) in a directory, use
-o /path/to/test-output/ (or the long option --output-dir).
You can also combine these.
adt-run ... --- schroot schroot-name
Run tests in the specified schroot. You can use mk-sbuild(1) to conveniently create schroots, and run this as normal user if you configured schroot accordingly.
This server is the fastest available that provides "proper" file system isolation and revert, but it does not provide enough isolation for tests that need to start services, reconfigure the network, or open TCP ports which are already open at the host. If your test does not need to do these things this is the recommended server, as schroots are also useful for other tasks like building packages with sbuild.
See adt-virt-schroot(1) manpage.
adt-run ... --- lxc container-name
Run tests in the specified LXC container. Containers provide full service and network isolation, but tests or packages cannot change the kernel or hardware configuration. If your test does not need that, this is the recommended server as it is faster than QEMU and works on all Linux architectures.
container-name will be cloned or be called with a temporary overlay file system if you specify the -e (--ephemeral) option, thus it will never be modified and you can run several tests in parallel safely. Unless your test or architecture or RAM availability doesn't work with overlayfs, using -e is highly recommended for better performance.
If your user can get root privileges with sudo, you can call adt-run as your normal user and specify -s (--sudo) so that the container can be started as root.
See adt-virt-lxc(1) manpage. This also explains how to build containers.
adt-run ... --- qemu path/to/image
Run tests with QEMU/KVM using the specified image. The image will be run with a temporary overlay file system, thus it will never be modified and you can run several tests in parallel safely.
If your test needs a full machine including kernel/hardware access, this is the recommended runner; it provides complete system isolation, revert and breaks-testbed capabilities. But it is also the one with the biggest overhead and only works well on architectures with KVM acceleration (i. e. mostly x86).
See adt-virt-qemu(1) manpage. This also explains how to build suitable images, and the requirements of the guest.
adt-run ... --- null
This does not do any virtualization, but runs tests straight on the host. Beware that this will leave some clutter on your system (installed test or build dependency packages, configuration changes that the tests might make, etc.). It is not able to run tests with the "breaks-testbed" restriction. See adt-virt-null(1) manpage.
adt-run ... --- chroot /path/to/chroot
Run tests in the specified chroot. You need to call adt-run as root for this. There is no automatic cleanup or revert for the chroot, so unless you can provide this by some other means, don't use this.
adt-run ... --- ssh -l joe -h testhost.example.com
This is a generic runner for an externally set up testbed which assumes nothing else than a working ssh connection. This can call a "setup script" to create/configure a testbed (such as spinning up a cloud VM with nova or setting up SSH on a phone through ADB). See the manpage for details. autopkgtest ships setup scripts for an adb host (mostly for Ubuntu Touch) and for nova currently, see their comment headers in /usr/share/autopkgtest/ssh-setup/.