Currently, the 'upstream' test runs all upstream-provided tests at tests/TEST-*,
except for any that are in a hardcoded list contained in this test.
Any change to the hardcoded blacklist involves patching the debian repo, which is
not an ideal way for upstream systemd to manage a hopefully temporary blacklist of
problematic tests.
There was discussion around making the blacklist entirely provided by an upstream
env var, configured in the upstream webhooks that control the Ubuntu CI tests,
but that results in 2 problems; 1) Debian would no longer be able to easily
control its own blacklist, and 2) upstream would only have an all-or-none control
over the blacklist, meaning that it could only enable or disable any specific
test for all PRs; it could not enable a test for only a specific PR that was
attempting to fix the flaky test (for example).
This approach moves blacklist control into the systemd tests themselves, by changing
the 'upstream' test to look for a file in the test directory, and skipping the test
if such file is found. This way, Debian (and Ubuntu and other Debian derivatives)
can continue to manage their own blacklist, but also upstream can control
the blacklist for individual tests on a per-PR basis.
For example, if upstream has the file 'tests/TEST-01-BASIC/blacklist-ubuntu-ci'
in its repo, Ubuntu CI will skip this test for all PRs opened. Then, a PR can
be opened that both fixes the test, as well as removes this file. The Ubuntu CI
will then run the test, but only for the PR that attempts to fix it. Once that
PR is merged, all future tests will then run the fixed test.
The specific naming of the per-test blacklist file, for tests run from the
upstream repo, is either "blacklist-ubuntu-ci" to completely blacklist the test
on all archs, or "blacklist-ubuntu-ci-$DPKGARCH" to blacklist the test only
for a specific arch, for example "blacklist-ubuntu-ci-amd64". For tests run
from Debian (or Ubuntu), which are run without the TEST_UPSTREAM param set,
the blacklist filename is 'blacklist-upstream-ci[-$DPKGARCH]'.
Note that the $DPKGARCH is specified as the value returned by
'dpkg --print-architecture', not the value returned from 'uname -m' (e.g.
to blacklist a test for intel 64-bit, 'amd64' should be used, not 'x86_64').
This naming matches the title of the ubuntu tests, such as 'bionic-amd64',
'bionic-arm64', etc.
run the 'clean' target before setup/run, to match the upstream
'all' target's behavior; also the clean (again) target must be
run separately, as the failure of setup or run will stop make,
and cleanup won't be done if called in the same make line.
Unfortunately, these tests are failing on i386 builds, very
intermittently; this causes headaches for upstream systemd
maintainers and submitters, when the Ubuntu CI fails, because
the failure isn't related to the PRs the tests are running for,
and determining that from the Ubuntu CI log file and other artifacts
is very time-consuming.
This just blacklists the tests for now, until we can figure out why
they are failing and fix them.
TEST-30 is discussed:
https://github.com/systemd/systemd/issues/12268
TEST-34 is discussed:
https://github.com/systemd/systemd/issues/12932
Otherwise logs are missing on failures:
cp: -r not specified; omitting directory '/var/tmp/systemd-test.Nq2jqR/journal/59852163a37d458f9d238b65f279b6fa'
Showing the entire debug log is too hard to scan visually, and most of
the time the warnings and errors are sufficient to explain a failure.
Put the journal files into the artifacts though, in case the debug
information is necessary.
_ninja_bin was added in https://github.com/systemd/systemd/pull/6544 in
order to make the tests work on CentOS. As we don't actually do a "ninja
install" and ninja is not available, replace the check with a dummy
value.
Makefile.guess tries to find the build directory if $BUILD_DIR is not
set. This doesn't always exist for an autopkgtest, thus fix it to ".";
it is not actually being used anyway. Adjust the sed for using the
system-installed nspawn binary accordingly.
"dpkg-deb -x" replaces the "/bin -> usr/bin" symlinks with directories, as
that's tar's default. Call tar manually with --dereference to keep the links.
https://github.com/systemd/systemd/issues/4630
Gbp-Dch: short
This avoids replicating the logic and also takes building helper binaries into
account.
The current Makefiles call "make all" on the entire upstream tree which we need
to disable -- first, we don't actually want/need to do that for testing .debs,
and second the build should not happen as root.
This runs the test/TEST* upstream integration tests in QEMU and nspawn.
We need to modify test-functions a bit: We don't want to run this against a
built source tree (which we don't even have for a distro test run), but against
the installed packages.
Skip TEST-06-SELINUX for now as this does not work in Ubuntu (even after
install policycoreutils): The test expects a /usr/libexec/selinux/hll/pp file
which does not exist.
