wasi-libc/test
Andrew Brown 2b853ff079
test: improve test infrastructure (#554)
This change represents a rather large re-design in how `wasi-libc`
builds and runs its tests. Initially, #346 retrieved the `libc-test`
repository and built a subset of those tests to give us some amount of
test coverage. Later, because there was no way to add custom C tests,
#522 added a `smoke` directory which allowed this. But (a) each of these
test suites was built and run separately and (b) it was unclear how to
add more tests flexibly--some tests should only run on `*p2` targets or
`*-threads` targets, e.g.

This change reworks all of this so that all tests are built the same
way, in the same place. For downloaded tests like those from
`libc-test`, I chose to add "stub tests" that `#include` the original
version. This not only keeps all enabled tests in one place, it also
allows us to add "directives," C comments that the `Makefile` uses to
filter out tests for certain targets or add special compile, link or run
flags. These rudimentary scripts, along with other Bash logic I moved
out of the Makefile now live in the `scripts` directory.

Finally, all of this is explained more clearly in an updated
`README.md`. The hope with documenting this a bit better is that it
would be easier for drive-by contributors to be able to either dump in
new C tests for regressions they may find or enable more libc-tests. As
of my current count, we only enable 40/75 of libc-test's functional
tests, 0/228 math tests, 0/69 regression tests, and 0/79 API tests.
Though many of these may not apply to WASI programs, it would be nice to
explore how many more of these tests can be enabled to increase
wasi-libc's test coverage. This change should explain how to do that
and, with directives, make it possible to condition how the tests
compile and run.
2024-12-09 13:45:31 -08:00
..
scripts test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00
src test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00
.gitignore test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00
Makefile test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00
README.md test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00

Test

This directory runs C tests compiled to WebAssembly against wasi-libc to check its functionality. It enables a subset of libc-test as well as adding custom C tests; all enabled tests are contained in the src directory.

Pre-requisites

  • Clang
  • [libc-test]
  • libclang_rt.builtins-wasm32.a
  • a WebAssembly engine
  • other WebAssembly tools, especially for wasm32-wasip2 support (see the [`Makefile] for a complete list)

All but Clang are downloaded automatically by the make download target.

Build and run

To build and run all tests:

$ make TARGET_TRIPLE=...
Tests passed

Note that wasm-ld must be available, so an additional CC=<wasi-sdk>/bin/clang may be necessary. Each test runs in a directory that looks like (see run-test.sh):

$ ls run/$TARGET_TRIPLE/misc/some-test
cmd.sh      # the exact command used to run the test
fs          # a directory containing any test-created files
output.log  # the captured printed output--only for errors

Adding tests

To add a test, create a new C file in src/misc:

//! filter.py(TARGET_TRIPLE): !wasm32-wasip2
//! add-flags.py(CFLAGS): ...
//! add-flags.py(LDFLAGS): ...
//! add-flags.py(RUN): ...
void main() { ... }
  • to pass, the main function must exit successfully and avoid printing output
  • the filter.py directive controls when the test builds and runs (e.g., not for wasip2)
  • the add-flags.py directive adds extra information for building or running the test (see the Makefile for precise use).

Enabling more libc-test tests

libc-test has more tests available that are not yet enabled (e.g., to count the enabled subset, find src -name *.c | wc -l). Each enabled test contains a stub file in src/libc-test that #includes its downloaded version and adds various filter.py and add-flags.py directives.

To quickly create stub files for not-yet-enabled tests:

$ make generate-stubs
$ git status
...
src/libc-test/functional/tls_align.c
src/libc-test/functional/tls_align_dlopen.c
src/libc-test/functional/tls_align_dso.c
src/libc-test/functional/tls_init.c

Then modify the directives for these new stub files to get the new tests to compile and successfully run.