![]() This change represents a rather large re-design in how `wasi-libc` builds and runs its tests. Initially, #346 retrieved the `libc-test` repository and built a subset of those tests to give us some amount of test coverage. Later, because there was no way to add custom C tests, #522 added a `smoke` directory which allowed this. But (a) each of these test suites was built and run separately and (b) it was unclear how to add more tests flexibly--some tests should only run on `*p2` targets or `*-threads` targets, e.g. This change reworks all of this so that all tests are built the same way, in the same place. For downloaded tests like those from `libc-test`, I chose to add "stub tests" that `#include` the original version. This not only keeps all enabled tests in one place, it also allows us to add "directives," C comments that the `Makefile` uses to filter out tests for certain targets or add special compile, link or run flags. These rudimentary scripts, along with other Bash logic I moved out of the Makefile now live in the `scripts` directory. Finally, all of this is explained more clearly in an updated `README.md`. The hope with documenting this a bit better is that it would be easier for drive-by contributors to be able to either dump in new C tests for regressions they may find or enable more libc-tests. As of my current count, we only enable 40/75 of libc-test's functional tests, 0/228 math tests, 0/69 regression tests, and 0/79 API tests. Though many of these may not apply to WASI programs, it would be nice to explore how many more of these tests can be enabled to increase wasi-libc's test coverage. This change should explain how to do that and, with directives, make it possible to condition how the tests compile and run. |
||
---|---|---|
.. | ||
scripts | ||
src | ||
.gitignore | ||
Makefile | ||
README.md |
Test
This directory runs C tests compiled to WebAssembly against wasi-libc
to check
its functionality. It enables a subset of libc-test as well as adding custom C
tests; all enabled tests are contained in the src
directory.
Pre-requisites
- Clang
- [
libc-test
] libclang_rt.builtins-wasm32.a
- a WebAssembly engine
- other WebAssembly tools, especially for
wasm32-wasip2
support (see the [`Makefile] for a complete list)
All but Clang are downloaded automatically by the make download
target.
Build and run
To build and run all tests:
$ make TARGET_TRIPLE=...
Tests passed
Note that wasm-ld
must be available, so an additional
CC=<wasi-sdk>/bin/clang
may be necessary. Each test runs in a directory that
looks like (see run-test.sh
):
$ ls run/$TARGET_TRIPLE/misc/some-test
cmd.sh # the exact command used to run the test
fs # a directory containing any test-created files
output.log # the captured printed output--only for errors
Adding tests
To add a test, create a new C file in src/misc
:
//! filter.py(TARGET_TRIPLE): !wasm32-wasip2
//! add-flags.py(CFLAGS): ...
//! add-flags.py(LDFLAGS): ...
//! add-flags.py(RUN): ...
void main() { ... }
- to pass, the
main
function must exit successfully and avoid printing output - the
filter.py
directive controls when the test builds and runs (e.g., not forwasip2
) - the
add-flags.py
directive adds extra information for building or running the test (see theMakefile
for precise use).
Enabling more libc-test tests
libc-test has more tests available that are not yet enabled (e.g., to count
the enabled subset, find src -name *.c | wc -l
). Each enabled test contains a
stub file in src/libc-test
that #include
s its downloaded version and adds
various filter.py
and add-flags.py
directives.
To quickly create stub files for not-yet-enabled tests:
$ make generate-stubs
$ git status
...
src/libc-test/functional/tls_align.c
src/libc-test/functional/tls_align_dlopen.c
src/libc-test/functional/tls_align_dso.c
src/libc-test/functional/tls_init.c
Then modify the directives for these new stub files to get the new tests to compile and successfully run.