wasi-libc/test/scripts/run-test.sh
Andrew Brown 2b853ff079
test: improve test infrastructure (#554)
This change represents a rather large re-design in how `wasi-libc`
builds and runs its tests. Initially, #346 retrieved the `libc-test`
repository and built a subset of those tests to give us some amount of
test coverage. Later, because there was no way to add custom C tests,
#522 added a `smoke` directory which allowed this. But (a) each of these
test suites was built and run separately and (b) it was unclear how to
add more tests flexibly--some tests should only run on `*p2` targets or
`*-threads` targets, e.g.

This change reworks all of this so that all tests are built the same
way, in the same place. For downloaded tests like those from
`libc-test`, I chose to add "stub tests" that `#include` the original
version. This not only keeps all enabled tests in one place, it also
allows us to add "directives," C comments that the `Makefile` uses to
filter out tests for certain targets or add special compile, link or run
flags. These rudimentary scripts, along with other Bash logic I moved
out of the Makefile now live in the `scripts` directory.

Finally, all of this is explained more clearly in an updated
`README.md`. The hope with documenting this a bit better is that it
would be easier for drive-by contributors to be able to either dump in
new C tests for regressions they may find or enable more libc-tests. As
of my current count, we only enable 40/75 of libc-test's functional
tests, 0/228 math tests, 0/69 regression tests, and 0/79 API tests.
Though many of these may not apply to WASI programs, it would be nice to
explore how many more of these tests can be enabled to increase
wasi-libc's test coverage. This change should explain how to do that
and, with directives, make it possible to condition how the tests
compile and run.
2024-12-09 13:45:31 -08:00

21 lines
622 B
Bash
Executable File

#/usr/bin/env bash
# Run a previously built test, generating in `$DIR`:
# - a `cmd.sh` script containing the actual command used to run the test in an
# engine
# - a `fs` directory which the test may have used for file IO
# - an `output.log` file containing the output of the test
#
# Usage: DIR=... WASM=... ENGINE=... ./run-test.sh
ENGINE="${ENGINE:-wasmtime}"
[ -n "$WASM" ] || (echo "missing WASM variable" && exit 1)
[ -n "$DIR" ] || (echo "missing DIR variable" && exit 1)
cd $DIR
mkdir -p fs
echo "$ENGINE $WASM" > cmd.sh
chmod +x cmd.sh
./cmd.sh &> output.log
[ $? -eq 0 ] || echo "Test failed" >> output.log