wasi-libc/test/scripts
Andrew Brown 2b853ff079
test: improve test infrastructure (#554)
This change represents a rather large re-design in how `wasi-libc`
builds and runs its tests. Initially, #346 retrieved the `libc-test`
repository and built a subset of those tests to give us some amount of
test coverage. Later, because there was no way to add custom C tests,
#522 added a `smoke` directory which allowed this. But (a) each of these
test suites was built and run separately and (b) it was unclear how to
add more tests flexibly--some tests should only run on `*p2` targets or
`*-threads` targets, e.g.

This change reworks all of this so that all tests are built the same
way, in the same place. For downloaded tests like those from
`libc-test`, I chose to add "stub tests" that `#include` the original
version. This not only keeps all enabled tests in one place, it also
allows us to add "directives," C comments that the `Makefile` uses to
filter out tests for certain targets or add special compile, link or run
flags. These rudimentary scripts, along with other Bash logic I moved
out of the Makefile now live in the `scripts` directory.

Finally, all of this is explained more clearly in an updated
`README.md`. The hope with documenting this a bit better is that it
would be easier for drive-by contributors to be able to either dump in
new C tests for regressions they may find or enable more libc-tests. As
of my current count, we only enable 40/75 of libc-test's functional
tests, 0/228 math tests, 0/69 regression tests, and 0/79 API tests.
Though many of these may not apply to WASI programs, it would be nice to
explore how many more of these tests can be enabled to increase
wasi-libc's test coverage. This change should explain how to do that
and, with directives, make it possible to condition how the tests
compile and run.
2024-12-09 13:45:31 -08:00
..
add-flags.py test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00
failed-tests.sh test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00
filter.py test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00
generate-stubs.sh test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00
run-test.sh test: improve test infrastructure (#554) 2024-12-09 13:45:31 -08:00