This change represents a rather large re-design in how `wasi-libc`
builds and runs its tests. Initially, #346 retrieved the `libc-test`
repository and built a subset of those tests to give us some amount of
test coverage. Later, because there was no way to add custom C tests,
#522 added a `smoke` directory which allowed this. But (a) each of these
test suites was built and run separately and (b) it was unclear how to
add more tests flexibly--some tests should only run on `*p2` targets or
`*-threads` targets, e.g.
This change reworks all of this so that all tests are built the same
way, in the same place. For downloaded tests like those from
`libc-test`, I chose to add "stub tests" that `#include` the original
version. This not only keeps all enabled tests in one place, it also
allows us to add "directives," C comments that the `Makefile` uses to
filter out tests for certain targets or add special compile, link or run
flags. These rudimentary scripts, along with other Bash logic I moved
out of the Makefile now live in the `scripts` directory.
Finally, all of this is explained more clearly in an updated
`README.md`. The hope with documenting this a bit better is that it
would be easier for drive-by contributors to be able to either dump in
new C tests for regressions they may find or enable more libc-tests. As
of my current count, we only enable 40/75 of libc-test's functional
tests, 0/228 math tests, 0/69 regression tests, and 0/79 API tests.
Though many of these may not apply to WASI programs, it would be nice to
explore how many more of these tests can be enabled to increase
wasi-libc's test coverage. This change should explain how to do that
and, with directives, make it possible to condition how the tests
compile and run.
This change is not meant to change any functionality, only move some
Bash-specific logic out of the `Makefile` into its own script:
`install-include-headers.sh`. This reduces the perceived complexity of
the `Makefile` but the complexity is still there, tucked away in this
script. This script also has the advantage that it can be run separately
if needed.
This commit comes after a few different attempts at building up
`Makefile` lists of headers to copy over along with the various
locations they must be copied from. It is certainly possible to do this,
but due to how we need to remove some headers from the list, it ends up
being easier to just `cp` and then `rm`, which this script retains.
This adds a `.stamp` file to the build directory that tracks when all
headers have been copied over to the sysroot `include` directory.
Previously, the `include_dirs` target was phony, which `make` will run
every time, regardless of whether it is needed. This was part of the
problem with all of libc being rebuilt at each `make` invocation.
https://github.com/WebAssembly/wasi-libc/pull/522 did not include the
necessary changes to the Makefile for libc_so build.
Additionally, updated CI to check `libc_so` build too to avoid future
breakage.
~~This patch series first starts with a number of commits stubbing out
functions in the existing `THREAD_model=posix` code. According to "The
Open Group Base Specifications Issue 7, 2018 edition", there are a
number of mandatory functions which have not been provided. There are
also some optional functions that have been partially provided in a
not-useful way (e.g. get but no set function). For these, I have chosen
to clean them up and remove the get functions for consistency.~~ EDIT:
These have been split off into separate PRs and merged.
The remainder of the patches then build up a stub implementation of
pthreads for `THREAD_MODEL=single`. I have done my best to try to make
sure that all functions are as conforming as possible (under the
assumption that another thread cannot ever be launched). This means that
objects such as mutexes and rwlocks actually do update their state and
will correctly fail when locks cannot be acquired.
When an inevitable deadlock occurs, I have chosen to return EDEADLK when
it has been explicitly listed as a permissible return value, and to
invoke `__builtin_trap` otherwise.
I have tested this by rebuilding libc++ with threads enabled and then
smoke-testing Clang/LLVM-on-WASI to make sure that it can compile a
simple program. I have not run any more-extensive conformance testing.
Fixes#501
Close https://github.com/WebAssembly/wasi-libc/issues/520
Add FTS implementation derived from musl-fts with a few modifications.
The compiled fts.o is included in the libc.a archive, and the fts.h
header is installed in the sysroot (`include/fts.h`).
* fts/musl-fts: Add a copy of the musl-fts sources with modifications.
* fts/patches: A set of patches to apply to the musl-fts sources.
* Upstream pull request: https://github.com/void-linux/musl-fts/pull/14
* fts/update-musl-fts.sh: A script to update the musl-fts sources with
the patches applied.
* fts/config.h: A configuration header included by the musl-fts sources.
* test/smoke: Add a test suite for wasi-libc specific features that
libc-test does not cover.
Hello,
While experimenting with the `wasm32-wasip2` target and CPython, I
discovered an issue with the `getaddrinfo()` implementation: it fails to
resolve the provided service into a port number, causing `sin_port` to
always be set to 0. This issue leads to failures in network-related
functions that rely on `getaddrinfo()`, such as Python's `urllib3`
library, which passes the result directly to `connect()`. This results
in connection attempts using a port value of 0, which naturally fails.
### Minimal example to reproduce the problem
```c
#include <arpa/inet.h>
#include <netdb.h>
#include <stdio.h>
int main(void) {
struct addrinfo *res = NULL;
getaddrinfo("google.com", "443", NULL, &res);
for (struct addrinfo *i = res; i != NULL; i = i->ai_next) {
char str[INET6_ADDRSTRLEN];
if (i->ai_addr->sa_family == AF_INET) {
struct sockaddr_in *p = (struct sockaddr_in *)i->ai_addr;
int port = ntohs(p->sin_port);
printf("%s: %i\n", inet_ntop(AF_INET, &p->sin_addr, str, sizeof(str)), port);
} else if (i->ai_addr->sa_family == AF_INET6) {
struct sockaddr_in6 *p = (struct sockaddr_in6 *)i->ai_addr;
int port = ntohs(p->sin6_port);
printf("%s: %i\n", inet_ntop(AF_INET6, &p->sin6_addr, str, sizeof(str)), port);
}
}
return 0;
}
```
```
$ /opt/wasi-sdk/bin/clang -target wasm32-wasip2 -o foo foo.c
$ wasmtime run -S allow-ip-name-lookup=y foo
216.58.211.238: 0
2a00:1450:4026:808::200e: 0
```
Expected output:
```
216.58.211.238: 443
2a00:1450:4026:808::200e: 443
```
### Root Cause
The root cause is that `getaddrinfo()` does not correctly translate the
provided service into a port number. As described in the `getaddrinfo()`
man [page](https://man7.org/linux/man-pages/man3/getaddrinfo.3.html),
the function should:
> service sets the port in each returned address structure. If
this argument is a service name (see
[services(5)](https://man7.org/linux/man-pages/man5/services.5.html)),
it is
translated to the corresponding port number. This argument can
also be specified as a decimal number, which is simply converted
to binary. If service is NULL, then the port number of the
returned socket addresses will be left uninitialized.
### Proposed Fix
This pull request addresses the issue by implementing the following
behavior for `getaddrinfo()`:
* If the service is `NULL`, the port number in the returned socket
addresses remains uninitialized.
* The value is converted to an integer and validated if the service is
numeric.
The PR does not currently add support for translating named services
into port numbers because `getservbyname()` has not been implemented. In
cases where a named service is provided, the `EAI_NONAME` error code is
returned.
This fixes errors like:
```
wasm-ld: error: /Volumes/PortableSSD/git/wasi-sdk/build/install/bin/../share/wasi-sysroot/lib/wasm32-wasi/llvm-lto/19.1.0-wasi-sdk/libsetjmp.a(rt.o): attempt to add bitcode file after LTO (__wasm_longjmp)
```
Note: Any symbol that the compiler might generate at bitcode compile
time either need to be unconditionally included at LTO time, or not
built as LTO. This is because LTO object files cannot be added to the
link after LTO time.
This is the next part of breaking up #518 into smaller PRs.
This is the rest of the commits which change the existing
`THREAD_MODEL=posix` functionality. It:
* _removes_ some functions which are optional and which were already
nonfunctional.
* (Continues to) establish a precedent of trying to remain as compatible
with Open Group specifications as possible, even when there are major
differences in WASI capabilities (i.e. thread cancellation)
Compared to the RFC PR, the `pthread_atfork` stub has been dropped as it
is now officially obsolete as of the very recent Issue 8 of the
specifications.
* Add LTO build option
An old PR which I used as a base:
https://github.com/WebAssembly/wasi-libc/pull/150
Co-Authored-by: Dan Gohman <dev@sunfishcode.online>
* Exclude atexit.c from LTO
This fixes a failure in wasi-sdk "make check".
("undefined symbol: __cxa_atexit" for ctors_dtors.c test)
* avoid specifying multiple lto flags for LIBC_NONLTO_OBJS
---------
Co-authored-by: Dan Gohman <dev@sunfishcode.online>
If thread-specific data is not set to 0 upon thread creation,
`__pthread_tsd_run_dtors` will end up running destructors passing
uninitialized memory as memory addresses, which can lead to memory
corruption.
This issue can be triggered when malloc() returns a memory address that
was freed before, as in that case memory is not zeroed out.
`tms_cutime` is the sum of the user times of child processes *excluding
the current process*. Since WASI doesn't provide a way to spawn a new
process, this value should always be 0.
* Makefile: separate the target to create empty dummy libraries
During develompment, it's sometimes convenient to have a separate target.
* ensure to create the destination directory
Per #501, this restores the pre-WASI-SDK-22 behavior of including a copy of
pthread.h for all targets -- not just the `*-threads` targets. This helps
minimize the number of preprocessor guards required to port existing POSIX
software to WASI. It also make it easier for projects using WASI-SDK versions
earlier than 22 to upgrade.
Note that this includes the pthread.h header, but no stub function definitions
to link against. We should probably provide the latter as well, but I'll leave
that for a separate PR.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
This implementation never changes actual memory protection, but it does
verify that the address range and flags are valid. The direct motivation
is fixing a linker error where LLVM links to `mprotect` in dead code.
This switches from `EXTRA_CFLAGS` to `CFLAGS` so when `EXTRA_CFLAGS` is
in the environment to specify `-g`, for example, it doesn't override
p2-handling logic.
Currently if `-g` is specified with build flags then wasi-libc fails to
build. I'm not certain why it shows up in object files, but it appears
to be a spurious error so this commit adds it to the list of variables
to ignore.
While malloc_stats and friends are disabled and unused for wasi-libc,
it's neater to be consistent.
Background: My colleagues for some reasons enabled malloc_stats
and asked me why it reports negative values.
Note: Depending __heap_base, init_top() adjusts the address for
alignment. I think the amount of this adjustment is reported as
"used" by malloc_stats. I don't bother to "fix" it.
Issue [#8392] in Wasmtime highlights how preopen registration can result
in a hang when compiled for a threaded environment. The problem is that
`internal_register_preopened_fd_unlocked` assumes it will not touch the
global lock because its caller, `__wasilibc_populate_preopens`, already
has taken the lock. Unfortunately, a refactoring in #408 (which
introduces `internal_register_preopened_fd_unlocked`) did not catch that
the `resize` function called internally also takes the global lock. This
change removes that locking in `resize` under the assumption that it
will only be called when the lock is already taken.
[#8392]: https://github.com/bytecodealliance/wasmtime/issues/8392
5bbf1ea8f1
added __wasm_multivalue__ and __wasm_reference_types__, and the latter
also makes libraries reference the undefined __indirect_function_table
symbol, which is provided by the linker.
This resets the preopens table to an uninitialized state, forcing it to be
reinitialized next time it is needed. This is useful when pre-initializing
using e.g. `wizer` or `component-init`. Such tools are capable of taking a
snapshot of a running application which may later be resumed on an unrelated
runtime (which may have its own, unrelated preopens).
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
This also includes stubs for `gethostbyname`, `gethostbyaddr`, etc. which were
necessary to get CPython to build. I believe it will be possible to implement
them all properly at some point, but don't have the bandwidth at the moment.
Finally, this includes a few fixes for issues I missed in earlier PRs that
surfaced when running the CPython `asyncio` test suite.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Co-authored-by: Dave Bakker <github@davebakker.io>
* Add libsetjmp.a/so
Add setjmp/longjump support based on Wasm EH proposal.
It's provided as a separate library (libsetjmp) from libc so that
runtimes w/o EH support can still load libc.so.
To use this setjmp/longjmp implementation, an application should
be compiled with `-mllvm -wasm-enable-sjlj` and linked with `-lsetjmp`.
(You need an LLVM with the change mentioned below.)
Also, you need a runtime with EH support to run such an application.
If you want to use the latest EH instructions, you can use
`binaryen --translate-eh-old-to-new` on your application.
Note: You don't need to translate libsetjmp.a/so to the new EH.
While LLVM currently produces bytecode for an old version of the EH
proposal, luckily for us, the bytecode used in this library (ie. the tag
definition and the "throw" instruction) is compatible with the latest
version of the proposal.
The runtime logic is basically copy-and-paste from:
https://github.com/yamt/garbage/tree/wasm-sjlj-alt2/wasm/longjmp
The corresponding LLVM change:
https://github.com/llvm/llvm-project/pull/84137
(Note: you need this change to build setjmp/longjmp using code.
otoh, you don't need this to build libsetjmp.)
A similar change for emscripten:
https://github.com/emscripten-core/emscripten/pull/21502
An older version of this PR, which doesn't require LLVM changes:
https://github.com/WebAssembly/wasi-libc/pull/467
Discussion:
https://docs.google.com/document/d/1ZvTPT36K5jjiedF8MCXbEmYjULJjI723aOAks1IdLLg/edit
An example to use the latest EH instructions:
```
clang -mllvm -wasm-enable-sjlj -o your_app.wasm your_app.c -lsetjmp
wasm-opt --translate-eh-old-to-new -o your_app.wasm your_app.wasm
toywasm --wasi your_app.wasm
```
Note: use toywasm built with `-DTOYWASM_ENABLE_WASM_EXCEPTION_HANDLING=ON`.
An example to use the older EH instructions, which LLVM currently produces:
```
clang -mllvm -wasm-enable-sjlj -o your_app.wasm your_app.c -lsetjmp
iwasm your_app.wasm
```
Note: use wasm-micro-runtime built with `-DWAMR_BUILD_EXCE_HANDLING=1`.
Note: as of writing this, only the classic interpreter supports EH.
* Make libsetjmp build optional
* CI: Disable libsetjmp for old LLVM
* libc-top-half/musl/include/setjmp.h: fix a rebase botch
This enables `wasm32-wasip2` support for `close`, `poll`, and `pselect`. I
cheated a bit for the latter by re-implementing `pselect` in terms of `poll` to
avoid having to implement wasip2 versions of both.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Co-authored-by: Dave Bakker <github@davebakker.io>
This adds `wasm32-wasip2` implementations of `shutdown`, `getsockopt`, and
`setsockopt`. It also extends the existing `ioctl` implementation to handle
both p1 and p2 file descriptors since we can't know until runtime which kind we
have. Once we've moved `wasm32-wasip2` fully to WASI 0.2 and remove the need for
the p1 adapter, we'll be able to switch to separate p1 and p2 `ioctl`
implementations.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Co-authored-by: Dave Bakker <github@davebakker.io>
This adds `bind`, `listen`, and `accept` implementations based on
`wasi-sockets` for the `wasm32-wasip2` target.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Co-authored-by: Dave Bakker <github@davebakker.io>
* implement basic TCP/UDP client support
This implements `socket`, `connect`, `recv`, `send`, etc. in terms of
`wasi-sockets` for the `wasm32-wasip2` target.
I've introduced a new public header file: `__wasi_snapshot.h`, which will define
a preprocessor symbol `__wasilibc_use_wasip2` if using the `wasm32-wasip2`
version of the header, in which case we provide features only available for that
target.
Co-authored-by: Dave Bakker <github@davebakker.io>
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* fix grammar in __wasi_snapshot.h comment
Co-authored-by: Dan Gohman <dev@sunfishcode.online>
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Co-authored-by: Dave Bakker <github@davebakker.io>
Co-authored-by: Dan Gohman <dev@sunfishcode.online>
* Start renaming preview1 to p1 and preview2 to p2
This is an initial start at renaming the "preview" terminology in WASI
targets to "pX". For example the `wasm32-wasi` target should transition
to `wasm32-wasip1`, `wasm32-wasi-preview2` should transition to
`wasm32-wasip2`, and `wasm32-wasi-threads` should transition to
`wasm32-wasip1-threads`. This commit applies a few renames in the
`Makefile` such as:
* `WASI_SNAPSHOT` is now either "p1" or "p2"
* The default p2 target triple is now `wasm32-wasip2` instead of
`wasm32-wasi-preview2` (in the hopes that it's early enough to change
the default).
* Bindings for WASIp2 were renamed from "preview2" terminology to "wasip2".
* The expected-defines files are renamed and the logic of which
expectation was used has been updated slightly.
With this commit the intention is that non-preview2 defaults do not
change. For example the default build still produces a `wasm32-wasi`
sysroot. If `TARGET_TRIPLE=wasm32-wasip1` is passed, however, then that
sysroot is produced instead. Similarly a `THREAD_MODEL=posix` build
produces a `wasm32-wasi-threads` sysroot target but you can now also
pass `TARGET_TRIPLE=wasm32-wasip1-threads` to rename the sysroot.
My hope is to integrate this into the wasi-sdk repository and build a
dual sysroot for these new targets for a release or two so both are
supported and then in the future the defaults can be switched away from
`wasm32-wasi` to `wasm32-wasip1` as built-by-default.
* Update builds in CI
* Update test workflow
* Fix test for wasm32-wasip1-threads
* Make github actions rules a bit more readable
* add descriptor table for mapping fds to handles
This introduces `descriptor_table.h` and `descriptor_table.c`, providing a
global hashtable for tracking `wasi-libc`-managed file descriptors.
WASI Preview 2 has no notion of file descriptors and instead uses unforgeable
resource handles. Moreover, there's not necessarily a one-to-one correspondence
between POSIX file descriptors and resource handles (e.g. a TCP connection may
require separate handles for reading, writing, and polling the same connection).
We use this table to map each POSIX descriptor to a set of one or more handles
and any extra state which libc needs to track.
Note that we've added `descriptor_table.h` to the
libc-bottom-half/headers/public/wasi directory, making it part of the public
API. The intention is to give applications access to the mapping, enabling them
to convert descriptors to handles and vice-versa should they need to
interoperate with both libc and WASI directly.
Co-authored-by: Dave Bakker <github@davebakker.io>
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* add dummy fields to otherwise empty structs
The C standard doesn't allow empty structs. Clang doesn't currently complain,
but we might as well stick to the spec in case it becomes more strict in the
future.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* move descriptor_table.h to headers/private
We're not yet ready to commit to making this API public, so we'll make it
private for now.
I've also expanded a comment in descriptor_table.c to explain the current ABI
for resource handles.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* re-run clang-format to fix indentation
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Co-authored-by: Dave Bakker <github@davebakker.io>
This file adds a custom section to each core module linked with wasi-libc. That
custom section contains component type information needed by e.g. `wasm-tools
component new` to generate a component from the module. It will be required
once we start using any part of WASI 0.2.0 directly (vs. via a Preview 1
adapter). In addition, it allows developers to `#include <wasi/preview2.h>` in
their code and make use of those APIs directly even if wasi-libc is not using
them yet.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* make the Makefiles a bit more robust
- Escape "." character in `sed` regex
- Ensure that %.wasm target fails cleanly (i.e. without generating the target file) if `wasm-tools` fails
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* split `component new` rule out of link rule for Preview 2
We now explicitly distinquish between core module files (%.core.wasm) and
component files (%.wasm), which helps avoid the trickery in my previous commit.
In order to test this properly, I needed to update the Wasmtime URL to point to
v17.0.0 instead of dev (which we needed to do anyway), and that in turn required
updating the bindings to use the final WASI 0.2.0 release.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
