Per https://github.com/WebAssembly/wasi-sdk/issues/373, LLVM's libc++ no longer
allows us to enable `<fstream>` and `<filesystem>` separately -- it's both or
neither. Consequently, we either need to patch libc++ to not use `statvfs`,
`chmod`, etc. or add stub functions for those features to `wasi-libc`. Since
we're planning to eventually support those features with WASI Preview 2 and
beyond, it makes sense to do the latter.
Note that since libc++ uses `DT_SOCK`, I've added a definition for it -- even
though WASI Preview 1 does not define it. No Preview 1 file will ever have that
type, so code that handles that type will never be reached, but defining it
allows us to avoid WASI-specific patches to libc++.
Related to `DT_SOCK`, I had to change the `S_IFIFO` value so it does not
conflict with `S_IFSOCK`, thereby avoiding ambiguity in `__wasilibc_iftodt`.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* add WASI Preview 2 bindings
This adds C bindings generated from the `wasi:cli/imports@0.2.0-rc-2023-12-05`
world, plus a makefile target to regenerate them from the WIT source files.
We'll use these bindings to call Preview 2 host functions when building for the
`wasm32-wasi-preview2` target.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* update to pre-release of `wit-bindgen` 0.17.0
This includes https://github.com/bytecodealliance/wit-bindgen/pull/804 (fix
broken indentation in generated code) and
https://github.com/bytecodealliance/wit-bindgen/pull/805 (support overriding
world name and adding a suffix to the component type custom section).
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* test all targets; update preview2 expected output files
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* build for `wasm32-wasi-threads` before testing it
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* move generated bindings per review feedback
Since these files aren't part of cloudlibc, no reason to put them under the
cloudlibc directory.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* move preview2.h to wasi directory
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Currently, this is identical to the `wasm32-wasi` in all but name. See #449 for
the next step, which is to incrementally add Preview 2 features,
e.g. `wasi-sockets`. Per the discussion in that PR, I've split the
`wasi-sysroot/include` directory into per-target directories. Eventually, we'll
want to build a separate sysroot for each target, but there's currently
uncertainty about how to configure the default sysroot for e.g. clang, so we're
not tackling that yet.
See also #447 for further details.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* add stubs for dlopen, dlsym, etc.
This adds weak exports for the POSIX `dlopen`, `dlsym`, `dlclose`, and `dlerror`
functions, allowing code which uses those features to compile. The
implementations are stubs which always fail since there is currently no official
standard for runtime dynamic linking.
Since the symbols are weak, they can be overriden with useful, runtime-specific
implementations, e.g. based on host functions or statically-generated tables
(see https://github.com/dicej/component-linking-demo for an example of the
latter).
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* move `dlopen` stubs out of libc and into libdl
Per review feedback, it's easier to simply replace libdl.so with a working
implementation at runtime than it is to override a handful of symbols in libc.
Note that I've both added libdl.so and replaced the empty libdl.a we were
previously creating with one that contains the stubs. I'm thinking we might as
well be consistent about what symbols the .so and the .a contain. Otherwise,
e.g. the CPython build gets confused when the dlfcn.h says `dlopen` etc. exist
but libdl.a is empty.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* customize dlfcn.h for WASI
For WASI, we use flag values which match MacOS rather than musl. This gives
`RTLD_LOCAL` a non-zero value, avoiding ambiguity and allowing us to defer the
decision of whether `RTLD_LOCAL` or `RTLD_GLOBAL` should be the default when
neither is specified.
We also avoid declaring `dladdr`, `dlinfo`, and friends on WASI since they are
neither supported nor stubbed at this time.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* use musl's RTLD_* flags except for RTLD_LOCAL
This minimizes the divergence from upstream while still giving us the
flexibility to choose a default value later.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* use `NULL` instead of `0` for null pointers
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
as we don't actually support thread cancellation.
note: currently we don't build pthread_cancel.c either. this commit
just disables it in our header too to make users notice that it isn't
provided a bit earlier.
should we disable other cancellation related functions like
pthread_testcancel? maybe. but they are harmless to ignore.
Per https://reviews.llvm.org/D156205 (which we're planning to backport to LLVM
17 and pull into `wasi-sdk`), we want to link crt1-reactor.o into libc.so so it
exports `_initialize` instead of `__wasm_call_ctors`.
* add `-nodefaultlibs` to libc.so link command
This ensures that `-lc` is not passed to `wasm-ld`.
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* add shared library support
This adds support for building WASI shared libraries per
https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md.
For the time being, the goal is to allow "pseudo-dynamic" linking using the
Component Model per
https://github.com/WebAssembly/component-model/blob/main/design/mvp/examples/SharedEverythingDynamicLinking.md.
This requires all libraries to be available when the component is created, but
still allows runtime symbol resolution via `dlopen`/`dlsym` backed by a static
lookup table. This is sufficient to support Python native extensions, for
example. A complete demo using `wit-component` is available at
https://github.com/dicej/component-linking-demo.
This commit adds support for building `libc.so`, `libc++.so`, and `libc++abi.so`
alongside their static counterparts.
Notes:
- I had to refactor `errno` support a bit to avoid a spurious `_ZTH5errno` (AKA "thread-local initialization routine for errno") import in `libc++.so`.
- Long double print and scan are included by default in `libc.so` rather than in a separate library.
- `__main_argc_argv` is now a weak symbol since it's not relevant for reactors.
- `dlopen`/`dlsym` rely on a lookup table provided by the "dynamic" linker via `__wasm_set_libraries`. Not all flags are supported yet, and unrecognized flags will result in an error.
- This requires https://reviews.llvm.org/D153293, which we will need to backport to LLVM 16 until 17 is released. I'll open a `wasi-sdk` PR with that change and various Makefile tweaks to support shared libraries.
- `libc.so` is temporarily disabled for the `wasi-threads` build until someone can make `wasi_thread_start.s` position-independent.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
build `-fPIC` .o files separately from non-`-fPIC` ones
This allows us to build both libc.so and libc.a without incurring indirection
penalties in the latter.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
only build libc.so when explicitly requested
Shared library support in LLVM for non-Emscripten Wasm targets will be added in
version 17, which has not yet been released, so we should not attempt to build
libc.so by default (at least not yet).
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
remove dl.c
I'll open a separate PR for this later.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
update `check-symbols` files
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* generate separate .so files for emulated features
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* revert errno changes in favor of a smaller change
@yamt pointed out there's an easier way to address the `_ZTH5errno` issue I
described in an earlier commit: use `_Thread_local` for both C and C++. This
gives us a simpler ABI and avoids needing to import a thread-local initializer
for `errno` in libc++.so.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* remove redundant `$(OBJDIR)/%.long-double.pic.o` rule in Makefile
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* consolidate libwasi-emulated-*.so into a single library
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* add comment explaining use of `--whole-archive`
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* Revert "remove redundant `$(OBJDIR)/%.long-double.pic.o` rule in Makefile"
This reverts commit dbe2cb10541dd27e4e0ed71d30ce304b9c9133d6.
* move `__main_void` from __main_void.c to crt1-command.c
This and `__main_argc_argv` are only relevant for commands (not reactors), so it
makes sense to scope them accordingly. In addition, the latter was being
imported from libc.so, forcing applications to provide it even if it wasn't
relevant.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* Revert "consolidate libwasi-emulated-*.so into a single library"
This reverts commit c6518223a49f60e4bb254a3e77a411fdade18df2.
* build crt1-*.o with `-fPIC`
This ensures they can be used in a PIE or PIC context.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* ignore `__memory_base` when checking undefined symbols
Whether this symbol appears varies between LLVM versions.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* Revert "move `__main_void` from __main_void.c to crt1-command.c"
This reverts commit f3038354610b7eb18bfd39092a2ccc3b72842dc4.
* add explanatory comments to __main_void.c
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* add `__wasilibc_unmodified_upstream` and comment to `__lctrans_cur`
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
the robust mutex logic in musl seems to assume that
the bit 29 of TIDs is always zero for some reasons.
from https://git.musl-libc.org/cgit/musl/commit/?id=099b89d3840c30d7dd962e18668c2e6d39f0c626
> note that the kernel ABI also reserves bit 29
> not to appear in any tid,
i'm not sure if the assumption is true or not, given that
FUTEX_TID_MASK is 0x3fffffff.
anyway, when using non-default type of mutex like recursive mutex,
it causes problems as we actually use TID 0x3fffffff for the main thread.
as we don't support robust mutex anyway, this commit simply
comments out the problematic condition.
fixes: https://github.com/bytecodealliance/wasm-micro-runtime/issues/2466
Given there are already AF_* definitions, and they are (now) essentially
synonyms, we add those definitions to enable compilation of code that
already use PF_* macros.
In a multi-threaded execution we need to make sure that only exactly one
thread initializes malloc. The function try_init_allocator() can't
easily be made thread-safe, so just move the call to
try_init_allocator() inside the block that holds the lock.
This commit effectively drops the support of older wasm-ld. (LLVM <15.0.7).
We have two relevant use cases:
* `memory.grow` use outside of malloc
(eg. used by polyfill preview1 binaries)
* `--init-memory` to somehow preallocate heap
(eg. avoid dynamic allocations, especially on small environments)
While https://github.com/WebAssembly/wasi-libc/pull/377
fixed the former, it broke the latter if you are using
an older LLVM, which doesn't provide the `__heap_end` symbol,
to link your module.
As we couldn't come up with a solution which satisfies all parties,
this commit simply makes it require new enough LLVM which provides
`__heap_end`. After all, a link-time failure is more friendly to users
than failing later in a subtle way.
This changes the front-page documentation to:
- use `wasi-libc` instead of "WASI Libc"
- explain how to build the pthreads-enabled `wasm32-wasi-threads` target
* Fix a use-after-free bug for detached threads
the issue was pointed out by @alexcrichton in
https://github.com/WebAssembly/wasi-libc/issues/405
* Rename map_base_lazy_free_queue as it only keeps a single item
Also, align the comment style with musl.
* Convert preopen initialization to be lazy.
Insteead of eagerly initializing the preopens in a static constructor,
perform preopen initialization the first time it's needed, or before a
close or a renumber which might disrupt the file descriptor space.
And, use a weak symbol with a stub function for use by `close` or
`fd_renumber`, we that they can trigger preopen initialization only
if it's actually needed.
This way, if a program doesn't contain any calls to any function that
needs preopens, it can avoid linking in the preopen initialization code.
And if it contains calls but doesn't execute them at runtime, it can
avoid executing the preopen intiailization code.
A downside here is that this may cause problems for users that call
`__wasi_fd_close` or `__wasi_fd_renumber` directly and close over
overwrite some preopens before libc has a chance to scan them. To
partially address this, this PR does add a declaration for
`__wasilibc_populate_preopens` to <wasi/libc.h> so that users can call
it manually if they need to.
* Fix calling `internal_register_preopened_fd` with the lock held.
Factor out the lock acquisition from the implementation of
`internal_register_preopened_fd` so that we can call it from
`__wasilibc_populate_preopens` with the lock held.
This commit fixes the ability to build `wasi-libc` with `-g` options and
possibly without `-O2` options as well. I've found this useful when
debugging issues as historically that the build fails when `-g` is
passed or optimizations are removed due to the checks against these
expectation files. This commit adds more filters to the list of macros
to ensure that optimization/debug related ones are all removed from the
expectation lists.
This makes the output of the build a lot more concise and easy to read.
The only real change here is to build each of the crt1 startup files
individually instead to trying to build them all in a single clang
invocation (that latter doesn't allow for -o to be specified which is
a pretty severe limitation, so its best avoided anyway).
It also reduces the size of the `ar` command line for libc itself from
78017 to 43609 (on my machine), which sadly is still tool long for win32
I believe.
Calling _initialize multiple times is undefined behavior, since the
ctors are not guaranteed to be idempotent. We should have this safety
check which is similar to #329.
* Change `wasm32-wasi-pthread` to `wasm32-wasi-threads`
After some thought, I think that we should rename the `THREAD_MODEL=posix` build to avoid confusion. Why? Though in this project the use of this target does involve pthreads, it will not be so in other standard libraries or languages (see, e.g., https://github.com/rust-lang/compiler-team/issues/574). I think it would be preferable to emphasize the "threads" Wasm-level proposal and the "wasi-threads" proposal rather than the specific details of which threading API is being exposed.
* fix: rename the `expected` output directory as well
* Don't use sbrk(0) to determine the initial heap size
This commit changes the `try_init_allocator` function as part of
dlmalloc to not use `sbrk(0)` to determine the initial heap size. The
purpose of this function is to use the extra memory at the end of linear
memory for the initial allocation heap before `memory.grow` is used to
allocate more memory. To learn the extent of this region the code
previously would use `sbrk(0)` to find the current size of linear
memory. This does not work, however, when other systems have called
`memory.grow` before this function is called. For example if another
allocator is used or if another component of a wasm binary grows memory
for its own purposes then that memory will be incorrectly claimed to be
owned by dlmalloc.
Instead this commit rounds up the `__heap_base` address to the nearest
page size, since that must be allocatable. Otherwise anything above this
rounded address is assumed to be used by something else, even if it's
addressable.
* Use `__heap_end` if defined
* Move mstate initialization earlier
* Implement the critical part of wasi_thread_start in asm
It's fragile to set up the critical part of C environment in C.
* Specify --target for asm files as well
* wasi_thread_start: Move __tls_base initialization to asm as well
To make it easier to create a sysroot with both triples.
Eg.
```
make -j4 CC=/opt/wasi-sdk-16.0/bin/clang
make -j4 CC=/opt/wasi-sdk-16.0/bin/clang THREAD_MODEL=posix
```
