This commit updates the test runner to automatically wait for
subtests to finish. This makes the experience more consistent
with suites and removes the need to await anything.
PR-URL: https://github.com/nodejs/node/pull/56664
Fixes: https://github.com/nodejs/node/issues/51292
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Pietro Marchini <pietro.marchini94@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Raz Luvaton <rluvaton@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
This commit adds a waitFor() method to the TestContext class in
the test runner. As the name implies, this method allows tests to
more easily wait for things to happen.
PR-URL: https://github.com/nodejs/node/pull/56595
Reviewed-By: Pietro Marchini <pietro.marchini94@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
This commit adds a t.assert.fileSnapshot() API to the test runner.
This is similar to how snapshot tests work in core, as well as
userland options such as toMatchFileSnapshot().
PR-URL: https://github.com/nodejs/node/pull/56459
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
This commit adds a top level assert.register() API to the test
runner. This function allows users to define their own custom
assertion functions on the TestContext.
Fixes: https://github.com/nodejs/node/issues/52033
PR-URL: https://github.com/nodejs/node/pull/56434
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Pietro Marchini <pietro.marchini94@gmail.com>
This commit refactors the buildSuite Promise logic in the Suite
constructor to use an async function instead of creating an
awkward primordial-based Promise chain.
PR-URL: https://github.com/nodejs/node/pull/55958
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Pietro Marchini <pietro.marchini94@gmail.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
This commit removes some asynchronous logic from the runHook()
method and replaces ArrayPrototypeReduce() with a for loop.
PR-URL: https://github.com/nodejs/node/pull/55963
Reviewed-By: Pietro Marchini <pietro.marchini94@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This commit marks the test runner's snapshot testing API as
stable.
PR-URL: https://github.com/nodejs/node/pull/55897
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Pietro Marchini <pietro.marchini94@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
PR-URL: https://github.com/nodejs/node/pull/55143
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
This commit adds a new 'test:summary' event to the test runner's
reporting interface. This new event serves two purposes:
- In the future, the test runner internals will no longer need to
change the process exit code. This may be important to run()
users. Unfortunately, this is a breaking change, so it needs to
be changed in a major version.
- The reporting interface now has a single event that can identify
passing or failing test runs.
Refs: https://github.com/nodejs/node/issues/53867
Refs: https://github.com/nodejs/node/issues/54812
PR-URL: https://github.com/nodejs/node/pull/54851
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit updates the way filtering is applied to tests and
suites. After this change, filters are applied just before the
test/suite is started. The results are the same, but this allows
us to eventually move away from the --test-only flag except
when process level isolation is used.
PR-URL: https://github.com/nodejs/node/pull/54832
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
This commit updates the way the test runner computes inherited
hooks. Instead of computing them when the Test/Suite is
constructed, they are now computed just prior to running the
Test/Suite. The reason is because when multiple test files are
run in the same process, it is possible for the inherited hooks
to change as more files are loaded.
PR-URL: https://github.com/nodejs/node/pull/53927
Fixes: https://github.com/nodejs/node/issues/51548
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit updates the test runner to wait for suites to finish
building before starting any tests. This is necessary when test
filtering is enabled, as suites may transition from filtered to
not filtered depending on what is inside of them.
Fixes: https://github.com/nodejs/node/issues/54084
Fixes: https://github.com/nodejs/node/issues/54154
PR-URL: https://github.com/nodejs/node/pull/54423
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com>
This commit updates the test runner's filter logic to handle
test suite failures during the build phase. Prior to this commit,
these suites were silently filtered.
PR-URL: https://github.com/nodejs/node/pull/54387
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
The lib/internal/test_runner/test.js should not use the
parseCommandLine() function. This commit refactors the code to
avoid doing so.
PR-URL: https://github.com/nodejs/node/pull/54353
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit makes hook creation more consistent by always
passing in a reference to the test that owns the hook. It also
removes some unnecessary validation on internal API.
PR-URL: https://github.com/nodejs/node/pull/54353
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit initializes the root harness object before the root
test and passes the harness as an option to the root test
constructor. This commit also attaches the global configuration
to the harness. This will allow the parseCommandLine() call in
test.js to be removed, as those values are now available via
the root test.
PR-URL: https://github.com/nodejs/node/pull/54353
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit updates parseCommandLine() to use the names supported
by run(). This removes some unnecessary renaming code, and allows
node:test and run() to more easily share code.
PR-URL: https://github.com/nodejs/node/pull/54353
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
If a test is run, but aborted, any after hooks should still be
run, as they may need to perform cleanup.
PR-URL: https://github.com/nodejs/node/pull/54151
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Raz Luvaton <rluvaton@gmail.com>
This commit refactors some of the internals of the test runner.
PR-URL: https://github.com/nodejs/node/pull/53921
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
This commit refactors the internals of snapshot tests to get the
name of the test file from the test context instead of passing
it to the SnapshotManager constructor. This is prep work for
supporting running test files in the test runner process.
PR-URL: https://github.com/nodejs/node/pull/53853
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
This commit adds a filePath getter to the TestContext and
SuiteContext classes. This allows a context to be mapped back to
the original test file that created it, even if it was imported
from another file. This is useful for mapping features like test
snapshots to the correct test file. This is also prep work for
supporting running test files in the test runner process.
PR-URL: https://github.com/nodejs/node/pull/53853
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
The type of the argument `positive` was declared as `boolean|number`,
which is misleading because the function treats it as a boolean only.
Some call sites even passed numbers, specifically, either `0` or `1`,
which happen to work as expected because they are interpreted as `false`
and `true`, respectively. However, passing `2` would silently lead to
unexpected behavior. Thus, strictly make the argument a boolean.
PR-URL: https://github.com/nodejs/node/pull/53307
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Zeyu "Alex" Yang <himself65@outlook.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This commit adds a t.assert.snapshot() method that implements
snapshot testing. Serialization uses JSON.stringify() by default,
but users can configure the serialization to meet their needs.
PR-URL: https://github.com/nodejs/node/pull/53169
Fixes: https://github.com/nodejs/node/issues/48260
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
This commit adds a fullName getter to the TestContext and
SuiteContext classes. This is similar to the existing name getter,
but also includes the name of all ancestor tests/suites.
PR-URL: https://github.com/nodejs/node/pull/53169
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Co-authored-by: Jacob Smith <jacob@frende.me>
The node:assert module contains several top level APIs that do
not make sense to expose as methods on t.assert. Examples include
AssertionError and CallTracker. This commit removes such APIs from
t.assert.
Refs: https://github.com/nodejs/node/pull/52860
PR-URL: https://github.com/nodejs/node/pull/53049
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/52625
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
This commit updates filteredRun() to call postRun() after a
microtask instead of synchronously. Currently, if approximately
1,545 subtests are filtered, enough synchronous calls can be
made to cause a call stack exceeded exception.
PR-URL: https://github.com/nodejs/node/pull/52488
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
This commit moves the end of work check from postRun() to
finalize(). The reasoning is that finalize() is guaranteed to
run in the order that the tests are defined, while postRun() is
not. This makes the check a little simpler.
PR-URL: https://github.com/nodejs/node/pull/52488
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
This commit updates the logic that checks for the end of the
test run. Prior to this change, it was possible for root.run() to
be called multiple times because of the way pending subtests
were tracked. The extra calls to root.run() were harmless, but
could trigger an EventEmitter leak warning due to 'abort'
listeners being created.
PR-URL: https://github.com/nodejs/node/pull/52326
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
This commit updates the test runner to suppress any output for
filtered tests. Filtered tests no longer generate reporter events,
and the unfiltered tests are renumbered in the output as though
the filtered tests were not present. Skipped tests that are not
filtered are still included in the output.
This change is particularly useful when filtering a large number
of tests, as the previously displayed skip output could be
distracting.
Fixes: https://github.com/nodejs/node/issues/51383
PR-URL: https://github.com/nodejs/node/pull/52221
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
This commit simplifies the logic for tracking test end time.
The end time is now only set in postRun(), which every test
runs when it ends.
PR-URL: https://github.com/nodejs/node/pull/52182
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
This commit simplifies the logic for tracking test start time.
The start time is now set only when a test/suite begins running.
If the test/suite never runs, a fallback is provided in postRun().
PR-URL: https://github.com/nodejs/node/pull/52182
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
