This commit updates the test() and suite() APIs to no longer
return a Promise.
Fixes: https://github.com/nodejs/node/issues/51292
PR-URL: https://github.com/nodejs/node/pull/56664
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Pietro Marchini <pietro.marchini94@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Raz Luvaton <rluvaton@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
This commit updates the test runner to automatically wait for
subtests to finish. This makes the experience more consistent
with suites and removes the need to await anything.
PR-URL: https://github.com/nodejs/node/pull/56664
Fixes: https://github.com/nodejs/node/issues/51292
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Pietro Marchini <pietro.marchini94@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Raz Luvaton <rluvaton@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
This commit adds a new 'test:summary' event to the test runner's
reporting interface. This new event serves two purposes:
- In the future, the test runner internals will no longer need to
change the process exit code. This may be important to run()
users. Unfortunately, this is a breaking change, so it needs to
be changed in a major version.
- The reporting interface now has a single event that can identify
passing or failing test runs.
Refs: https://github.com/nodejs/node/issues/53867
Refs: https://github.com/nodejs/node/issues/54812
PR-URL: https://github.com/nodejs/node/pull/54851
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit updates the way the test runner processes 'only'
tests when node:test files are run without the --test CLI.
This is a breaking change.
PR-URL: https://github.com/nodejs/node/pull/54881
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit updates the way the test runner processes 'only'
tests when process-based test isolation is disabled. The
--test-only flag is no longer necessary in this scenario. The
test runner will automatically detect 'only' tests and apply the
appropriate filtering.
PR-URL: https://github.com/nodejs/node/pull/54832
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
This commit updates the way filtering is applied to tests and
suites. After this change, filters are applied just before the
test/suite is started. The results are the same, but this allows
us to eventually move away from the --test-only flag except
when process level isolation is used.
PR-URL: https://github.com/nodejs/node/pull/54832
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
The test runner's code coverage leaves old coverage data in the
temp directory. This commit updates the cleanup logic to:
- Stop code collection. Otherwise V8 would write collection data
again when the process exits.
- Remove the temp directory containing the coverage data.
- Attempt to clean up the coverage data even if parsing the
data resulted in an error.
With this change, I no longer see any coverage data left behind
in the system temp directory.
Refs: https://github.com/nodejs/build/issues/3864
Refs: https://github.com/nodejs/build/issues/3887
PR-URL: https://github.com/nodejs/node/pull/54856
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: James M Snell <jasnell@gmail.com>
This commit introduces a new --experimental-test-isolation flag
that, when set to 'none', causes the test runner to execute all
tests in the same process. By default, this is the main test
runner process, but if watch mode is enabled, it spawns a separate
process that runs all of the tests.
The default value of the new flag is 'process', which uses the
existing behavior of running each test file in its own child
process.
It is worth noting that when the isolation mode is 'none', globals
and all other top level logic (such as top level before() and after()
hooks) is shared among all files.
Co-authored-by: Moshe Atlow <moshe@atlow.co.il>
PR-URL: https://github.com/nodejs/node/pull/53927
Fixes: https://github.com/nodejs/node/issues/51548
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit updates the test runner to wait for suites to finish
building before starting any tests. This is necessary when test
filtering is enabled, as suites may transition from filtered to
not filtered depending on what is inside of them.
Fixes: https://github.com/nodejs/node/issues/54084
Fixes: https://github.com/nodejs/node/issues/54154
PR-URL: https://github.com/nodejs/node/pull/54423
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com>
Now that parseCommandLine() returns run() compatible arguments,
it makes sense to return setupTestReporters() as the setup()
argument to run(). This also removes another problematic use of
parseCommandLine() in setupTestReporters().
PR-URL: https://github.com/nodejs/node/pull/54353
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
The global configuration should already be known when
createTestTree() is called. This commit updates that function
to take the global configuration as an input.
PR-URL: https://github.com/nodejs/node/pull/54353
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit initializes the root harness object before the root
test and passes the harness as an option to the root test
constructor. This commit also attaches the global configuration
to the harness. This will allow the parseCommandLine() call in
test.js to be removed, as those values are now available via
the root test.
PR-URL: https://github.com/nodejs/node/pull/54353
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
The test runner bootstrap process awaits a Promise and then sets
a boolean flag. This commit consolidates the Promise and boolean
into a single value. This commit also ensures that the globalRoot
test is always assigned in createTestTree() in order to better
consolidate the CLI/run() and non-CLI configuration.
PR-URL: https://github.com/nodejs/node/pull/54013
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
This commit refactors some of the internals of the test runner.
PR-URL: https://github.com/nodejs/node/pull/53921
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
This commit adds a filePath getter to the TestContext and
SuiteContext classes. This allows a context to be mapped back to
the original test file that created it, even if it was imported
from another file. This is useful for mapping features like test
snapshots to the correct test file. This is also prep work for
supporting running test files in the test runner process.
PR-URL: https://github.com/nodejs/node/pull/53853
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
This commit adds a t.assert.snapshot() method that implements
snapshot testing. Serialization uses JSON.stringify() by default,
but users can configure the serialization to meet their needs.
PR-URL: https://github.com/nodejs/node/pull/53169
Fixes: https://github.com/nodejs/node/issues/48260
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Prior to this commit, each top level test awaited the same
global promise for setting up test reporters. This commit
updates the logic to only await the promise the first time
it is encountered.
PR-URL: https://github.com/nodejs/node/pull/52185
Refs: https://github.com/nodejs/node/pull/47164
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
This commit updates the test harness to prevent top level
tests from executing immediately. This allows certain config
data, such as filtering options, to be discovered before running
the tests.
PR-URL: https://github.com/nodejs/node/pull/52092
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
This commit updates the test runner to allow a forced exit once
all known tests have finished running.
Fixes: https://github.com/nodejs/node/issues/49925
PR-URL: https://github.com/nodejs/node/pull/52038
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Raz Luvaton <rluvaton@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
This commit updates the test runner's uncaughtException handler
to abort tests instead of assuming they finished running.
Fixes: https://github.com/nodejs/node/issues/51381
PR-URL: https://github.com/nodejs/node/pull/51996
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
This commit moves the global after() hook execution from the
'beforeExit' event to the point where all tests have finished
running. This gives the global after() a chance to clean up
handles that would otherwise prevent the 'beforeExit' event
from being emitted.
PR-URL: https://github.com/nodejs/node/pull/49059
Fixes: https://github.com/nodejs/node/issues/49056
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
This commit adds each test's line and column number to the reporter
output. This will aid in debugging test suite failures when error
stacks are not helpful, test suites are large, or tests have the
same name. This data is also exposed on the spec reporter.
This commit also replaces the filename that was previously being
reported, with the filename where the test actually exists. These
are normally correct, but could be wrong if tests were run from
a file other than the user's entrypoint.
PR-URL: https://github.com/nodejs/node/pull/48975
Fixes: https://github.com/nodejs/node/issues/48457
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
This commit moves code coverage collection from the test
harness exit handler to the postRun() function of the root
test.
This is necessary preparatory work for supporting
code coverage with --test. The reason is that --test is
implemented on top of run(), and that function calls the root
test's postRun() function, which outputs the test summary. This
happens before the harness exit handler.
PR-URL: https://github.com/nodejs/node/pull/47651
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
The wasRootSetup Set in the test harness appears to be
redundant, since the startTime field can be used interchangeably.
This commit removes wasRootSetup.
PR-URL: https://github.com/nodejs/node/pull/46961
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
This commit addresses a previously untested branch of the code.
It is possible when using the test runner that an error occurs
outside of a test. In this case, the test runner would simply
rethrow the error. This commit updates the logic to handle the
error in the same fashion as other uncaughtExceptions.
PR-URL: https://github.com/nodejs/node/pull/46962
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
This commit updates the test harness to re-throw uncaught errors
if bootstrapping has not completed. This updates the existing
logic which tried to detect a specific error code.
PR-URL: https://github.com/nodejs/node/pull/46962
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
This commit updates the test harness and root test to track
when bootstrapping has completed.
PR-URL: https://github.com/nodejs/node/pull/46962
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
This commit replaces the 'coverage' reference inside of the Test
class with a more generic harness reference which includes
coverage. This will let the root test more easily track process
level state such as code coverage, uncaughtException handlers,
and the state of bootstrapping.
PR-URL: https://github.com/nodejs/node/pull/46962
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
PR-URL: https://github.com/nodejs/node/pull/46889
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
