The (h|readableH|writableH)ighWaterMark options should only permit
positive numbers and zero.
PR-URL: https://github.com/nodejs/node/pull/18098
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Adds support for Symbol.asyncIterator into the Readable class.
The stream is destroyed when the loop terminates with break or throw.
Fixes: https://github.com/nodejs/node/issues/15709
PR-URL: https://github.com/nodejs/node/pull/17755
Fixes: https://github.com/nodejs/node/issues/15709
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Emit 'readable' always in the next tick, resulting in a single
call to _read() per microtick. This removes the need for the
user to implement buffering if they wanted to call this.push()
multiple times in an asynchronous fashion, as this.push() triggers
this._read() call.
PR-URL: https://github.com/nodejs/node/pull/17979
Fixes: https://github.com/nodejs/node/issues/3203
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
As part of the readableState/writableState mega issue #445, this
removes all of the references to .length on those properties and
replaces them with a readableLength and writableLength getter.
See: https://github.com/nodejs/node/issues/445
PR-URL: https://github.com/nodejs/node/pull/12857
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This adds computed properties to readable and writable streams to
allow access to the readable buffer, the writable buffer, and flow
state without accessing the readable or writable state.
These are the only uses of readable and writable state in the docs
so adding these work arounds allows them to be removed from the docs.
This also updates net, http_client and http_server to use the new
methods instead of manipulating readable and writable state directly.
See: https://github.com/nodejs/node/issues/445
PR-URL: https://github.com/nodejs/node/pull/12855
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
The `expected` argument of `ERR_INVALID_ARG_TYPE` can be an
array, which is better than a single string.
PR-URL: https://github.com/nodejs/node/pull/17145
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Replaced _readableState.highWaterMark with a .readableHighWaterMark
getter and _writableState.highWaterMark with a .writableHighWaterMark
getter.
The getters are non-enumerable because they break some prototype
manipulation that happen in the ecosystem.
Ref: https://github.com/nodejs/node/issues/445.
PR-URL: https://github.com/nodejs/node/pull/12860
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Complete the migration to the new error system of _stream_readable
and _stream_writable. Adds the corresponding documentation.
PR-URL: https://github.com/nodejs/node/pull/16589
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
This change is to unify the declaration for constants into using
destructuring on the top-level-module scope, reducing some redundant
code.
PR-URL: https://github.com/nodejs/node/pull/16063
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
With the update to v8 6.1 this todo can now be addressed.
PR-URL: https://github.com/nodejs/node/pull/15667
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Benedikt Meurer <benedikt.meurer@gmail.com>
Reviewed-By: Yuta Hiroto <hello@about-hiroppy.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
This condition could never be met because all calling functions
guarded against this.
PR-URL: https://github.com/nodejs/node/pull/15665
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
This commits adds support for readableHighWaterMark and
writableHighWaterMark in Duplex stream, so that they can be set without
accessing the internal state.
Fixes: https://github.com/nodejs/node/issues/14555
PR-URL: https://github.com/nodejs/node/pull/14636
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
In anticipation of stricter linting for indentation issues, modify
ternary operators in lib that do not conform with the expected ESLint
settings.
PR-URL: https://github.com/nodejs/node/pull/14078
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
With the new Ignition+TurboFan pipeline, the instanceof fast-path can be
missed if the right-hand side needs a TDZ check, i.e. is const declared
on a surrounding scope. This doesn't apply to Node 8 at this point,
where it's at V8 5.8, but it applies as soon as V8 5.9 rolls. There's
work going on in Ignition (and TurboFan) to optimize those TDZ checks
properly, but those changes will land in V8 6.1, so might not end up in
Node 8.
One way to work-around this in Node core for now is to use var instead
of const for those right-hand sides for instanceof for now, especially
Buffer in case of streams. This is not beautiful, but proper ducktape.
Improves readable-bigread.js by ~23% with Node LKGR.
PR-URL: https://github.com/nodejs/node/pull/13403
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Calvin Metcalf <calvin.metcalf@gmail.com>
Adds destroy() and _destroy() methods to Readable, Writable, Duplex
and Transform. It also standardizes the behavior and the implementation
of destroy(), which has been inconsistent in userland and core.
This PR also updates all the subsystems of core to use the new
destroy().
PR-URL: https://github.com/nodejs/node/pull/12925
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Calvin Metcalf <calvin.metcalf@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Fix the uncommon situation when a readable stream is piped twice into
the same destination stream, and then unpiped once.
Previously, the `unpipe` event handlers weren’t able to tell whether
they were corresponding to the “right” conceptual pipe that was being
removed; this fixes this by adding a counter to the `unpipe` event
handler and only removing a single piping destination at most.
Fixes: https://github.com/nodejs/node/issues/12718
PR-URL: https://github.com/nodejs/node/pull/12746
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Delete invalid parameter.
Fixes: https://github.com/nodejs/node/issues/8153
PR-URL: https://github.com/nodejs/node/pull/12767
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: David Cai <davidcai1993@yahoo.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Currently when the destination emits an 'error', 'finish' or 'close'
event the pipe calls unpipe to emit 'unpipe' and trigger the clean up
of all it's listeners.
When the source emits an 'end' event without {end: false} it calls
end() on the destination leading it to emit a 'close', this will again
lead to the pipe calling unpipe. However the source emitting an 'end'
event along side {end: false} is the only time the cleanup gets ran
directly without unpipe being called. This fixes that so the 'unpipe'
event does get emitted and cleanup in turn gets ran by that event.
Fixes: https://github.com/nodejs/node/issues/11837
PR-URL: https://github.com/nodejs/node/pull/11876
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
It seems that the comment for sync flag of ReadableState is
copied from WritableState without modification.
PR-URL: https://github.com/nodejs/node/pull/11139
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
This is to be consistent with the other operators and helps
understanding the context when the code is grepped.
PR-URL: https://github.com/nodejs/node/pull/10213
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Teddy Katz <teddy.katz@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Since 2e568d9 there is a bug where unpiping a stream
from a readable stream that has `_readableState.pipesCount > 1`
will cause it to remove the first stream in the
`_.readableState.pipes` array no matter where in the list the
`dest` stream was.
This patch corrects that problem.
PR-URL: https://github.com/nodejs/node/pull/9171
Fixes: https://github.com/nodejs/node/issues/9170
Reviewed-By: Evan Lucas <evanlucas@me.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Myles Borins <myles.borins@gmail.com>
This is a known de-opt. It may not be 100% necessary in all cases but it
seems like a decent enough idea to avoid it.
PR-URL: https://github.com/nodejs/node/pull/8873
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: Ilkka Myller <ilkka.myller@nodefield.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Improve message when tranform._transform() method is not implemented
Improve error message when Readable._read() is not implemented
Remove extra word in err msg when Writable._write() when not implemented
Remove extra word in err msg when Transform._transform() when not implemented
PR-URL: https://github.com/nodejs/node/pull/8801
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ilkka Myller <ilkka.myller@nodefield.com>
This removes the cached check for EE.prototype.prependListener
because we can't have nice things. More specifically some
libraries will bundle their own event emitter implementation.
PR-URL: https://github.com/nodejs/node/pull/8018
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This change is in preparation for lint-enforced brace style.
PR-URL: https://github.com/nodejs/node/pull/7630
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: Michaël Zasso <mic.besace@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Roman Reiss <me@silverwind.io>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
Guard against the call to write() inside pipe's ondata pushing more data
back onto the Readable, thus causing ondata to be called again.
This is fine but results in awaitDrain being increased more than once.
The problem with that is when the destination does drain, only a single
'drain' event is emitted, so awaitDrain in this case will never reach
zero and we end up with a permanently paused stream.
Fixes: https://github.com/nodejs/node/issues/7278
PR-URL: https://github.com/nodejs/node/pull/7292
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
read() performance is improved most by switching from an array to
a linked list for storing buffered data. However, other changes that
also contribute include: making some hot functions inlinable, faster
read() argument checking, and misc code rearrangement to avoid
unnecessary code execution.
PR-URL: https://github.com/nodejs/node/pull/7077
Reviewed-By: Calvin Metcalf <calvin.metcalf@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reset the `readableState.awaitDrain` counter after manual calls to
`.resume()`.
What might happen otherwise is that a slow consumer at the end of the
pipe could end up stalling the piping in the following scenario:
1. The writable stream indicates that its buffer is full.
2. This leads the readable stream to `pause()` and increase its
`awaitDrain` counter, which will be decreased by the writable’s next
`drain` event.
3. Something calls `.resume()` manually.
4. The readable continues to pipe to the writable, but once again
the writable stream indicates that the buffer is full.
5. The `awaitDrain` counter is thus increased again, but since it has
now been increased twice for a single piping destination, the next
`drain` event will not be able to reset `awaitDrain` to zero.
6. The pipe is stalled and no data is passed along anymore.
The solution in this commit is to reset the `awaitDrain` counter to
zero when `resume()` is called.
Fixes: https://github.com/nodejs/node/issues/7159
PR-URL: https://github.com/nodejs/node/pull/7160
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
A handful of modules (including readable-streams) make
inappropriate use of the internal _events property. One
such use is to prepend an event listener to the front
of the array of listeners.
This adds EE.prototype.prependListener() and
EE.prototype.prependOnceListener() methods to add handlers
to the *front* of the listener array.
Doc update and test case is included.
Fixes: https://github.com/nodejs/node/issues/1817
PR-URL: https://github.com/nodejs/node/pull/6032
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Brian White <mscdex@mscdex.net>
In 68990948fe (https://github.com/nodejs/node/pull/2325),
the conditions for increasing `readableState.awaitDrain` when
writing to a piping destination returns false were changed so
that they could not actually be met, effectively leaving
`readableState.awaitDrain` with a constant value of 0.
This patch changes the conditions to testing whether the
stream for which `.write()` returned false is still a piping
destination, which was likely the intention of the original
patch.
Fixes: https://github.com/nodejs/node/issues/5820
Fixes: https://github.com/nodejs/node/issues/5257
PR-URL: https://github.com/nodejs/node/pull/6023
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Readable.resume() schedules the resume operation onto the next tick,
whereas pause() has immediate effect. This means that in a sequence
stream.resume();
stream.pause();
.. the 'pause' event will be triggered before the resume operation
is performed.
For process.stdin, we are relying on the 'pause' event to stop reading
on the underlying handle. This fix ensures that reads are started and
stopped in the same order as resume() and pause() are called.
PR-URL: https://github.com/nodejs/node/pull/5776
Reviewed-By: cjihrig - Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: jasnell - James M Snell <jasnell@gmail.com>
Several changes:
* Soft-Deprecate Buffer() constructors
* Add `Buffer.from()`, `Buffer.alloc()`, and `Buffer.allocUnsafe()`
* Add `--zero-fill-buffers` command line option
* Add byteOffset and length to `new Buffer(arrayBuffer)` constructor
* buffer.fill('') previously had no effect, now zero-fills
* Update the docs
PR-URL: https://github.com/nodejs/node/pull/4682
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Before this commit, it was possible to push a partial character
to a readable stream where it was decoded as an empty string and
then added to the internal buffer. This caused the stream to not
emit any data, even when the rest of the character bytes were pushed
separately, because of a non-zero length check of the first chunk in
the internal buffer.
Fixes: https://github.com/nodejs/node/issues/5223
PR-URL: https://github.com/nodejs/node/pull/5226
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
`lib/_stream_readable.js` contained three instances of `var`
declarations occurring twice in the same scope. Refactored to `const` or
`let` as appropriate.
PR-URL: https://github.com/nodejs/node/pull/4816
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
ReadableState has the resumeScheduled property that helps determine if
a stream should be resumed. It was not assigned in the constructor.
When stream.resume is called on a readable stream that is not flowing,
it is set to true. This changes the property map of the ReadableState
which can cause a deopt in onEofChunk and needMoreData.
PR-URL: https://github.com/nodejs/node/pull/4761
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Brian White <mscdex@mscdex.net>
Add a bytesRead property for readable is
useful in some use cases.
When user want know how many bytes read of
readable, need to caculate it in userland.
If encoding is specificed, get the value is
very slowly.
PR-URL: https://github.com/nodejs/node/pull/4372
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
As of 34b535f4c, test-child-process-flush-stdio was failing
on CentOS 5 systems in CI due to the change in stream state
checking in `child_process`. This commit fixes those failures
by making readable streams less eager in setting their readable
flag on EOF.
Fixes: https://github.com/nodejs/node/issues/4125
PR-URL: https://github.com/nodejs/node/pull/4141
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
These properties were initially used to determine stream status
back in node v0.8 and earlier. Since streams2 however, these
properties were *always* true, which can be misleading for
example if you are trying to immediately determine whether
a Writable stream is still writable or not (to avoid a "write after
end" exception).
PR-URL: https://github.com/nodejs/node/pull/4083
Reviewed-By: Chris Dickinson <christopher.s.dickinson@gmail.com>
This commit fixes some error messages that are not consistent with
some general rules which most of the error messages follow.
PR-URL: https://github.com/nodejs/node/pull/3374
Reviewed-By: Roman Reiss <me@silverwind.io>
Avoids doing a buffer.concat on the internal buffer
when that array has only a single thing in it.
Reviewed-By: Chris Dickinson <chris@neversaw.us>
Reviewed-By: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/3300
The `events` module already exports `EventEmitter` constructor function
So, we don't have to use `events.EventEmitter` to access it.
Refer: https://github.com/nodejs/node/pull/2896
PR-URL: https://github.com/nodejs/node/pull/2921
Reviewed-By: Roman Reiss <me@silverwind.io>
Reviewed-By: Michaël Zasso <mic.besace@gmail.com>
Now parts of our public and public-ish APIs fall back to old-style
listenerCount() if the emitter does not have a listenerCount function.
Fixes: https://github.com/nodejs/node/issues/2655
Refs: 8f58fb92ff
PR-URL: https://github.com/nodejs/node/pull/2661
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
roundUpToNextPowerOf2() does more than just rounding up to the next
power of two. Rename it to computeNewHighWaterMark().
PR-URL: https://github.com/nodejs/node/pull/2479
Reviewed-By: Chris Dickinson <christopher.s.dickinson@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Don't iterate over all 32 bits, use some hacker's delight bit twiddling
to compute the next power of two.
The logic can be reduced to `n = 1 << 32 - Math.clz32(n)` but then it
can't easily be backported to v2.x; Math.clz32() was added in V8 4.3.
PR-URL: https://github.com/nodejs/node/pull/2479
Reviewed-By: Chris Dickinson <christopher.s.dickinson@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
The high watermark is capped at 8 MB, not 128 MB like the comment
in lib/_stream_readable.js said.
PR-URL: https://github.com/nodejs/node/pull/2479
Reviewed-By: Chris Dickinson <christopher.s.dickinson@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
As per the discussion in #734, this patch deprecates the usage of
`EventEmitter.listenerCount` static function in the docs, and introduces
the `listenerCount` function in the prototype of `EventEmitter` itself.
PR-URL: https://github.com/nodejs/node/pull/2349
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Brian White <mscdex@mscdex.net>
Many of the util.is*() methods used to check data types
simply compare against a single value or the result of
typeof. This commit replaces calls to these methods with
equivalent checks. This commit does not touch calls to the
more complex methods (isRegExp(), isDate(), etc.).
Fixes: https://github.com/iojs/io.js/issues/607
PR-URL: https://github.com/iojs/io.js/pull/647
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
This commit replaces a number of var statements throughout
the lib code with const statements.
PR-URL: https://github.com/iojs/io.js/pull/541
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
The copyright and license notice is already in the LICENSE file. There
is no justifiable reason to also require that it be included in every
file, since the individual files are not individually distributed except
as part of the entire package.
Turn on strict mode for the files in the lib/ directory. It helps
catch bugs and can have a positive effect on performance.
PR-URL: https://github.com/node-forward/node/pull/64
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
net Sockets were calling read(0) to start reading, without
checking to see if they were paused first. This would result
in paused Socket objects keeping the event loop alive.
Fixes#8200
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
A streams1 stream will have its falsy values such as 0, false, or ""
eaten by the upgrade to streams2, even when objectMode is enabled.
Include test for said cases.
Reviewed-by: isaacs <i@izs.me>
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
A ReadableStream with a base64 StringDecoder backed by only
one or two bytes would fail to output its partial data before
ending. This fix adds a check to see if the `read` was triggered
by an internal `flow`, and if so, empties any remaining data.
fixes#7914.
Signed-off-by: Fedor Indutny <fedor@indutny.com>
Switch condition order to check for null before calling isNaN().
Also remove two unnecessary calls to isNaN() that are already
covered by calls to isFinite(). This commit targets v0.10, as
opposed to #7891, which targets master (suggested by
@bnoordhuis). Closes#7840.
Signed-off-by: Fedor Indutny <fedor@indutny.com>
Default highWaterMark is now set properly when using stream Duplex's
writableObjectMode and readableObjectMode options.
Added condition to the already existing split objectMode test to ensure
the highWaterMark is being set to the correct default value on both the
ReadableState and WritableState for readableObjectMode and
writableObjectMode.
Signed-off-by: Fedor Indutny <fedor@indutny.com>
The [Stream documentation for .push](http://nodejs.org/api/stream.html#stream_readable_push_chunk_encoding)
explicitly states multiple times that null is a special cased value
that indicates the end of a stream. It is confusing and undocumented
that undefined *also* ends the stream, even though in object mode
there is a distinct and important difference.
The docs for Object-Mode also explicitly mention null as the *only*
special cased value, making no mention of undefined.
Signed-off-by: Fedor Indutny <fedor@indutny.com>
This commit introduces `readableObjectMode` and
`writableObjectMode` options for Duplex streams.
This can be used mostly to make parsers and
serializers with Transform streams.
Also the docs section about stream state objects
is removed, because it is not relevant anymore.
The example from the section is remade to show
new options.
fixes#6284
Signed-off-by: Timothy J Fontaine <tjfontaine@gmail.com>
In this situation:
writable.on('error', handler);
readable.pipe(writable);
writable.removeListener('error', handler);
writable.emit('error', new Error('boom'));
there is actually no error handler, but it doesn't throw, because of the
fix for stream.once('error', handler), in 23d92ec.
Note that simply reverting that change is not valid either, because
otherwise this will emit twice, being handled the first time, and then
throwing the second:
writable.once('error', handler);
readable.pipe(writable);
writable.emit('error', new Error('boom'));
Fix this with a horrible hack to make the stream pipe onerror handler
added before any other userland handlers, so that our handler is not
affected by adding or removing any userland handlers.
Closes#6007.
When a stream is flowing, and not in the middle of a sync read, and
the read buffer currently has a length of 0, we can just emit a 'data'
event rather than push it onto the array, emit 'readable', and then
automatically call read().
As it happens, this is quite a frequent occurrence! Making this change
brings the HTTP benchmarks back into a good place after the removal of
the .ondata/.onend socket kludge methods.
If an error listener is added to a stream using once() before it is
piped, it is invoked and removed during pipe() but before pipe() sees it
which causes it to be emitted again.
Fixes#4155#4978
This prevents the following sort of thing from being confusing:
```javascript
stream.on('data', function() { console.error('got data'); });
stream.pause(); // stop reading
// turns out no data is available
stream.push(null);
// Hand the stream to someone else, who does stuff...
setTimeout(function() {
// too late! 'end' is already emitted!
stream.on('end', function() { console.error('got end'); });
});
```
With this change, the `end` event is not emitted until you call `read()`
*past* the EOF null. So, a paused stream will not swallow the `end`
event and emit it before you `resume()` the stream.
Closes#5860
In streams2, there is an "old mode" for compatibility. Once switched
into this mode, there is no going back.
With this change, there is a "flowing mode" and a "paused mode". If you
add a data listener, then this will start the flow of data. However,
hitting the `pause()` method will switch *back* into a non-flowing mode,
where the `read()` method will pull data out.
Every time `read()` returns a data chunk, it also emits a `data` event.
In this way, a passive data listener can be added, and the stream passed
off to some other reader, for use with progress bars and the like.
There is no API change beyond this added flexibility.
In some cases, the http CONNECT/Upgrade API is unshifting an empty
bodyHead buffer onto the socket.
Normally, stream.unshift(chunk) does not set state.reading=false.
However, this check was not being done for the case when the chunk was
empty (either `''` or `Buffer(0)`), and as a result, it was causing the
socket to think that a read had completed, and to stop providing data.
This bug is not limited to http or web sockets, but rather would affect
any parser that unshifts data back onto the source stream without being
very careful to never unshift an empty chunk. Since the intent of
unshift is to *not* change the state.reading property, this is a bug.
Fixes#5557FixesLearnBoost/socket.io#1242
Pretty much everything assumes strings to be utf-8, but crypto
traditionally used binary strings, so we need to keep the default
that way until most users get off of that pattern.
If there is an encoding, and we do 'stream.push(chunk, enc)', and the
encoding argument matches the stated encoding, then we're converting from
a string, to a buffer, and then back to a string. Of course, this is a
completely pointless bit of work, so it's best to avoid it when we know
that we can do so safely.
Fix#5272
The consumption of a readable stream is a dance with 3 partners.
1. The specific stream Author (A)
2. The Stream Base class (B), and
3. The Consumer of the stream (C)
When B calls the _read() method that A implements, it sets a 'reading'
flag, so that parallel calls to _read() can be avoided. When A calls
stream.push(), B knows that it's safe to start calling _read() again.
If the consumer C is some kind of parser that wants in some cases to
pass the source stream off to some other party, but not before "putting
back" some bit of previously consumed data (as in the case of Node's
websocket http upgrade implementation). So, stream.unshift() will
generally *never* be called by A, but *only* called by C.
Prior to this patch, stream.unshift() *also* unset the state.reading
flag, meaning that C could indicate the end of a read, and B would
dutifully fire off another _read() call to A. This is inappropriate.
In the case of fs streams, and other variably-laggy streams that don't
tolerate overlapped _read() calls, this causes big problems.
Also, calling stream.shift() after the 'end' event did not raise any
kind of error, but would cause very strange behavior indeed. Calling it
after the EOF chunk was seen, but before the 'end' event was fired would
also cause weird behavior, and could lead to data being lost, since it
would not emit another 'readable' event.
This change makes it so that:
1. stream.unshift() does *not* set state.reading = false
2. stream.unshift() is allowed up until the 'end' event.
3. unshifting onto a EOF-encountered and zero-length (but not yet
end-emitted) stream will defer the 'end' event until the new data is
consumed.
4. pushing onto a EOF-encountered stream is now an error.
So, if you read(), you have that single tick to safely unshift() data
back into the stream, even if the null chunk was pushed, and the length
was 0.
In cases where a stream may have data added to the read queue before the
user adds a 'readable' event, there is never any indication that it's
time to start reading.
True, there's already data there, which the user would get if they
checked However, as we use 'readable' event listening as the signal to
start the flow of data with a read(0) call internally, we ought to
trigger the same effect (ie, emitting a 'readable' event) even if the
'readable' listener is added after the first emission.
To avoid confusing weirdness, only the *first* 'readable' event listener
is granted this privileged status. After we've started the flow (or,
alerted the consumer that the flow has started) we don't need to start
it again. At that point, it's the consumer's responsibility to consume
the stream.
Closes#5141
Also, set paused=false *before* calling resume(). Otherwise,
there's an edge case where an immediately-emitted chunk might make
it call pause() again incorrectly.
This solves the problem of calling `readable.pipe(writable)` after the
readable stream has already emitted 'end', as often is the case when
writing simple HTTP proxies.
The spirit of streams2 is that things will work properly, even if you
don't set them up right away on the first tick.
This approach breaks down, however, because pipe()ing from an ended
readable will just do nothing. No more data will ever arrive, and the
writable will hang open forever never being ended.
However, that does not solve the case of adding a `on('end')` listener
after the stream has received the EOF chunk, if it was the first chunk
received (and thus, length was 0, and 'end' got emitted). So, with
this, we defer the 'end' event emission until the read() function is
called.
Also, in pipe(), if the source has emitted 'end' already, we call the
cleanup/onend function on nextTick. Piping from an already-ended stream
is thus the same as piping from a stream that is in the process of
ending.
Updates many tests that were relying on 'end' coming immediately, even
though they never read() from the req.
Fix#4942
In the function that pre-emptively fills the Readable queue, it relies
on a recursion through:
stream.push(chunk) ->
maybeReadMore(stream, state) ->
if (not reading more and < hwm) stream.read(0) ->
stream._read() ->
stream.push(chunk) -> repeat.
Since this was only calling read() a single time, and then relying on a
future nextTick to collect more data, it ends up causing a nextTick
recursion error (and potentially a RangeError, even) if you have a very
high highWaterMark, and are getting very small chunks pushed
synchronously in _read (as happens with TLS, or many simple test
streams).
This change implements a new approach, so that read(0) is called
repeatedly as long as it is effective (that is, the length keeps
increasing), and thus quickly fills up the buffer for streams such as
these, without any stacks overflowing.
Now that highWaterMark increases when there are large reads, this
greatly reduces the number of calls necessary to _read(size), assuming
that _read actually respects the size argument.
When a readable listener is added, call read(0) so that data will flow in, up to
the high water mark.
Otherwise, it's somewhat confusing that you have to listen for readable,
and ALSO call read() (when it will certainly return null) just to get some
data out of the stream.
See: #4720
Ability to return just the length of listeners for a given type, using
EventEmitter.listenerCount(emitter, event). This will be a lot cheaper
than creating a copy of the listeners array just to check its length.
This makes it so that `stream.push(chunk)` is the only way to signal the
end of reading, removing the confusing disparity between the
callback-style _read method, and the fact that most real-world streams
do not have a 1:1 corollation between the "please give me data" event,
and the actual arrival of a chunk of data.
It is still possible, of course, to implement a `CallbackReadable` on
top of this. Simply provide a method like this as the callback:
function readCallback(er, chunk) {
if (er)
stream.emit('error', er);
else
stream.push(chunk);
}
However, *only* fs streams actually would behave in this way, so it
makes not a lot of sense to make TCP, TLS, HTTP, and all the rest have
to bend into this uncomfortable paradigm.
A primary motivation of this is to make the onread function more
inline-friendly, but also to make it more easy to explore not having
onread at all, in favor of always using push() to signal the end of
reading.
The Readable and Writable classes will nextTick certain things
if in sync mode. The sync flag gets unset after a call to _read
or _write. However, most of these behaviors should also be
deferred until nextTick if no reads have been made (for example,
the automatic '_read up to hwm' behavior on Readable.push(chunk))
Set the sync flag to true in the constructor, so that it will not
trigger an immediate 'readable' event, call to _read, before the
user has had a chance to set a _read method implementation.
There are cases where a push() call would return true, even though
the thing being pushed was in fact way way larger than the high
water mark, simply because the 'needReadable' was already set, and
would not get unset until nextTick.
In some cases, this could lead to an infinite loop of pushing data
into the buffer, never getting to the 'readable' event which would
unset the needReadable flag.
Fix by splitting up the emitReadable function, so that it always
sets the flag on this tick, even if it defers until nextTick to
actually emit the event.
Also, if we're not ending or already in the process of reading, it
now calls read(0) if we're below the high water mark. Thus, the
highWaterMark value is the intended amount to buffer up to, and it
is smarter about hitting the target.
It seems like a good idea on the face of it, but lowWaterMarks are
actually not useful, and in practice should always be set to zero.
It would be worthwhile for writers if we actually did some kind of
writev() type of thing, but actually this just delays calling write()
and the overhead of doing a bunch of Buffer copies is not worth the
slight benefit of calling write() fewer times.
This is causing the CryptoStreams to get into an awful state when
there is a tight loop calling connection.write(chunk) waiting for
a false return.
Because CryptoStreams use read(0) to cycle data, this was causing
the encrypted side to pull way too much data in from the cleartext
side, since the read(0) would make it always call _read.
The unfortunate side effect, fixed in the next patch, is that
CryptoStreams don't automatically cycle when the Socket drains.
Otherwise sockets that are 'finish'ed won't be unpiped and `writing to
ended stream` error will arise.
This might sound unrealistic, but it happens in net.js. When
`socket.allowHalfOpen === false`, EOF will cause `.destroySoon()` call which
ends the writable side of net.Socket.
Those values, if passed to the _read() cb, will not signal an EOF. Only
null or undefined will mark the end of data, and trigger the end event.
However, great care must be taken if you are returning an empty string
or buffer! There must be some other thing somewhere that will trigger
a read() call, because there will never be a readable event fired later.
This is in preparation for CryptoStreams being ported to streams2, where
it is safe to simply stop reading, because the crypto cycle process will
cause it to read(0) again at some future date.
We detect for non-string and non-buffer values in onread and
turn the stream into an "objectMode" stream.
If we are in "objectMode" mode then howMuchToRead will
always return 1, state.length will always have 1 appended
to it when there is a new item and fromList always takes
the first value from the list.
This means that for object streams, the n in read(n) is
ignored and read() will always return a single value
Fixed a bug with unpipe where the pipe would break because
the flowing state was not reset to false.
Fixed a bug with sync cb(null, null) in _read which would
forget to end the readable stream
Problem 1: If stream.push() triggers a 'readable' event, and the user
calls `read(n)` with some n > the highWaterMark, then the push() will
return false (indicating that they should not push any more), but no
future 'readable' event is coming (because we're above the
highWaterMark).
Solution: return true from push() when needReadable is set.
Problem 2: A read(n) for n != 0, after the stream had encountered an
EOF, would not trigger the 'end' event if the EOF was pushed in
synchronously by the _read() function.
Solution: Check for ended in stream.read() and schedule an end event if
the length now equals 0.
Fix#4585
There was previously an assert() in there, but this part of the code is
so high-volume that the added cost made a measurable dent in http_simple.
Just checking inline is fine, though, and prevents a lot of potential
hazards.
Say that a stream's current read queue has 101 bytes in it, and the
underlying resource has ended (ie, reached EOF).
If you do something like this:
stream.read(100); // leave a byte behind
stream.read(0); // read(0) for some reason
then the read(0) will get 0 from the howMuchToRead function. Since the
stream was ended, this was incorrectly treating the 0 as a "there is no
more in the buffer", and emitting 'end' before that last byte was read.
Why have the read(0) in the first place? We do this in some cases to
trigger the last few bytes of a net socket (such as a child process's
stdio pipes). This was causing issues when piping a `git archive` job
to a file: the resulting tarball was incomplete, because it occasionally
was not getting the last chunk.
When switching into compatibility mode by setting `data` event listener,
`_read()` method will be called immediately. If method implementation
invokes callback in the same tick - all emitted `data` events will be
discarded, because `data` listener wasn't set yet.
