mirror of
https://git.proxmox.com/git/rustc
synced 2025-08-17 00:53:04 +00:00
253 lines
8.4 KiB
Markdown
253 lines
8.4 KiB
Markdown
bstr
|
|
====
|
|
This crate provides a `BString` and `BStr` types that are conventionally UTF-8
|
|
for Rust. They differ from the standard library's `String` and `str` types in
|
|
that they are not required to be valid UTF-8, but may be fully or partially
|
|
valid UTF-8.
|
|
|
|
[](https://travis-ci.org/BurntSushi/bstr)
|
|
[](https://ci.appveyor.com/project/BurntSushi/bstr)
|
|
[](https://crates.io/crates/bstr)
|
|
|
|
|
|
### Documentation
|
|
|
|
https://docs.rs/bstr
|
|
|
|
|
|
### When should I use byte strings?
|
|
|
|
See this part of the documentation for more details:
|
|
https://docs.rs/bstr/0.1.0/bstr/#when-should-i-use-byte-strings.
|
|
|
|
The short story is that byte strings are useful when it is inconvenient or
|
|
incorrect to require valid UTF-8.
|
|
|
|
|
|
### Usage
|
|
|
|
Add this to your `Cargo.toml`:
|
|
|
|
```toml
|
|
[dependencies]
|
|
bstr = "0.1"
|
|
```
|
|
|
|
|
|
### Examples
|
|
|
|
The following two examples exhibit both the API features of byte strings and
|
|
the I/O convenience functions provided for reading line-by-line quickly.
|
|
|
|
This first example simply shows how to efficiently iterate over lines in
|
|
stdin, and print out lines containing a particular substring:
|
|
|
|
```rust
|
|
use std::error::Error;
|
|
use std::io::{self, Write};
|
|
|
|
use bstr::io::BufReadExt;
|
|
|
|
fn main() -> Result<(), Box<Error>> {
|
|
let stdin = io::stdin();
|
|
let mut stdout = io::BufWriter::new(io::stdout());
|
|
|
|
stdin.lock().for_byte_line_with_terminator(|line| {
|
|
if line.contains("Dimension") {
|
|
stdout.write_all(line.as_bytes())?;
|
|
}
|
|
Ok(true)
|
|
})?;
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
This example shows how to count all of the words (Unicode-aware) in stdin,
|
|
line-by-line:
|
|
|
|
```rust
|
|
use std::error::Error;
|
|
use std::io;
|
|
|
|
use bstr::io::BufReadExt;
|
|
|
|
fn main() -> Result<(), Box<Error>> {
|
|
let stdin = io::stdin();
|
|
let mut words = 0;
|
|
stdin.lock().for_byte_line_with_terminator(|line| {
|
|
words += line.words().count();
|
|
Ok(true)
|
|
})?;
|
|
println!("{}", words);
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
This example shows how to convert a stream on stdin to uppercase without
|
|
performing UTF-8 validation _and_ amortizing allocation. On standard ASCII
|
|
text, this is quite a bit faster than what you can (easily) do with standard
|
|
library APIs. (N.B. Any invalid UTF-8 bytes are passed through unchanged.)
|
|
|
|
```rust
|
|
use std::error::Error;
|
|
use std::io::{self, Write};
|
|
|
|
use bstr::BString;
|
|
use bstr::io::BufReadExt;
|
|
|
|
fn main() -> Result<(), Box<Error>> {
|
|
let stdin = io::stdin();
|
|
let mut stdout = io::BufWriter::new(io::stdout());
|
|
|
|
let mut upper = BString::new();
|
|
stdin.lock().for_byte_line_with_terminator(|line| {
|
|
upper.clear();
|
|
line.to_uppercase_into(&mut upper);
|
|
stdout.write_all(upper.as_bytes())?;
|
|
Ok(true)
|
|
})?;
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
This example shows how to extract the first 10 visual characters (as grapheme
|
|
clusters) from each line, where invalid UTF-8 sequences are generally treated
|
|
as a single character and are passed through correctly:
|
|
|
|
```rust
|
|
use std::error::Error;
|
|
use std::io::{self, Write};
|
|
|
|
use bstr::io::BufReadExt;
|
|
|
|
fn main() -> Result<(), Box<Error>> {
|
|
let stdin = io::stdin();
|
|
let mut stdout = io::BufWriter::new(io::stdout());
|
|
|
|
stdin.lock().for_byte_line_with_terminator(|line| {
|
|
let end = line
|
|
.grapheme_indices()
|
|
.map(|(_, end, _)| end)
|
|
.take(10)
|
|
.last()
|
|
.unwrap_or(line.len());
|
|
stdout.write_all(line[..end].trim_end().as_bytes())?;
|
|
stdout.write_all(b"\n")?;
|
|
Ok(true)
|
|
})?;
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
|
|
### Cargo features
|
|
|
|
This crates comes with a few features that control standard library, serde
|
|
and Unicode support.
|
|
|
|
* `std` - **Enabled** by default. This provides APIs that require the standard
|
|
library, such as `BString`.
|
|
* `unicode` - **Enabled** by default. This provides APIs that require sizable
|
|
Unicode data compiled into the binary. This includes, but is not limited to,
|
|
grapheme/word/sentence segmenters. When this is disabled, basic support such
|
|
as UTF-8 decoding is still included.
|
|
* `serde1` - **Disabled** by default. Enables implementations of serde traits
|
|
for the `BStr` and `BString` types.
|
|
* `serde1-nostd` - **Disabled** by default. Enables implementations of serde
|
|
traits for the `BStr` type only, intended for use without the standard
|
|
library. Generally, you either want `serde1` or `serde1-nostd`, not both.
|
|
|
|
|
|
### Minimum Rust version policy
|
|
|
|
This crate's minimum supported `rustc` version (MSRV) is `1.28.0`.
|
|
|
|
In general, this crate will be conservative with respect to the minimum
|
|
supported version of Rust. MSRV may be bumped in minor version releases.
|
|
|
|
|
|
### Future work
|
|
|
|
Since this is meant to be a core crate, getting a `1.0` release is a priority.
|
|
My hope is to move to `1.0` within the next year and commit to its API so that
|
|
`bstr` can be used as a public dependency.
|
|
|
|
A large part of the API surface area was taken from the standard library, so
|
|
from an API design perspective, a good portion of this crate should be mature.
|
|
The main differences from the standard library are in how the various substring
|
|
search routines work. The standard library provides generic infrastructure for
|
|
supporting different types of searches with a single method, where as this
|
|
library prefers to define new methods for each type of search and drop the
|
|
generic infrastructure.
|
|
|
|
Some _probable_ future considerations for APIs include, but are not limited to:
|
|
|
|
* A convenience layer on top of the `aho-corasick` crate.
|
|
* Unicode normalization.
|
|
* More sophisticated support for dealing with Unicode case, perhaps by
|
|
combining the use cases supported by [`caseless`](http://docs.rs/caseless)
|
|
and [`unicase`](https://docs.rs/unicase).
|
|
* Add facilities for dealing with OS strings and file paths, probably via
|
|
simple conversion routines.
|
|
|
|
Here are some examples that are _probably_ out of scope for this crate:
|
|
|
|
* Regular expressions.
|
|
* Unicode collation.
|
|
|
|
The exact scope isn't quite clear, but I expect we can iterate on it.
|
|
|
|
In general, as stated below, this crate is an experiment in bringing lots of
|
|
related APIs together into a single crate while simultaneously attempting to
|
|
keep the total number of dependencies low. Indeed, every dependency of `bstr`,
|
|
except for `memchr`, is optional.
|
|
|
|
|
|
### High level motivation
|
|
|
|
Strictly speaking, the `bstr` crate provides very little that can't already be
|
|
achieved with a `Vec<u8>`/`&[u8]` and the ecosystem of library crates. For
|
|
example:
|
|
|
|
* The standard library's
|
|
[`Utf8Error`](https://doc.rust-lang.org/std/str/struct.Utf8Error.html)
|
|
can be used for incremental lossy decoding of `&[u8]`.
|
|
* The
|
|
[`unicode-segmentation`](https://unicode-rs.github.io/unicode-segmentation/unicode_segmentation/index.html)
|
|
crate can be used for iterating over graphemes (or words), but is only
|
|
implemented for `&str` types. One could use `Utf8Error` above to implement
|
|
grapheme iteration with the same semantics as what `bstr` provides (automatic
|
|
Unicode replacement codepoint substitution).
|
|
* The [`twoway`](https://docs.rs/twoway/0.2.0/twoway/) crate can be used for
|
|
fast substring searching on `&[u8]`.
|
|
|
|
So why create `bstr`? Part of the point of the `bstr` crate is to provide a
|
|
uniform API of coupled components instead of relying on users to piece together
|
|
loosely coupled components from the crate ecosystem. For example, if you wanted
|
|
to perform a search and replace in a `Vec<u8>`, then writing the code to do
|
|
that with the `twoway` crate is not that difficult, but it's still additional
|
|
glue code you have to write. This work adds up depending on what you're doing.
|
|
Consider, for example, trimming and splitting, along with their different
|
|
variants.
|
|
|
|
In other words, `bstr` is partially a way of pushing back against the
|
|
micro-crate ecosystem that appears to be evolving. It's not clear to me whether
|
|
this experiment will be successful or not, but it is definitely a goal of
|
|
`bstr` to keep its dependency list lightweight. For example, `serde` is an
|
|
optional dependency because there is no feasible alternative, but `twoway` is
|
|
not, where we instead prefer to implement our own substring search. In service
|
|
of this philosophy, currently, the only required dependency of `bstr` is
|
|
`memchr`.
|
|
|
|
|
|
### License
|
|
|
|
This project is licensed under either of
|
|
|
|
* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
|
|
http://www.apache.org/licenses/LICENSE-2.0)
|
|
* MIT license ([LICENSE-MIT](LICENSE-MIT) or
|
|
http://opensource.org/licenses/MIT)
|
|
|
|
at your option.
|