UTF-8 v0.1.0

MaxScalarValue

pub const MaxScalarValue: uint32 = 0x10FFFF;

The largest Unicode scalar value.

Source

FirstSurrogate

pub const FirstSurrogate: uint32 = 0xD800;

The first value of the surrogate range, which holds no scalar values.

Source

LastSurrogate

pub const LastSurrogate: uint32 = 0xDFFF;

The last value of the surrogate range.

Source

MaxEncodedLength

pub const MaxEncodedLength: uint = 4;

The most bytes any one scalar value takes in UTF-8.

Source

DecodedScalar

pub struct DecodedScalar {
    pub scalar: char32;
    pub width: uint;
}

One scalar value decoded from UTF-8, with the number of bytes it occupied.

Source

Fields

Utf8Failure

pub struct Utf8Failure {
    pub reason: Utf8Error;
    pub index: uint;
}

Where and why a byte sequence stopped being well-formed UTF-8.

Source

Fields

IsScalarValue

pub func IsScalarValue(value: uint32) -> bool

Whether value is a Unicode scalar value: at most MaxScalarValue, and not a surrogate half. Cannot fail. Runs in constant time and allocates nothing.

Source

Parameters

Returns: whether the value is a Unicode scalar value, which excludes the surrogates

EncodedLength

pub func EncodedLength(scalar: char32) -> uint

How many bytes scalar takes in UTF-8, or zero when it is not a scalar value. Cannot fail. Runs in constant time and allocates nothing.

Source

Parameters

Returns: how many bytes the scalar's encoding occupies, one through four

IsContinuation

pub func IsContinuation(byte: char8) -> bool

Whether byte is a continuation byte, the 10xxxxxx shape every byte after a leading one has.

Source

Parameters

Returns: whether the byte continues a sequence rather than starting one

Encode

pub func Encode(
    scalar: char32,
    into: var char8[..]
) -> uint ! Utf8Error

Writes scalar into into as UTF-8 and reports how many bytes it took. into is borrowed mutably and must have room for the encoding — at most MaxEncodedLength bytes. Fails with InvalidStart for a value that is not a scalar value, and Truncated when there is not enough room; nothing is written in either case. Cannot allocate. Runs in constant time.

Source

Parameters

Returns: how many bytes were written, or why none were

DecodeAt

pub func DecodeAt(
    bytes: char8[..],
    index: uint
) -> DecodedScalar ! Utf8Error

Decodes the scalar value starting at index, reporting it and how many bytes it occupied. bytes is borrowed. On a failure the caller decides what to do — a decoder replacing bad input with U+FFFD advances one byte and carries on, and one rejecting it stops. Cannot allocate. Runs in constant time.

Source

Parameters

Returns: the scalar and its width, or why the bytes there are not one

Validate

pub func Validate(bytes: char8[..]) -> ! Utf8Failure

Whether bytes is well-formed UTF-8 from end to end, reporting where it first is not. bytes is borrowed. A failure carries the index of the byte that starts the offending sequence. Cannot allocate. Runs in time proportional to the length.

Source

Parameters

Returns: nothing when every byte is part of a valid sequence, or where and why the first one is not

IsBoundary

pub func IsBoundary(bytes: char8[..], index: uint) -> bool

Whether index is where a scalar value starts, and so a place a string may be cut. The length counts as a boundary, so the end of a string is always a legal cut. An index past the length is not. Cannot fail. Runs in constant time and allocates nothing.

Source

Parameters

Returns: whether index falls between two scalar values rather than inside one

CountScalars

pub func CountScalars(bytes: char8[..]) -> uint

How many scalar values bytes holds, assuming it is well-formed. Counts leading bytes, which is one per scalar value in well-formed UTF-8. Bytes that are not well-formed give a count that means nothing; Validate is what answers whether they are. Cannot fail. Runs in time proportional to the length and allocates nothing.

Source

Parameters

Returns: how many scalar values the bytes encode

BoundaryAtOrBelow

pub func BoundaryAtOrBelow(
    bytes: char8[..],
    index: uint
) -> uint

The index of the scalar boundary at or below index. An index inside a sequence is walked back to the start of that sequence, which is what a caller cutting at an arbitrary byte offset needs. An index at or past the length gives the length.

Source

Parameters

Returns: the largest boundary at or below index, so a truncation splits nothing