TomlLexer structv0.1.0

A TOML lexer over a byte slice.

pub struct TomlLexer {
    pub line: uint;
    // private fields
}

Fields

Name	Type	Description
`line`	`uint`	Which line that is on.

Constructors

`TomlLexer`

pub func TomlLexer(input: char8[..]) -> TomlLexer

A lexer over input.

Name	Type	Description
`input`	`char8[..]`	the text to read

Returns: a lexer positioned at the start of the input

Source

Methods

`Next`

pub func Next(self: &var TomlLexer) -> TomlToken

The next token, or End at the end of the input.

Returns: the next token, or the end-of-input token

Source

`TextOf`

pub func TextOf(
    self: &TomlLexer,
    token: &TomlToken
) -> char8[..]

The bytes a token spans, which for a text token excludes the quotes and is still escaped.

Name	Type	Description
`token`	`&TomlToken`	the token to read

Returns: the token's text, borrowed from the input

Source

`TomlTokenKind`

pub enum TomlTokenKind: int32 {
    End = 0,
    BareKey = 1,
    Text = 2,
    Integer = 3,
    Float = 4,
    Boolean = 5,
    DateTime = 6,
    Equals = 7,
    Dot = 8,
    Comma = 9,
    LeftBracket = 10,
    RightBracket = 11,
    LeftBrace = 12,
    RightBrace = 13,
    Newline = 14,
    Error = 15
}

What kind of token this is.

Source

Cases

Name	Value	Description
`End`	`0`	The input is exhausted.
`BareKey`	`1`	A bare key: letters, digits, underscores and hyphens.
`Text`	`2`	A quoted key or a string value; the span excludes the quotes and the text may need unescaping.
`Integer`	`3`	An integer, in any of the four bases.
`Float`	`4`	A float, including the infinities and NaN that TOML — unlike JSON — can spell.
`Boolean`	`5`	`true` or `false`.
`DateTime`	`6`	An offset date-time, a local date-time, a local date, or a local time.
`Equals`	`7`	`=`
`Dot`	`8`	`.`
`Comma`	`9`	`,`
`LeftBracket`	`10`	`[`
`RightBracket`	`11`	`]`
`LeftBrace`	`12`	`{`
`RightBrace`	`13`	`}`
`Newline`	`14`	One or more newlines, which TOML treats as significant.
`Error`	`15`	Something that is not TOML.

`TomlTextForm`

pub enum TomlTextForm: int32 {
    NotText = 0,
    Basic = 1,
    MultiLineBasic = 2,
    Literal = 3,
    MultiLineLiteral = 4,
    Bare = 5
}

Which of the four string forms a text token was written in.

Source

Cases

Name	Value	Description
`NotText`	`0`	Not a text token.
`Basic`	`1`	A basic string: escapes honoured, no raw newlines.
`MultiLineBasic`	`2`	A multi-line basic string: escapes honoured, newlines allowed, a leading newline trimmed.
`Literal`	`3`	A literal string: no escapes at all, which is what makes it right for Windows paths and regular expressions.
`MultiLineLiteral`	`4`	A multi-line literal string: no escapes, newlines allowed.
`Bare`	`5`	A bare key, which has no quotes and no escapes.

`TomlLexError`

pub enum TomlLexError: int32 {
    None = 0,
    UnexpectedByte = 1,
    UnterminatedText = 2,
    BadEscape = 3,
    BadUnicodeEscape = 4,
    MalformedNumber = 5,
    NewlineInText = 6,
    UnescapedControl = 7,
    MalformedDateTime = 8
}

Why lexing stopped.

Source

Cases

Name	Value	Description
`None`	`0`	Nothing went wrong.
`UnexpectedByte`	`1`	A byte that begins no token.
`UnterminatedText`	`2`	A string the input ended before closing.
`BadEscape`	`3`	An escape sequence TOML does not define.
`BadUnicodeEscape`	`4`	A unicode escape that is not the right number of digits, or names no scalar value.
`MalformedNumber`	`5`	A number whose shape TOML grammar does not allow.
`NewlineInText`	`6`	A raw newline inside a single-line string.
`UnescapedControl`	`7`	A control character appearing unescaped where TOML forbids it.
`MalformedDateTime`	`8`	A date or time that is not the shape RFC 3339 defines.

`TomlToken`

pub struct TomlToken {
    pub kind: TomlTokenKind;
    pub form: TomlTextForm;
    pub start: uint;
    pub end: uint;
    pub line: uint;
    pub column: uint;
    pub error: TomlLexError;
}

One token, and where it came from.

Source

Fields

Name	Type	Description
`kind`	`TomlTokenKind`	What kind of token this is.
`form`	`TomlTextForm`	Which string form, for a text token.
`start`	`uint`	Where its text starts, as a byte offset.
`end`	`uint`	Where its text ends.
`line`	`uint`	Which line it starts on, counting from one.
`column`	`uint`	Which column it starts at, counting from one, in bytes.
`error`	`TomlLexError`	Why lexing stopped, when the kind is `Error`.

`TomlDecodeText`

pub func TomlDecodeText(
    allocator: Allocator,
    span: char8[..],
    form: TomlTextForm,
    bad: *var bool
) -> String ! TextError

Decodes a text token's span into an owned string, honouring the escapes its form allows. A literal string is copied unchanged — that is what makes it the right form for a Windows path — while a basic string has its escapes resolved, including the line-ending backslash that lets a multi-line string be wrapped in the source without the wrapping appearing in the value.

Source

Parameters

Name	Type	Description
`allocator`	`Allocator`	where the storage comes from
`span`	`char8[..]`	the raw text the token covers
`form`	`TomlTextForm`	which quoting form the text was written in
`bad`	`*var bool`	receives the offset of the first byte that could not be decoded

Returns: the decoded text, or why it could not be stored

TomlEventReader

Reads a document as events, in order, without building a tree.

TomlParseFailure

Why a document was refused, and where in it.

Edit this page on GitHub