Token Reference

The lexer turns a source file into a sequence of tokens, reading each one by the longest-match rule. This page lists every token kind under the name the compiler gives it.

Literals

Token	Examples	See
`IntLiteral`	`42`, `0xFF`, `0o77`, `0b1010`, `200u8`	Integer literals
`FloatLiteral`	`3.14`, `1.0e-9`, `0.5f32`	Floating-point literals
`StringLiteral`	`"hello"`, `c8"hello"`, `c16"hello"`, `c32"hello"`	String literals
`CharLiteral`	`'A'`, `'\n'`, `c8'A'`, `c16'A'`, `c32'A'`, `c64'A'`, `c128'A'`, `c256'A'`, `c512'A'`	Character literals
`BoolLiteral`	`true`, `false`	Boolean literals

A numeric suffix and a character or string prefix are part of the literal's token.

Identifiers

Token	Examples
`Identifier`	`count`, `Point`, `_tmp`

Contextual words such as asm, sizeof and Self are identifiers to the lexer; see Keywords.

Keywords

Token	Spelling	Token	Spelling
`AsKeyword`	`as`	`LetKeyword`	`let`
`BreakKeyword`	`break`	`LoopKeyword`	`loop`
`CatchKeyword`	`catch`	`MatchKeyword`	`match`
`ConstKeyword`	`const`	`ModuleKeyword`	`module`
`ContinueKeyword`	`continue`	`MoveKeyword`	`move`
`DeferKeyword`	`defer`	`NoneKeyword`	`none`
`DoKeyword`	`do`	`NullKeyword`	`null`
`ElseKeyword`	`else`	`PubKeyword`	`pub`
`EnumKeyword`	`enum`	`ReturnKeyword`	`return`
`ExtendKeyword`	`extend`	`SelfKeyword`	`self`
`ExternKeyword`	`extern`	`StructKeyword`	`struct`
`FailKeyword`	`fail`	`ThrowKeyword`	`throw`
`FinallyKeyword`	`finally`	`ThrowsKeyword`	`throws`
`ForKeyword`	`for`	`TryKeyword`	`try`
`FuncKeyword`	`func`	`TypeKeyword`	`type`
`IfKeyword`	`if`	`UnionKeyword`	`union`
`ImportKeyword`	`import`	`UnsafeKeyword`	`unsafe`
`InKeyword`	`in`	`VarKeyword`	`var`
`InterfaceKeyword`	`interface`	`VariantKeyword`	`variant`
`IntrinsicKeyword`	`intrinsic`	`WhenKeyword`	`when`
`IsKeyword`	`is`	`WhileKeyword`	`while`

Together with true and false, which are BoolLiteral tokens, these are the 44 reserved words. fail and finally are reserved but start nothing: fail value; is refused with help: write 'throw value;', and finally with help: use 'defer' inside the try block. See Keywords.

Punctuation

Token	Spelling	Token	Spelling
`LeftParen`	`(`	`Dot`	`.`
`RightParen`	`)`	`DotDot`	`..`
`LeftBrace`	`{`	`DotDotDot`	`...`
`RightBrace`	`}`	`DotDotEqual`	`..=`
`LeftBracket`	`[`	`Arrow`	`->`
`RightBracket`	`]`	`FatArrow`	`=>`
`Comma`	`,`	`At`	`@`
`Semicolon`	`;`	`Hash`	`#`
`Colon`	`:`	`Question`	`?`
`ColonColon`	`::`	`QuestionQuestion`	`??`

Operators

Token	Spelling	Group	Token	Spelling	Group
`Plus`	`+`	Arithmetic	`Equal`	`==`	Comparison
`Minus`	`-`	Arithmetic	`BangEqual`	`!=`	Comparison
`Star`	`*`	Arithmetic	`Less`	`<`	Comparison
`Slash`	`/`	Arithmetic	`LessEqual`	`<=`	Comparison
`Percent`	`%`	Arithmetic	`Greater`	`>`	Comparison
`PlusPlus`	`++`	Arithmetic	`GreaterEqual`	`>=`	Comparison
`MinusMinus`	`--`	Arithmetic	`MoveArrow`	`<-`	Rejected
`Amp`	`&`	Bitwise	`Assign`	`=`	Assignment
`Pipe`	`\|`	Bitwise	`PlusAssign`	`+=`	Assignment
`Caret`	`^`	Bitwise	`MinusAssign`	`-=`	Assignment
`Tilde`	`~`	Bitwise	`StarAssign`	`*=`	Assignment
`LessLess`	`<<`	Bitwise	`SlashAssign`	`/=`	Assignment
`GreaterGreater`	`>>`	Bitwise	`PercentAssign`	`%=`	Assignment
`GreaterGreaterGreater`	`>>>`	Bitwise	`AmpAssign`	`&=`	Assignment
`AmpAmp`	`&&`	Logical	`PipeAssign`	`\|=`	Assignment
`PipePipe`	`\|\|`	Logical	`CaretAssign`	`^=`	Assignment
`Bang`	`!`	Logical	`LessLessAssign`	`<<=`	Assignment
`PlusPercent`	`+%`	Wrapping	`GreaterGreaterAssign`	`>>=`	Assignment
`MinusPercent`	`-%`	Wrapping	`GreaterGreaterGreaterAssign`	`>>>=`	Assignment
`StarPercent`	`*%`	Wrapping	`PlusPercentAssign`	`+%=`	Assignment
			`MinusPercentAssign`	`-%=`	Assignment
			`StarPercentAssign`	`*%=`	Assignment

Operators and Punctuation says what each one means. <- is still read as a token only so that the parser can refuse it with the fix: ownership moves with move, as in let name = move source; and target = move source;.

Special tokens

Token	Meaning
`DocComment`	A documentation comment, `/// …` or `/** … */`, kept so the parser can attach it to the next declaration. Ordinary comments produce no token.
`EndOfFile`	Appended after the last token of every file
`Unknown`	A character that starts no token, kept so the error can quote it
`NewLine`	Defined but never produced: line breaks are whitespace