Token Reference

The lexer turns a source file into a sequence of tokens, reading each one by the longest-match rule. This page lists every token kind under the name the compiler gives it.

Literals

Token	Examples	See
`IntLiteral`	`42`, `0xFF`, `0o77`, `0b1010`, `200u8`	Integer literals
`FloatLiteral`	`3.14`, `1.0e-9`, `0.5f32`	Floating-point literals
`StringLiteral`	`"hello"`, `c8"hello"`, `c16"hello"`, `c32"hello"`	String literals
`CharLiteral`	`'A'`, `'\n'`, `c8'A'`, `c16'A'`, `c32'A'`, `c64'A'`, `c128'A'`, `c256'A'`, `c512'A'`	Character literals
`BoolLiteral`	`true`, `false`	Boolean literals

A numeric suffix and a character or string prefix are part of the literal's token.

Identifiers

Token	Examples
`Identifier`	`count`, `Point`, `_tmp`

Contextual words such as asm, sizeof and Self are identifiers to the lexer; see Keywords.

Keywords

Token	Spelling	Token	Spelling
`AsKeyword`	`as`	`InterfaceKeyword`	`interface`
`BreakKeyword`	`break`	`IntrinsicKeyword`	`intrinsic`
`CatchKeyword`	`catch`	`IsKeyword`	`is`
`ConstKeyword`	`const`	`LetKeyword`	`let`
`ContinueKeyword`	`continue`	`LoopKeyword`	`loop`
`DeferKeyword`	`defer`	`MatchKeyword`	`match`
`DoKeyword`	`do`	`ModuleKeyword`	`module`
`ElseKeyword`	`else`	`NoneKeyword`	`none`
`EnumKeyword`	`enum`	`NullKeyword`	`null`
`ExtendKeyword`	`extend`	`PubKeyword`	`pub`
`ExternKeyword`	`extern`	`ReturnKeyword`	`return`
`FailKeyword`	`fail`	`SelfKeyword`	`self`
`ForKeyword`	`for`	`StructKeyword`	`struct`
`FuncKeyword`	`func`	`TypeKeyword`	`type`
`IfKeyword`	`if`	`UnionKeyword`	`union`
`ImportKeyword`	`import`	`VarKeyword`	`var`
`InKeyword`	`in`	`VariantKeyword`	`variant`
		`WhenKeyword`	`when`
		`WhileKeyword`	`while`

Together with true and false, which are BoolLiteral tokens, these are the 38 reserved words.

Punctuation

Token	Spelling	Token	Spelling
`LeftParen`	`(`	`Dot`	`.`
`RightParen`	`)`	`DotDot`	`..`
`LeftBrace`	`{`	`DotDotDot`	`...`
`RightBrace`	`}`	`DotDotEqual`	`..=`
`LeftBracket`	`[`	`Arrow`	`->`
`RightBracket`	`]`	`FatArrow`	`=>`
`Comma`	`,`	`At`	`@`
`Semicolon`	`;`	`Hash`	`#`
`Colon`	`:`	`Question`	`?`
`ColonColon`	`::`	`QuestionQuestion`	`??`

Operators

Token	Spelling	Group	Token	Spelling	Group
`Plus`	`+`	Arithmetic	`Equal`	`==`	Comparison
`Minus`	`-`	Arithmetic	`BangEqual`	`!=`	Comparison
`Star`	`*`	Arithmetic	`Less`	`<`	Comparison
`Slash`	`/`	Arithmetic	`LessEqual`	`<=`	Comparison
`Percent`	`%`	Arithmetic	`Greater`	`>`	Comparison
`PlusPlus`	`++`	Arithmetic	`GreaterEqual`	`>=`	Comparison
`MinusMinus`	`--`	Arithmetic	`MoveArrow`	`<-`	Assignment
`Amp`	`&`	Bitwise	`Assign`	`=`	Assignment
`Pipe`	`\|`	Bitwise	`PlusAssign`	`+=`	Assignment
`Caret`	`^`	Bitwise	`MinusAssign`	`-=`	Assignment
`Tilde`	`~`	Bitwise	`StarAssign`	`*=`	Assignment
`LessLess`	`<<`	Bitwise	`SlashAssign`	`/=`	Assignment
`GreaterGreater`	`>>`	Bitwise	`PercentAssign`	`%=`	Assignment
`GreaterGreaterGreater`	`>>>`	Bitwise	`AmpAssign`	`&=`	Assignment
`AmpAmp`	`&&`	Logical	`PipeAssign`	`\|=`	Assignment
`PipePipe`	`\|\|`	Logical	`CaretAssign`	`^=`	Assignment
`Bang`	`!`	Logical	`LessLessAssign`	`<<=`	Assignment
			`GreaterGreaterAssign`	`>>=`	Assignment
			`GreaterGreaterGreaterAssign`	`>>>=`	Assignment

Operators and Punctuation says what each one means.

Special tokens

Token	Meaning
`DocComment`	A documentation comment, `/// …` or `/** … */`, kept so the parser can attach it to the next declaration. Ordinary comments produce no token.
`EndOfFile`	Appended after the last token of every file
`Unknown`	A character that starts no token, kept so the error can quote it
`NewLine`	Defined but never produced: line breaks are whitespace