Unicode v0.1.0

Unicode character properties, case, normalization and segmentation

Installation

Add Unicode to the [Dependencies] table of your package's Rux.toml:

[Dependencies]
Unicode = { Namespace = "Rux", Version = "0.1.0" }

Standard packages are declared by hand: the registry does not serve them, so rux add Rux/Unicode cannot add them.

Guide

Unicode character properties, case, normalization and segmentation, from tables generated out of the Unicode 17.0.0 Character Database.

What it provides

Properties — the general category, canonical combining class, and the White_Space, Alphabetic and numeric properties, answered by binary search over compressed ranges.
Case — simple per-character mappings, the unconditional full mappings, locale-independent full case folding, and NextInCaseOrbit, which walks every value with the same simple folding, such as k, K and the Kelvin sign. The conditional mappings are deliberately absent: they depend on locale or context, and applying them silently is how i stops round-tripping in Turkish.
Normalization — canonical and compatibility decomposition (Hangul arithmetically), canonical ordering, and composition: all four normal forms, over scalar-value slices.
Graphemes — extended grapheme cluster boundaries by the UAX #29 rules, emoji sequences and regional-indicator pairs included.
Scripts — the Script property: ScriptOf gives the Script a scalar belongs to, such as Latin, Greek or Common, and Script::FromName finds one by its long name, matched loosely.

The tables under Src/Generated*.rux are committed output of Tools/UnicodeGen, which records the SHA-256 of every input it read; regenerate with Bin/Tools/rux-unicode-gen after changing the data under Tools/UnicodeGen/Data.

Value model

Unicode scalar values, GeneralCategory, and GraphemeBreak are structural Copy values. Text is borrowed through char32[..] and results are written into caller-owned var char32[..] storage; the package allocates and owns nothing. Length results remain raw writable pointers, so pass an address such as @written. It must be non-null and valid for one uint throughout the call. The slices carry their bounds but not ownership, and their backing storage must outlive the operation.

Example

import Unicode::ToUpperFull;

var output: char32[3];
var written: uint = 0;
ToUpperFull(0xDFu32 as char32, (@output[0])[..3], @written);

Index

Types

Name	Summary
`GraphemeBreak`	The grapheme break class of a scalar value.
`GeneralCategory`	The Unicode general category, one per scalar value.
`Script`	A script.

Functions

Name	Summary
`ToUpperSimple`	The simple uppercase of `scalar`: one value to one, itself when it has no mapping.
`ToLowerSimple`	The simple lowercase of `scalar`.
`ToTitleSimple`	The simple titlecase of `scalar`: the uppercase, except for the few dozen characters whose titlecase differs.
`NextInCaseOrbit`	The next scalar value that case-insensitive comparison treats as `scalar`, going round the set of values with the same simple case folding: `k`, `K` and the Kelvin sign, or `σ`, `ς` and `Σ`.
`ToUpperFull`	Writes the full uppercase of `scalar` into `into` and reports how many values it took.
`ToLowerFull`	Writes the full lowercase of `scalar` into `into`.
`Fold`	Writes the full, locale-independent case folding of `scalar` into `into`.
`GraphemeBreakOf`	The break class of `scalar`.
`IsPictographic`	Whether `scalar` is an extended pictograph: the emoji and everything reserved to become one.
`IsGraphemeBoundary`	Whether a cluster boundary falls before `index` in `text`.
`CountGraphemes`	How many grapheme clusters `text` holds.
`DecomposeCanonical`	Writes the canonical decomposition of `text` into `into`, marks in canonical order: NFD.
`DecomposeCompatibility`	Writes the compatibility decomposition of `text` into `into`, marks in canonical order: NFKD.
`NormalizeCanonical`	Writes the NFC of `text` into `into`: canonical decomposition, canonical order, then composition.
`NormalizeCompatibility`	Writes the NFKC of `text` into `into`: compatibility decomposition, canonical order, then composition.
`CategoryOf`	The general category of `scalar`.
`CombiningClassOf`	The canonical combining class of `scalar`: zero for nearly everything, and the reordering key for marks.
`IsWhiteSpace`	Whether `scalar` is whitespace by Unicode's `White_Space` property.
`IsAlphabetic`	Whether `scalar` is alphabetic by the derived `Alphabetic` property.
`IsNumeric`	Whether `scalar` is numeric: a decimal digit, a letterlike number, or another number form.
`ScriptOf`	The script `scalar` belongs to.
`ScriptCount`	How many scripts there are, Unknown among them.

Constants

Name	Summary
`SimpleUpperPairs`	Simple uppercase pairs: code point, its uppercase.
`SimpleLowerPairs`	Simple lowercase pairs: code point, its lowercase.
`SimpleTitlePairs`	Simple titlecase pairs, for the points whose titlecase differs from their uppercase.
`CaseOrbitPairs`	Case orbits the simple mappings do not pair: code point, the next point that folds to the same point, the largest followed by the smallest.
`FullFoldIndex`	Full case folding index: code point, offset, length.
`FullFoldElements`	Full case folding elements, indexed by FullFoldIndex.
`FullUpperIndex`	Unconditional full uppercase index: code point, offset, length.
`FullUpperElements`	Unconditional full uppercase elements.
`FullLowerIndex`	Unconditional full lowercase index: code point, offset, length.
`FullLowerElements`	Unconditional full lowercase elements.
`NormalizationVectorIndex`	Normalization vector index: five (offset, length) pairs per row, for the columns c1..c5.
`NormalizationVectorElements`	The scalar values of every conformance vector, run together end to end.
`GraphemeVectorIndex`	Grapheme vector index: offset, length per row.
`GraphemeVectorPoints`	Grapheme vector code points.
`GraphemeVectorBreaks`	Grapheme vector break masks: bit N set when a boundary precedes point N, the highest bit for the end of the row.
`GraphemeBreakRanges`	Grapheme break class ranges: first, last, class ordinal.
`PictographicRanges`	Extended_Pictographic ranges: first, last.
`ConjunctConsonantRanges`	InCB=Consonant ranges: first, last.
`ConjunctLinkerRanges`	InCB=Linker ranges: first, last.
`ConjunctExtendRanges`	InCB=Extend ranges: first, last.
`CanonicalDecompositionIndex`	Canonical decomposition index: code point, offset, length.
`CanonicalDecompositionElements`	The scalar values every canonical decomposition expands to, run together end to end.
`CompatibilityDecompositionIndex`	Compatibility decomposition index: code point, offset, length.
`CompatibilityDecompositionElements`	The scalar values every compatibility decomposition expands to, run together end to end.
`CompositionTriples`	Primary composite triples: starter, combining, composed.
`CategoryRanges`	General category ranges: first, last, category ordinal.
`CombiningClassRanges`	Canonical combining class ranges: first, last, class.
`WhiteSpaceRanges`	White_Space ranges: first, last.
`AlphabeticRanges`	Alphabetic ranges: first, last.
`NumericRanges`	Numeric ranges — the Nd, Nl and No categories: first, last.
`ScriptRanges`	Script ranges: first, last, script number.
`ScriptNameIndex`	Script names: offset into ScriptNameBytes and length, by script number.
`ScriptNameBytes`	The bytes of the scripts' long names, such as Latin or Old_Italic, all ASCII.
`UnknownScript`	The script number of Unknown, the script of a point in no range.
`HangulBase`	The first Hangul syllable, LV and LVT composition's base.

WriteFormat

Writes format into writer, with each {} replaced by the next argument.

GeneralCategory

The Unicode general category, one per scalar value.

Edit this page on GitHub