Regular expressions
Is this an order code? Where in the log is the first time of day? Questions like these can be answered with StartsWith, IndexOf and a loop over the bytes — and the loop grows a special case for every rule. A regular expression states the rules instead, as a short pattern: \d{2}:\d{2} is "two digits, a colon, two digits", and the Regex package does the searching.
import Regex::{ Escape, Regex, RegexError, RegexErrorKind };
Writing a pattern
Most characters in a pattern match themselves. A few have a meaning of their own:
| Pattern | Matches |
|---|---|
abc | the text abc |
. | any one character |
[A-Z] | one character from a class: here a capital letter |
\d, \w, \s | a digit, a word character, a space of any kind |
x*, x+, x? | zero or more, one or more, or an optional x |
x{4}, x{2,3} | exactly four, or two to three, of x |
a|b | a or b |
^, $ | the start and the end of the text |
\., \* | a character with a meaning, taken literally |
\d, \w and \s know Unicode, so \d matches an Arabic-Indic digit as well as 7, and (?i) at the start of a pattern makes it ignore case. The syntax is RE2's, the one Go and Rust use.
A pattern is written in a Rux string, where a backslash starts an escape of its own. So every backslash meant for the pattern is doubled:
let code = Regex::New(allocator, "^[A-Z]{3}-\\d{4}$")?;
Compiling once
Regex::New compiles the pattern into a Regex, which is then searched as many times as needed. It returns Regex ! RegexError: a pattern can be malformed, and that is found here, never in the middle of a search.
flowchart LR
p(["pattern text"]) --> n["`Regex::New`"]
n -- "well formed" --> r["`Regex`"]
n -- "malformed" --> e["`RegexError`<br/>kind and byte"]
r --> s["`IsMatch`, `Find`, `FindAt`<br/>as often as needed"]A RegexError has two fields: kind, a RegexErrorKind enum saying what went wrong, and offset, the byte of the pattern where it went wrong. The program turns a few kinds into words:
func Check(allocator: Allocator, pattern: char8[..]) {
match Regex::New(allocator, pattern) {
.Success(_) => PrintLine("{:<10} compiles", pattern),
.Failure(error) => PrintLine("{:<10} {} at byte {}", pattern, Reason(error.kind),
error.offset)
}
}
The last pattern it checks, (a)\1, is a backreference — "whatever group 1 matched, again". The package refuses it, and lookaround too, as RegexErrorKind::Unsupported. Without them a search never needs to go back and try again, so it takes time in proportion to the length of the text, whatever the pattern. A pattern that would make a backtracking matcher run for years, such as (a*)*b, is searched here as quickly as any other.
Testing: IsMatch
IsMatch answers whether the pattern matches anywhere in the text. It returns bool ! RegexError — even a search can fail, since it takes working memory from the regex's allocator — so it needs its ?:
for text in ["ABC-1234", "abc-1234", "ABC-12345", "XY-0001"] {
PrintLine("{:<10} {}", text, code.IsMatch(text)?);
}
The ^ and $ matter. Without them the pattern only has to match somewhere, and ABC-12345 would pass, since ABC-1234 is inside it. To check that a whole text has a form, anchor both ends.
Finding: Find and FindAt
Find returns where the leftmost match is, as a Match?. A Match holds two byte positions, start and end, and Slice cuts the matched text out of the text that was searched:
match time.Find(log)? {
found? => PrintLine("first {} at bytes {}..{}", found.Slice(log), found.start,
found.end),
none => PrintLine("no time")
}
FindAt takes a byte to start from. Starting each search where the previous match ended walks through every match in turn, and ?? break ends the loop when there are no more:
var at: uint = 0;
while true {
let found = time.FindAt(log, at)? ?? break;
PrintLine("time {}", found.Slice(log));
at = found.end;
}
The next lesson's FindAll does this walk for you, and handles one case this loop does not — see the common mistakes below.
Searching for plain text: Escape
Sometimes the thing to find is fixed text that happens to contain ., + or (. Written as a pattern, v1.2 also matches v102, because . is any character. Escape turns text into the pattern that matches exactly that text — here v1\.2 — and returns it as a String:
let pattern = Escape(allocator, "v1.2")?;
let exact = Regex::New(allocator, pattern.Bytes())?;
Use it whenever part of a pattern comes from a person or a file, so that their text cannot change what the pattern means.
The program
The whole lesson is one package in the Examples repository. Its comments explain every step.
// A regular expression is a small pattern that describes a whole family of texts. `\d{4}` is
// "four digits"; `^[A-Z]{3}-\d{4}$` is "three capital letters, a hyphen and four digits, and
// nothing else". One pattern can check what a person typed, or find what you are looking for in
// a page of text, where the same job by hand would be a loop full of special cases.
//
// `Regex::New` compiles a pattern once, and the compiled `Regex` is then searched as often as
// needed. Compiling can fail — the pattern may be malformed — so it returns `Regex ! RegexError`,
// and the error says what went wrong and at which byte of the pattern.
//
// The package matches in linear time: a search never takes more than a fixed amount of work per
// byte of text, whatever the pattern. The price is that it refuses the few features that would
// need backtracking, such as backreferences.
import Allocator::{ Allocator, SystemAllocator };
import Io::PrintLine;
import Regex::{ Escape, Regex, RegexError, RegexErrorKind };
func Reason(kind: RegexErrorKind) -> char8[..] {
return match kind {
.UnclosedGroup => "a ( with no )",
.UnclosedClass => "a [ with no ]",
.NothingToRepeat => "nothing before the repetition",
.Unsupported => "needs backtracking",
else => "another mistake"
};
}
// Compiles `pattern` only to see whether it is one, and reports why not.
func Check(allocator: Allocator, pattern: char8[..]) {
match Regex::New(allocator, pattern) {
.Success(_) => PrintLine("{:<10} compiles", pattern),
.Failure(error) => PrintLine("{:<10} {} at byte {}", pattern, Reason(error.kind),
error.offset)
}
}
func Main() -> ! RegexError {
var system = SystemAllocator();
let allocator: Allocator = system;
// A backslash in a Rux string must itself be escaped, so the pattern `\d` is written "\\d".
// Written as "\d", it never reaches the Regex package: the compiler stops at the string.
//
// error: escape sequence '\d' is not recognized
//
// `^` and `$` pin the match to the start and end of the text: without them, `IsMatch` would
// also accept a code with something before or after it.
let code = Regex::New(allocator, "^[A-Z]{3}-\\d{4}$")?;
for text in ["ABC-1234", "abc-1234", "ABC-12345", "XY-0001"] {
PrintLine("{:<10} {}", text, code.IsMatch(text)?);
}
// Even `IsMatch` can fail, since a search takes working memory from the allocator, so the `?`
// is needed. `if code.IsMatch(text) { … }` is refused with:
//
// error: condition for 'if' must have type 'bool', but found 'bool8 ! RegexError'
// `Find` answers with where the leftmost match is, as a `Match` of byte positions, or `none`.
// `Slice` cuts the matched text out of the text that was searched.
let log = "disk 91% full at 09:41; retry at 09:45, give up at 10:15";
let time = Regex::New(allocator, "\\d{2}:\\d{2}")?;
match time.Find(log)? {
found? => PrintLine("first {} at bytes {}..{}", found.Slice(log), found.start,
found.end),
none => PrintLine("no time")
}
// `FindAt` searches from a byte position onwards, so starting each search where the last
// match ended walks through every match in turn.
var at: uint = 0;
while true {
let found = time.FindAt(log, at)? ?? break;
PrintLine("time {}", found.Slice(log));
at = found.end;
}
// In a pattern `.` means "any character", so the pattern "v1.2" also matches "v102". `Escape`
// turns text into a pattern that matches exactly that text, here `v1\.2`.
let release = "v102 was replaced by v1.2";
let loose = Regex::New(allocator, "v1.2")?;
let pattern = Escape(allocator, "v1.2")?;
let exact = Regex::New(allocator, pattern.Bytes())?;
PrintLine("loose {}", (loose.Find(release)? ?? return).Slice(release));
PrintLine("exact {} from {}", (exact.Find(release)? ?? return).Slice(release), pattern);
// A malformed pattern is refused when it is compiled, never halfway through a search.
Check(allocator, "\\d+-\\d+");
Check(allocator, "(\\d+");
Check(allocator, "[a-z");
Check(allocator, "*.txt");
Check(allocator, "(a)\\1");
}
Besides Io, its Rux.toml lists Allocator and Regex under [Dependencies].
Run it
cd Examples/Text/Regex
rux run
ABC-1234 true
abc-1234 false
ABC-12345 false
XY-0001 false
first 09:41 at bytes 17..22
time 09:41
time 09:45
time 10:15
loose v102
exact v1.2 from v1\.2
\d+-\d+ compiles
(\d+ a ( with no ) at byte 0
[a-z a [ with no ] at byte 0
*.txt nothing before the repetition at byte 0
(a)\1 needs backtracking at byte 3
Common mistakes
Try it yourself
- Change the order-code pattern to accept lower-case letters too, first with
[A-Za-z], then with(?i)at the start. - Search the log for percentages with
\d+%. Where is the match, in bytes? - Use
Escapeon"1+1=2"and print the pattern it makes. Which characters gained a backslash? - Add
Check(allocator, "[z-a]")andCheck(allocator, "a{3,2}")to the program, with a reason for each new kind inReason.
Learn more
- Regex captures — groups, every match, splitting and replacing
- String view —
IndexOfandStartsWith, enough when the text is fixed - Coalesce exit — the
?? breakthat ends the search loop - Regex in the API reference — the full pattern syntax
- Escape sequences in the Rux Reference — why the backslashes are doubled
14.14 Lines and joining
Walk the lines of a text with Lines, put pieces back together with Join, and change the case of whole texts with ToUpper, ToLower and FoldCase, which follow Unicode's full mappings.
14.16 Regex captures
Pull the parts of a match out with numbered and named groups through Capture, walk every match with FindAll and the text between them with Split, and make new text with Replace and ReplaceFirst.