Text · Lesson 14.15

Regular expressions

Source
Compile a regular expression with Regex::New, which returns Regex ! RegexError, test text with IsMatch, find where a match is with Find and FindAt, and turn plain text into a pattern with Escape.

Is this an order code? Where in the log is the first time of day? Questions like these can be answered with StartsWith, IndexOf and a loop over the bytes — and the loop grows a special case for every rule. A regular expression states the rules instead, as a short pattern: \d{2}:\d{2} is "two digits, a colon, two digits", and the Regex package does the searching.

import Regex::{ Escape, Regex, RegexError, RegexErrorKind };

Writing a pattern

Most characters in a pattern match themselves. A few have a meaning of their own:

PatternMatches
abcthe text abc
.any one character
[A-Z]one character from a class: here a capital letter
\d, \w, \sa digit, a word character, a space of any kind
x*, x+, x?zero or more, one or more, or an optional x
x{4}, x{2,3}exactly four, or two to three, of x
a|ba or b
^, $the start and the end of the text
\., \*a character with a meaning, taken literally

\d, \w and \s know Unicode, so \d matches an Arabic-Indic digit as well as 7, and (?i) at the start of a pattern makes it ignore case. The syntax is RE2's, the one Go and Rust use.

A pattern is written in a Rux string, where a backslash starts an escape of its own. So every backslash meant for the pattern is doubled:

let code = Regex::New(allocator, "^[A-Z]{3}-\\d{4}$")?;

Compiling once

Regex::New compiles the pattern into a Regex, which is then searched as many times as needed. It returns Regex ! RegexError: a pattern can be malformed, and that is found here, never in the middle of a search.

flowchart LR
    p(["pattern text"]) --> n["`Regex::New`"]
    n -- "well formed" --> r["`Regex`"]
    n -- "malformed" --> e["`RegexError`<br/>kind and byte"]
    r --> s["`IsMatch`, `Find`, `FindAt`<br/>as often as needed"]

A RegexError has two fields: kind, a RegexErrorKind enum saying what went wrong, and offset, the byte of the pattern where it went wrong. The program turns a few kinds into words:

func Check(allocator: Allocator, pattern: char8[..]) {
    match Regex::New(allocator, pattern) {
        .Success(_) => PrintLine("{:<10} compiles", pattern),
        .Failure(error) => PrintLine("{:<10} {} at byte {}", pattern, Reason(error.kind),
            error.offset)
    }
}

The last pattern it checks, (a)\1, is a backreference — "whatever group 1 matched, again". The package refuses it, and lookaround too, as RegexErrorKind::Unsupported. Without them a search never needs to go back and try again, so it takes time in proportion to the length of the text, whatever the pattern. A pattern that would make a backtracking matcher run for years, such as (a*)*b, is searched here as quickly as any other.

Testing: IsMatch

IsMatch answers whether the pattern matches anywhere in the text. It returns bool ! RegexError — even a search can fail, since it takes working memory from the regex's allocator — so it needs its ?:

for text in ["ABC-1234", "abc-1234", "ABC-12345", "XY-0001"] {
    PrintLine("{:<10} {}", text, code.IsMatch(text)?);
}

The ^ and $ matter. Without them the pattern only has to match somewhere, and ABC-12345 would pass, since ABC-1234 is inside it. To check that a whole text has a form, anchor both ends.

Finding: Find and FindAt

Find returns where the leftmost match is, as a Match?. A Match holds two byte positions, start and end, and Slice cuts the matched text out of the text that was searched:

match time.Find(log)? {
    found? => PrintLine("first     {} at bytes {}..{}", found.Slice(log), found.start,
        found.end),
    none => PrintLine("no time")
}

FindAt takes a byte to start from. Starting each search where the previous match ended walks through every match in turn, and ?? break ends the loop when there are no more:

var at: uint = 0;
while true {
    let found = time.FindAt(log, at)? ?? break;
    PrintLine("time      {}", found.Slice(log));
    at = found.end;
}

The next lesson's FindAll does this walk for you, and handles one case this loop does not — see the common mistakes below.

Searching for plain text: Escape

Sometimes the thing to find is fixed text that happens to contain ., + or (. Written as a pattern, v1.2 also matches v102, because . is any character. Escape turns text into the pattern that matches exactly that text — here v1\.2 — and returns it as a String:

let pattern = Escape(allocator, "v1.2")?;
let exact = Regex::New(allocator, pattern.Bytes())?;

Use it whenever part of a pattern comes from a person or a file, so that their text cannot change what the pattern means.

The program

The whole lesson is one package in the Examples repository. Its comments explain every step.

Src/Main.rux
// A regular expression is a small pattern that describes a whole family of texts. `\d{4}` is
// "four digits"; `^[A-Z]{3}-\d{4}$` is "three capital letters, a hyphen and four digits, and
// nothing else". One pattern can check what a person typed, or find what you are looking for in
// a page of text, where the same job by hand would be a loop full of special cases.
//
// `Regex::New` compiles a pattern once, and the compiled `Regex` is then searched as often as
// needed. Compiling can fail — the pattern may be malformed — so it returns `Regex ! RegexError`,
// and the error says what went wrong and at which byte of the pattern.
//
// The package matches in linear time: a search never takes more than a fixed amount of work per
// byte of text, whatever the pattern. The price is that it refuses the few features that would
// need backtracking, such as backreferences.
import Allocator::{ Allocator, SystemAllocator };
import Io::PrintLine;
import Regex::{ Escape, Regex, RegexError, RegexErrorKind };

func Reason(kind: RegexErrorKind) -> char8[..] {
    return match kind {
        .UnclosedGroup => "a ( with no )",
        .UnclosedClass => "a [ with no ]",
        .NothingToRepeat => "nothing before the repetition",
        .Unsupported => "needs backtracking",
        else => "another mistake"
    };
}

// Compiles `pattern` only to see whether it is one, and reports why not.
func Check(allocator: Allocator, pattern: char8[..]) {
    match Regex::New(allocator, pattern) {
        .Success(_) => PrintLine("{:<10} compiles", pattern),
        .Failure(error) => PrintLine("{:<10} {} at byte {}", pattern, Reason(error.kind),
            error.offset)
    }
}

func Main() -> ! RegexError {
    var system = SystemAllocator();
    let allocator: Allocator = system;

    // A backslash in a Rux string must itself be escaped, so the pattern `\d` is written "\\d".
    // Written as "\d", it never reaches the Regex package: the compiler stops at the string.
    //
    //     error: escape sequence '\d' is not recognized
    //
    // `^` and `$` pin the match to the start and end of the text: without them, `IsMatch` would
    // also accept a code with something before or after it.
    let code = Regex::New(allocator, "^[A-Z]{3}-\\d{4}$")?;
    for text in ["ABC-1234", "abc-1234", "ABC-12345", "XY-0001"] {
        PrintLine("{:<10} {}", text, code.IsMatch(text)?);
    }

    // Even `IsMatch` can fail, since a search takes working memory from the allocator, so the `?`
    // is needed. `if code.IsMatch(text) { … }` is refused with:
    //
    //     error: condition for 'if' must have type 'bool', but found 'bool8 ! RegexError'

    // `Find` answers with where the leftmost match is, as a `Match` of byte positions, or `none`.
    // `Slice` cuts the matched text out of the text that was searched.
    let log = "disk 91% full at 09:41; retry at 09:45, give up at 10:15";
    let time = Regex::New(allocator, "\\d{2}:\\d{2}")?;
    match time.Find(log)? {
        found? => PrintLine("first     {} at bytes {}..{}", found.Slice(log), found.start,
            found.end),
        none => PrintLine("no time")
    }

    // `FindAt` searches from a byte position onwards, so starting each search where the last
    // match ended walks through every match in turn.
    var at: uint = 0;
    while true {
        let found = time.FindAt(log, at)? ?? break;
        PrintLine("time      {}", found.Slice(log));
        at = found.end;
    }

    // In a pattern `.` means "any character", so the pattern "v1.2" also matches "v102". `Escape`
    // turns text into a pattern that matches exactly that text, here `v1\.2`.
    let release = "v102 was replaced by v1.2";
    let loose = Regex::New(allocator, "v1.2")?;
    let pattern = Escape(allocator, "v1.2")?;
    let exact = Regex::New(allocator, pattern.Bytes())?;
    PrintLine("loose     {}", (loose.Find(release)? ?? return).Slice(release));
    PrintLine("exact     {} from {}", (exact.Find(release)? ?? return).Slice(release), pattern);

    // A malformed pattern is refused when it is compiled, never halfway through a search.
    Check(allocator, "\\d+-\\d+");
    Check(allocator, "(\\d+");
    Check(allocator, "[a-z");
    Check(allocator, "*.txt");
    Check(allocator, "(a)\\1");
}

Besides Io, its Rux.toml lists Allocator and Regex under [Dependencies].

Run it

cd Examples/Text/Regex
rux run
ABC-1234   true
abc-1234   false
ABC-12345  false
XY-0001    false
first     09:41 at bytes 17..22
time      09:41
time      09:45
time      10:15
loose     v102
exact     v1.2 from v1\.2
\d+-\d+    compiles
(\d+       a ( with no ) at byte 0
[a-z       a [ with no ] at byte 0
*.txt      nothing before the repetition at byte 0
(a)\1      needs backtracking at byte 3

Common mistakes

Try it yourself

  1. Change the order-code pattern to accept lower-case letters too, first with [A-Za-z], then with (?i) at the start.
  2. Search the log for percentages with \d+%. Where is the match, in bytes?
  3. Use Escape on "1+1=2" and print the pattern it makes. Which characters gained a backslash?
  4. Add Check(allocator, "[z-a]") and Check(allocator, "a{3,2}") to the program, with a reason for each new kind in Reason.

Learn more

  • Regex captures — groups, every match, splitting and replacing
  • String view — IndexOf and StartsWith, enough when the text is fixed
  • Coalesce exit — the ?? break that ends the search loop
  • Regex in the API reference — the full pattern syntax
  • Escape sequences in the Rux Reference — why the backslashes are doubled