v0.4.0
Data formats · Lesson 21.3

Streaming JSON

Source
Read JSON as a stream of events with JsonEventReader, keeping only the fields you want and stopping cleanly on a broken document.

Parse builds the whole tree before your program sees any of it. For a settings file that is ideal. For a log of a million records it is not: the tree costs several times the document's size in memory, and a program that only wants one field from each record pays for all of them.

JsonEventReader reads the same text as a stream of events instead — an object opened, a name, a text value, an array closed — pulling bytes from a source as it needs them and keeping only the token it is on. The program keeps whatever it wants and lets the rest go by. This lesson prints the title of every book in a catalogue without ever building a tree.

import Io::{ IoError, IoErrorKind, PrintLine, Reader };
import Json::{ JsonEvent, JsonEventReader, JsonLimits, JsonReadError };

A source of bytes

The event reader reads from any Reader — the interface that files and network connections implement. The lesson stands one in with TextSource, which serves a string at most 8 bytes per read, so tokens arrive split across reads just as they would from a real stream:

struct TextSource {
    text: char8[..];
    offset: uint;
}

Its Read method copies up to 8 bytes into the buffer it is given and returns how many, or throws an IoError of kind EndOfStream when nothing is left. Inside Titles, the reader is built on the source. The reader keeps it for as long as it lives, so it holds a Reader handle, made from the source's address inside unsafe: the handle copies nothing, and the unsafe block is where the function promises that source outlives the reader, which it does, being declared first:

var source = TextSource { text: document, offset: 0 };
// Safety: `source` is a local declared before `reader`, so it outlives it, and is never moved.
var reader = JsonEventReader(allocator, unsafe { @source as Reader }, JsonLimits::Default());

The constructor allocates the reader's buffer. JsonLimits::Default() sets the limits a document must stay within — its size, the length of one string, how deeply it nests — generous for ordinary documents and well short of what would exhaust the machine.

Events

Each call to Next() returns the next JsonEvent, and Depth() says how many arrays and objects are open. The first book of the catalogue produces:

TokenEventDepth()Text()
[ArrayStart1
{ObjectStart2
"title"Name2title
"Dune"Text2Dune
"year"Name2year
1965Number21965
}ObjectEnd1

…and at the very end, ArrayEnd. Commas and colons produce no events, and True, False and Null complete the set. There is no event for the end of the document and none for a mistake. Next is declared:

Next(&var self) -> JsonEvent? throws JsonReadError

It answers none once the document is over, and throws when the document goes wrong.

The loop

Titles asks for events until there are none, counts each object one level inside the top-level array as a book, and prints the text that follows a title name:

var books: uint = 0;
var wantTitle = false;
while (try reader.Next()) is .Some(event) {
    // Each book is an object one level inside the top-level array.
    if event == JsonEvent::ObjectStart && reader.Depth() == 2 {
        books += 1;
    }
    // Print the title now: its text is gone once `Next()` is called again.
    if event == JsonEvent::Text && wantTitle {
        PrintLine("  title: {}", reader.Text());
    }
    wantTitle = event == JsonEvent::Name && IsTitle(reader.Text());
}
return books;

The loop condition does three jobs. Plain try passes a failure on to the caller of Titles; is .Some(event) binds the event for the body; and when Next() answers none, the test fails and the loop ends. The parentheses are needed because try covers everything to its right — without them it would take in the is test too.

Order inside a book does not matter: Emma's year comes before its title, and the title is still found, because the program reacts to whatever name it meets.

Two jobs that move onto the program

Streaming saves memory by keeping almost nothing, and that hands two jobs to the program.

Lifetimes. Text() borrows the reader's buffer, and the next Next() reuses it. So the text of an event must be used — or copied — before asking for another. That is why the title is printed the moment it arrives, and why wantTitle remembers only a bool, not the name's text. The buffer itself belongs to the reader, and the reader's destructor gives it back however Titles ends: at the return, or with an error passed on from Next().

Errors arrive late. A stream cannot know that the end of a document is wrong until it gets there. When it does, Next() throws, and the error says why and at which byte of the whole document — however it was split into reads. By then the earlier events have already been delivered:

flowchart LR
    n["`Next()`"] --> e{"Answer?"}
    e -- "`ObjectStart`, `Name`,<br/>`Text`, …" --> use["use it now"] --> n
    e -- "`none`" --> ok(["`books`"])
    e -- "throws" --> f(["`JsonReadError`"])

In the output, both broken catalogues print their titles before being refused. A program that streams must be ready to discover, late, that its input was bad — and must not have committed anything it cannot take back.

Two kinds of failure

Titles is declared -> uint throws JsonReadError, a variant with one case per way reading can stop:

JsonReadErrorMeans
Malformed(JsonParseError)the document is not JSON; the parse error says why and at which byte
Source(IoError, uint)the source failed with something other than its end, after so many bytes

Report calls Titles in a try block and takes the cases apart in its catch arms:

try {
    let books = Titles(allocator, document);
    PrintLine("  {} books", books);
} catch {
    .Malformed(reason) => PrintLine("  refused at byte {}: {}", reason.Offset(), reason),
    .Source(_, read) => PrintLine("  the source failed after {} bytes", read)
}

The Malformed case carries the same JsonParseError, with the same offsets, that Parse would report for the same text.

The program

The whole lesson is one package in the Examples repository. Its comments explain every step.

Src/Main.rux
// `Parse` builds the whole tree before a program sees any of it, so a large document costs its
// size several times over in memory. `JsonEventReader` reads the same text as a stream of
// events instead — an object opened, a name, a text value, an array closed — pulling bytes from
// a `Reader` as it needs them and keeping only the token it is on. The program keeps whatever it
// wants and lets the rest go by.
//
// Streaming moves two jobs onto the program:
//
//   - Lifetimes. `Text()` borrows the reader's buffer, which the next `Next()` reuses, so the text
//     of an event must be used or copied before asking for another. The reader owns that buffer,
//     and its destructor gives it back however the function holding it ends.
//   - Errors. `Next()` answers `none` at the end of the document and throws a `JsonReadError`
//     when the document goes wrong. Its `Malformed` case holds the `JsonParseError` that says why
//     and at which byte of the whole document, however it was split into reads; a source that
//     fails is its `Source` case, with the source's own `IoError` and how far it got. Events
//     before the mistake were already delivered, so a program must be ready to find out late
//     that its input was bad.
import Allocator::{ Allocator, SystemAllocator };
import Io::{ IoError, IoErrorKind, PrintLine, Reader };
import Json::{ JsonEvent, JsonEventReader, JsonLimits, JsonReadError };
import Text::StringView;

// Stands in for a file or a socket: serves its text at most 8 bytes per read, so tokens arrive
// split across reads, as they would from a real stream.
struct TextSource {
    text: char8[..];
    offset: uint;
}

extend TextSource : Reader {
    func Read(&var self, into: var char8[..]) -> uint throws IoError {
        let remaining = self.text.length - self.offset;
        if remaining == 0 {
            throw IoError::Of(IoErrorKind::EndOfStream);
        }
        var count: uint = 8;
        if remaining < count { count = remaining; }
        if into.length < count { count = into.length; }
        for i in 0..count {
            into[i] = self.text[self.offset + i];
        }
        self.offset += count;
        return count;
    }
}

func IsTitle(text: char8[..]) -> bool {
    let title = StringView::FromValidated("title");
    return StringView::FromValidated(text).Equals(title);
}

// Prints the title of every book and counts the books, without ever building a tree.
func Titles(allocator: Allocator, document: char8[..]) -> uint throws JsonReadError {
    var source = TextSource { text: document, offset: 0 };
    // The reader keeps a `Reader` handle that addresses `source`, so `source` has to outlive it.
    // The reader's buffer lives as long as `reader` does. Whether this function ends at the
    // `return` or with an error passed on from `Next()`, the destructor frees it on the way out.
    // Safety: `source` is a local declared before `reader`, so it outlives it, and is never moved.
    var reader = JsonEventReader(allocator, unsafe { @source as Reader }, JsonLimits::Default());
    var books: uint = 0;
    var wantTitle = false;
    // `try` passes a failure on to the caller; the loop ends when `Next()` answers `none`.
    while (try reader.Next()) is .Some(event) {
        // Each book is an object one level inside the top-level array.
        if event == JsonEvent::ObjectStart && reader.Depth() == 2 {
            books += 1;
        }
        // Print the title now: its text is gone once `Next()` is called again.
        if event == JsonEvent::Text && wantTitle {
            PrintLine("  title: {}", reader.Text());
        }
        wantTitle = event == JsonEvent::Name && IsTitle(reader.Text());
    }
    return books;
}

func Report(allocator: Allocator, document: char8[..]) {
    PrintLine("{}", document);
    try {
        let books = Titles(allocator, document);
        PrintLine("  {} books", books);
    } catch {
        .Malformed(reason) => PrintLine("  refused at byte {}: {}", reason.Offset(), reason),
        .Source(_, read) => PrintLine("  the source failed after {} bytes", read)
    }
}

func Main() -> int {
    let allocator = SystemAllocator();
    let catalog = "[{\"title\": \"Dune\", \"year\": 1965}, {\"year\": 1815, \"title\": \"Emma\"}]";
    Report(allocator, catalog);

    // Two broken documents. In both, titles were printed before the mistake was found: a
    // stream cannot know the end is wrong until it gets there.
    Report(allocator, "[{\"title\": \"Dune\"}, {\"title\": \"Emma\"]");
    Report(allocator, "[{\"title\": \"Dune\"}, {\"title\":");
    return 0;
}

Besides Io, its Rux.toml lists Allocator, Json and Text under [Dependencies].

Run it

cd Examples/DataFormats/JsonStream
rux run
[{"title": "Dune", "year": 1965}, {"year": 1815, "title": "Emma"}]
  title: Dune
  title: Emma
  2 books
[{"title": "Dune"}, {"title": "Emma"]
  title: Dune
  title: Emma
  refused at byte 36: token that cannot appear here
[{"title": "Dune"}, {"title":
  title: Dune
  refused at byte 29: document ended in the middle of a value

Common mistakes

Try it yourself

  1. Print each book's year as well: react to a Number event that follows a year name.
  2. Change TextSource to serve one byte per read. Does any line of the output change — including the byte offsets?
  3. Add a third book that has no title. Does the count still include it?
  4. Count how many events the first catalogue produces in all.

Learn more

  • JSON — the tree parser, for documents small enough to hold at once
  • Interface value — how TextSource becomes a Reader
  • Destructor — how the reader's buffer is returned on every path
  • Error variant and Catch — the cases of JsonReadError and how Report splits them
  • Buffered I/O — reading a real file a block at a time