v0.4.0
Files · Lesson 19.11

Memory streams

Source
Write binary records into memory with a ByteWriter and read them back with a ByteReader, using the same Writer and Reader code that also writes and reads a file, then test a cut-short input without damaging a file.

Every lesson in this part so far has moved bytes between a program and a file. But the code that decides which bytes — a record format, a header, a message — has nothing to do with files. In Binary, ReadUint16 and WriteUint16 take a Reader and a Writer, the interfaces any stream implements, and Buffered I/O slipped a buffer in between without the writing code noticing.

A memory stream is a Reader or a Writer whose bytes live in memory. With one, the same format code builds a message in memory, reads bytes that arrived some other way, and — most usefully — is tested without touching the disk:

import Io::{ ByteReader, ByteWriter, Endian, IoError, Print, PrintLine, ReadInt16, ReadUint16,
             Reader, WriteInt16, WriteUint16, Writer };

Code written against the interfaces

The program's format is a count, then each reading's sensor and temperature, two bytes each, little endian. The function that writes it takes a &var Writer — an interface parameter — so it neither knows nor cares where the bytes go:

func WriteReadings(out: &var Writer, readings: Reading[..]) throws IoError {
    try WriteUint16(out, Endian::Little, readings.length as uint16);
    for reading in readings {
        try WriteUint16(out, Endian::Little, reading.sensor);
        try WriteInt16(out, Endian::Little, reading.tenths);
    }
}

ReadReadings is the other half, from a &var Reader. Neither function names a file or a memory stream; Main decides, by what it passes:

flowchart LR
    w["`WriteReadings`"] --> wi{"`&var Writer`"}
    wi --> bw["`ByteWriter`<br/>bytes in memory"]
    wi --> fw["`File`<br/>bytes on disk"]
    r["`ReadReadings`"] --> ri{"`&var Reader`"}
    ri --> br["`ByteReader`<br/>over a slice"]
    ri --> fr["`File`"]

Writing into memory

A ByteWriter keeps everything written in a block of memory that it grows as needed, doubling each time, so a long run of small writes stays cheap. It is made from an allocator, and Bytes() hands the bytes back as a char8[..] view:

var memory = ByteWriter(allocator);
try WriteReadings(memory, readings[..]);

Three readings make 14 bytes — the count, then three pairs — and the program prints them in hexadecimal: 03 00 is the count 3 with its low byte first, d7 00 is 215, the 21.5 °C of sensor 1.

MethodReturnsDoes
ByteWriterByteWriteran empty writer drawing on an allocator
Writeuintappends, or overwrites after a seek back; throws IoError
Byteschar8[..]everything written, valid until the next change
Lengthuinthow many bytes have been written
Seekuint64moves where the next write lands, within what is written; throws
Clear—forgets the bytes, keeping the block for reuse
Release—gives the block back; the destructor does this too

Reading from memory

A ByteReader reads a slice of bytes you already have — here the writer's own — from the start. It borrows them, so they must outlive it, and it throws EndOfStream once they are used up, exactly like a file:

var reader = ByteReader(memory.Bytes());
try ReadReadings(reader);

Position() tells how far it has read and Remaining() returns the bytes not yet read.

The same code, a file

Main then passes a File to the very same two functions. The file holds the same 14 bytes, and reads back the same three readings — the format code never changed.

That is the point of a memory stream: the format is proved right in memory, where a test is fast and leaves nothing behind, and then trusted with a file. The last part of the program goes further. A damaged input — a stream that ends in the middle of a reading — normally means writing a file and cutting it short, as Binary did. With a ByteReader, it is just a shorter slice:

var cut = ByteReader(memory.Bytes()[..9]);

The first reading comes through whole. The second has its sensor but only one byte of its temperature, so ReadInt16 throws UnexpectedEnd. The program's catch picks that kind out with a guard, as Binary did, and the read is refused rather than finished with a made-up value.

Seeking

Both streams can Seek within what they hold. A writer that seeks back overwrites in place, which is how a format whose header holds a count or a length is written when that number is only known at the end:

var memory = ByteWriter(allocator);
try WriteAll(memory, "count=0;a;b;c");
// Go back and fix the count once it is known.
try memory.Seek(SeekFrom::Start, 6);
try WriteAll(memory, "3");
PrintLine("{}", memory.Bytes());

This prints count=3;a;b;c. A seek before the start or past the end of what is held throws an error of kind IoErrorKind::InvalidInput.

The program

The whole lesson is one package in the Examples repository. Its comments explain every step.

Src/Main.rux
// Code that reads and writes bytes is usually written against the `Reader` and `Writer`
// interfaces rather than against `File`, and the Binary lesson's `ReadUint16` and `WriteUint16`
// are written that way. A memory stream is a `Reader` or `Writer` whose bytes live in memory, so
// the same code can run without a file at all:
//
//     ByteWriter    a `Writer` that keeps everything written in a block it grows as needed;
//                   `Bytes()` hands the bytes back
//     ByteReader    a `Reader` over a slice of bytes you already have; at the end it throws
//                   `EndOfStream`, like a file
//
// That is how a format is tested without touching the disk, how a message is built in memory
// before it is sent in one piece, and how bytes that arrived some other way — over a network,
// embedded in the program — are read with the code that reads files.
//
// The file this program also writes lives in its package's `Bin/` folder, and is deleted at
// the end.
import Allocator::SystemAllocator;
import FileSystem::{ DeleteFile, File, OpenOptions };
import Io::{ ByteReader, ByteWriter, Endian, IoError, IoErrorKind, Print, PrintLine, ReadInt16, ReadUint16,
             Reader, WriteInt16, WriteUint16, Writer };
import Path::{ OsString, Path };

// A sensor's reading, in tenths of a degree: 215 is 21.5 °C.
struct Reading {
    sensor: uint16;
    tenths: int16;
}

// The format: a count, then each reading's sensor and temperature, two bytes each, little endian.
// It takes any `Writer`, so it neither knows nor cares where the bytes go.
func WriteReadings(out: &var Writer, readings: Reading[..]) throws IoError {
    try WriteUint16(out, Endian::Little, readings.length as uint16);
    for reading in readings {
        try WriteUint16(out, Endian::Little, reading.sensor);
        try WriteInt16(out, Endian::Little, reading.tenths);
    }
}

// The other half of the format, from any `Reader`. A stream that ends part-way through a value
// throws `UnexpectedEnd`.
func ReadReadings(input: &var Reader) throws IoError {
    let count = try ReadUint16(input, Endian::Little);
    for _ in 0..count {
        let sensor = try ReadUint16(input, Endian::Little);
        let tenths = try ReadInt16(input, Endian::Little);
        PrintLine("  sensor {}  {:.1} C", sensor, tenths as float64 / 10.0);
    }
}

func Main() throws IoError {
    let allocator = SystemAllocator();
    let readings: Reading[3] = [Reading { sensor: 1, tenths: 215 },
                                Reading { sensor: 2, tenths: 198 },
                                Reading { sensor: 7, tenths: 230 }];

    // Into memory. The writer grows its block as bytes arrive.
    var memory = ByteWriter(allocator);
    try WriteReadings(memory, readings[..]);
    Print("memory: {} bytes:", memory.Length());
    for byte in memory.Bytes() {
        Print(" {:02x}", byte as uint8);
    }
    PrintLine();

    // Back out of memory, with the code that would read a file. Reading moves the reader's
    // position along, so it must be `var`; a `let` reader is refused:
    //
    //     error: argument 1 to 'ReadReadings' cannot borrow immutable 'reader' as
    //            '&var Rux/Io::Reader'
    //     help: declare 'reader' with 'var' to make it mutable
    var reader = ByteReader(memory.Bytes());
    try ReadReadings(reader);

    // The same two functions, given a file.
    var holder = OsString::FromText(allocator, "Bin/readings.bin");
    let path = Path::FromView(holder.View());
    var writing = try File::Open(allocator, path, OpenOptions::Writing());
    try WriteReadings(writing, readings[..]);
    try writing.Close();
    var reading = try File::Open(allocator, path, OpenOptions::Reading());
    try PrintLine("file:   {} bytes", reading.Size());
    try ReadReadings(reading);
    try reading.Close();
    try DeleteFile(allocator, path);

    // Damaged input, made without damaging a file: only the first 9 bytes of the 14.
    var cut = ByteReader(memory.Bytes()[..9]);
    PrintLine("cut:    9 bytes");
    try ReadReadings(cut) catch {
        error if error.kind == IoErrorKind::UnexpectedEnd => {
            PrintLine("  ended inside a reading: refused");
        },
        error => throw error
    };
}

Besides Io, its Rux.toml lists Allocator, FileSystem and Path under [Dependencies].

Run it

cd Examples/Files/MemoryStream
rux run
memory: 14 bytes: 03 00 01 00 d7 00 02 00 c6 00 07 00 e6 00
  sensor 1  21.5 C
  sensor 2  19.8 C
  sensor 7  23.0 C
file:   14 bytes
  sensor 1  21.5 C
  sensor 2  19.8 C
  sensor 7  23.0 C
cut:    9 bytes
  sensor 1  21.5 C
  ended inside a reading: refused

Common mistakes

Try it yourself

  1. Write the readings into a ByteWriter, then write the same count and readings a second time, and read both copies back with one ByteReader.
  2. Give the format a header: write a placeholder count of 0, the readings, then Seek back to the start and write the real count, as in the seeking example.
  3. Read from a ByteReader over only the first two bytes, memory.Bytes()[..2]: a count of 3 with no readings after it. Which call throws, and with which kind?
  4. Change Endian::Little to Endian::Big in both functions and compare the printed bytes.

Learn more