Files · Lesson 19.11

Memory streams

Source
Write binary records into memory with a ByteWriter and read them back with a ByteReader, using the same Writer and Reader code that also writes and reads a file, then test a cut-short input without damaging a file.

Every lesson in this part so far has moved bytes between a program and a file. But the code that decides which bytes — a record format, a header, a message — has nothing to do with files. In Binary, ReadUint16 and WriteUint16 take a Reader and a Writer, the interfaces any stream implements, and Buffered I/O slipped a buffer in between without the writing code noticing.

A memory stream is a Reader or a Writer whose bytes live in memory. With one, the same format code builds a message in memory, reads bytes that arrived some other way, and — most usefully — is tested without touching the disk:

import Io::{ ByteReader, ByteWriter, Endian, IoError, Print, PrintLine, ReadInt16, ReadUint16,
             Reader, WriteInt16, WriteUint16, Writer };

Code written against the interfaces

The program's format is a count, then each reading's sensor and temperature, two bytes each, little endian. The function that writes it takes a &var Writer — an interface parameter — so it neither knows nor cares where the bytes go:

func WriteReadings(out: &var Writer, readings: Reading[..]) -> ! IoError {
    WriteUint16(out, Endian::Little, readings.length as uint16)?;
    for reading in readings {
        WriteUint16(out, Endian::Little, reading.sensor)?;
        WriteInt16(out, Endian::Little, reading.tenths)?;
    }
}

ReadReadings is the other half, from a &var Reader. Neither function names a file or a memory stream; Main decides, by what it passes:

flowchart LR
    w["`WriteReadings`"] --> wi{"`&var Writer`"}
    wi --> bw["`ByteWriter`<br/>bytes in memory"]
    wi --> fw["`File`<br/>bytes on disk"]
    r["`ReadReadings`"] --> ri{"`&var Reader`"}
    ri --> br["`ByteReader`<br/>over a slice"]
    ri --> fr["`File`"]

Writing into memory

A ByteWriter keeps everything written in a block of memory that it grows as needed, doubling each time, so a long run of small writes stays cheap. It is made from an allocator, and Bytes() hands the bytes back as a char8[..] view:

var memory = ByteWriter::New(allocator);
WriteReadings(memory, readings[..])?;

Three readings make 14 bytes — the count, then three pairs — and the program prints them in hexadecimal: 03 00 is the count 3 with its low byte first, d7 00 is 215, the 21.5 °C of sensor 1.

MethodReturnsDoes
NewByteWriteran empty writer drawing on an allocator
Writeuint ! IoErrorappends, or overwrites after a seek back
Byteschar8[..]everything written, valid until the next change
Lengthuinthow many bytes have been written
Seekuint64 ! IoErrormoves where the next write lands, within what is written
Clear—forgets the bytes, keeping the block for reuse
Release—gives the block back; the destructor does this too

Reading from memory

A ByteReader reads a slice of bytes you already have — here the writer's own — from the start. It borrows them, so they must outlive it, and it fails with EndOfStream once they are used up, exactly like a file:

var reader = ByteReader::New(memory.Bytes());
ReadReadings(reader)?;

Position() tells how far it has read and Remaining() returns the bytes not yet read.

The same code, a file

Main then passes a File to the very same two functions. The file holds the same 14 bytes, and reads back the same three readings — the format code never changed.

That is the point of a memory stream: the format is proved right in memory, where a test is fast and leaves nothing behind, and then trusted with a file. The last part of the program goes further. A damaged input — a stream that ends in the middle of a reading — normally means writing a file and cutting it short, as Binary did. With a ByteReader, it is just a shorter slice:

var cut = ByteReader::New(memory.Bytes()[..9]);

The first reading comes through whole. The second has its sensor but only one byte of its temperature, so ReadInt16 fails with EndOfStream, and the read is refused rather than finished with a made-up value.

Seeking

Both streams can Seek within what they hold. A writer that seeks back overwrites in place, which is how a format whose header holds a count or a length is written when that number is only known at the end:

var memory = ByteWriter::New(allocator);
WriteAll(memory, "count=0;a;b;c")?;
// Go back and fix the count once it is known.
memory.Seek(SeekFrom::Start, 6)?;
WriteAll(memory, "3")?;
PrintLine("{}", memory.Bytes());

This prints count=3;a;b;c. A seek before the start or past the end of what is held fails with IoErrorKind::InvalidInput.

The program

The whole lesson is one package in the Examples repository. Its comments explain every step.

Src/Main.rux
// Code that reads and writes bytes is usually written against the `Reader` and `Writer`
// interfaces rather than against `File`, and the Binary lesson's `ReadUint16` and `WriteUint16`
// are written that way. A memory stream is a `Reader` or `Writer` whose bytes live in memory, so
// the same code can run without a file at all:
//
//     ByteWriter    a `Writer` that keeps everything written in a block it grows as needed;
//                   `Bytes()` hands the bytes back
//     ByteReader    a `Reader` over a slice of bytes you already have; at the end it fails
//                   with `EndOfStream`, like a file
//
// That is how a format is tested without touching the disk, how a message is built in memory
// before it is sent in one piece, and how bytes that arrived some other way — over a network,
// embedded in the program — are read with the code that reads files.
//
// The file this program also writes lives in its package's `Bin/` folder, and is deleted at
// the end.
import Allocator::{ Allocator, SystemAllocator };
import FileSystem::{ DeleteFile, File, OpenOptions };
import Io::{ ByteReader, ByteWriter, Endian, IoError, Print, PrintLine, ReadInt16, ReadUint16,
             Reader, WriteInt16, WriteUint16, Writer };
import Path::{ OsString, Path };
import Text::TextError;

// A sensor's reading, in tenths of a degree: 215 is 21.5 °C.
struct Reading {
    sensor: uint16;
    tenths: int16;
}

// The format: a count, then each reading's sensor and temperature, two bytes each, little endian.
// It takes any `Writer`, so it neither knows nor cares where the bytes go.
func WriteReadings(out: &var Writer, readings: Reading[..]) -> ! IoError {
    WriteUint16(out, Endian::Little, readings.length as uint16)?;
    for reading in readings {
        WriteUint16(out, Endian::Little, reading.sensor)?;
        WriteInt16(out, Endian::Little, reading.tenths)?;
    }
}

// The other half of the format, from any `Reader`. A stream that ends part-way fails it.
func ReadReadings(input: &var Reader) -> ! IoError {
    let count = ReadUint16(input, Endian::Little)?;
    for _ in 0..count {
        let sensor = ReadUint16(input, Endian::Little)?;
        let tenths = ReadInt16(input, Endian::Little)?;
        PrintLine("  sensor {}  {:.1} C", sensor, tenths as float64 / 10.0);
    }
}

func Main() -> ! IoError | TextError {
    var system = SystemAllocator();
    let allocator: Allocator = system;
    let readings: Reading[3] = [Reading { sensor: 1, tenths: 215 },
                                Reading { sensor: 2, tenths: 198 },
                                Reading { sensor: 7, tenths: 230 }];

    // Into memory. The writer grows its block as bytes arrive.
    var memory = ByteWriter::New(allocator);
    WriteReadings(memory, readings[..])?;
    Print("memory: {} bytes:", memory.Length());
    for byte in memory.Bytes() {
        Print(" {:02x}", byte as uint8);
    }
    PrintLine();

    // Back out of memory, with the code that would read a file. Reading moves the reader's
    // position along, so it must be `var`; a `let` reader is refused:
    //
    //     error: argument 1 to 'ReadReadings' cannot borrow immutable 'reader' as
    //            '&var rux/io::Reader'
    var reader = ByteReader::New(memory.Bytes());
    ReadReadings(reader)?;

    // The same two functions, given a file.
    var holder = OsString::FromText(allocator, "Bin/readings.bin")?;
    let path = Path::FromView(holder.View());
    var writing = File::Open(allocator, path, OpenOptions::Writing())?;
    WriteReadings(writing, readings[..])?;
    writing.Close()?;
    var reading = File::Open(allocator, path, OpenOptions::Reading())?;
    PrintLine("file:   {} bytes", reading.Size()?);
    ReadReadings(reading)?;
    reading.Close()?;
    DeleteFile(allocator, path)?;

    // Damaged input, made without damaging a file: only the first 9 bytes of the 14.
    var cut = ByteReader::New(memory.Bytes()[..9]);
    PrintLine("cut:    9 bytes");
    ReadReadings(cut) catch {
        error if error.IsEnd() => {
            PrintLine("  ended inside a reading: refused");
        },
        error => fail error
    };
}

Besides Io, its Rux.toml lists Allocator, FileSystem, Path and Text under [Dependencies].

Run it

cd Examples/Files/MemoryStream
rux run
memory: 14 bytes: 03 00 01 00 d7 00 02 00 c6 00 07 00 e6 00
  sensor 1  21.5 C
  sensor 2  19.8 C
  sensor 7  23.0 C
file:   14 bytes
  sensor 1  21.5 C
  sensor 2  19.8 C
  sensor 7  23.0 C
cut:    9 bytes
  sensor 1  21.5 C
  ended inside a reading: refused

Common mistakes

Try it yourself

  1. Write the readings into a ByteWriter, then write the same count and readings a second time, and read both copies back with one ByteReader.
  2. Give the format a header: write a placeholder count of 0, the readings, then Seek back to the start and write the real count, as in the seeking example.
  3. Read from a ByteReader over only the first two bytes, memory.Bytes()[..2]: a count of 3 with no readings after it. Which call fails, and with what?
  4. Change Endian::Little to Endian::Big in both functions and compare the printed bytes.

Learn more