Memory streams
Every lesson in this part so far has moved bytes between a program and a file. But the code that decides which bytes — a record format, a header, a message — has nothing to do with files. In Binary, ReadUint16 and WriteUint16 take a Reader and a Writer, the interfaces any stream implements, and Buffered I/O slipped a buffer in between without the writing code noticing.
A memory stream is a Reader or a Writer whose bytes live in memory. With one, the same format code builds a message in memory, reads bytes that arrived some other way, and — most usefully — is tested without touching the disk:
import Io::{ ByteReader, ByteWriter, Endian, IoError, Print, PrintLine, ReadInt16, ReadUint16,
Reader, WriteInt16, WriteUint16, Writer };
Code written against the interfaces
The program's format is a count, then each reading's sensor and temperature, two bytes each, little endian. The function that writes it takes a &var Writer — an interface parameter — so it neither knows nor cares where the bytes go:
func WriteReadings(out: &var Writer, readings: Reading[..]) -> ! IoError {
WriteUint16(out, Endian::Little, readings.length as uint16)?;
for reading in readings {
WriteUint16(out, Endian::Little, reading.sensor)?;
WriteInt16(out, Endian::Little, reading.tenths)?;
}
}
ReadReadings is the other half, from a &var Reader. Neither function names a file or a memory stream; Main decides, by what it passes:
flowchart LR
w["`WriteReadings`"] --> wi{"`&var Writer`"}
wi --> bw["`ByteWriter`<br/>bytes in memory"]
wi --> fw["`File`<br/>bytes on disk"]
r["`ReadReadings`"] --> ri{"`&var Reader`"}
ri --> br["`ByteReader`<br/>over a slice"]
ri --> fr["`File`"]Writing into memory
A ByteWriter keeps everything written in a block of memory that it grows as needed, doubling each time, so a long run of small writes stays cheap. It is made from an allocator, and Bytes() hands the bytes back as a char8[..] view:
var memory = ByteWriter::New(allocator);
WriteReadings(memory, readings[..])?;
Three readings make 14 bytes — the count, then three pairs — and the program prints them in hexadecimal: 03 00 is the count 3 with its low byte first, d7 00 is 215, the 21.5 °C of sensor 1.
| Method | Returns | Does |
|---|---|---|
New | ByteWriter | an empty writer drawing on an allocator |
Write | uint ! IoError | appends, or overwrites after a seek back |
Bytes | char8[..] | everything written, valid until the next change |
Length | uint | how many bytes have been written |
Seek | uint64 ! IoError | moves where the next write lands, within what is written |
Clear | — | forgets the bytes, keeping the block for reuse |
Release | — | gives the block back; the destructor does this too |
Reading from memory
A ByteReader reads a slice of bytes you already have — here the writer's own — from the start. It borrows them, so they must outlive it, and it fails with EndOfStream once they are used up, exactly like a file:
var reader = ByteReader::New(memory.Bytes());
ReadReadings(reader)?;
Position() tells how far it has read and Remaining() returns the bytes not yet read.
The same code, a file
Main then passes a File to the very same two functions. The file holds the same 14 bytes, and reads back the same three readings — the format code never changed.
That is the point of a memory stream: the format is proved right in memory, where a test is fast and leaves nothing behind, and then trusted with a file. The last part of the program goes further. A damaged input — a stream that ends in the middle of a reading — normally means writing a file and cutting it short, as Binary did. With a ByteReader, it is just a shorter slice:
var cut = ByteReader::New(memory.Bytes()[..9]);
The first reading comes through whole. The second has its sensor but only one byte of its temperature, so ReadInt16 fails with EndOfStream, and the read is refused rather than finished with a made-up value.
Seeking
Both streams can Seek within what they hold. A writer that seeks back overwrites in place, which is how a format whose header holds a count or a length is written when that number is only known at the end:
var memory = ByteWriter::New(allocator);
WriteAll(memory, "count=0;a;b;c")?;
// Go back and fix the count once it is known.
memory.Seek(SeekFrom::Start, 6)?;
WriteAll(memory, "3")?;
PrintLine("{}", memory.Bytes());
This prints count=3;a;b;c. A seek before the start or past the end of what is held fails with IoErrorKind::InvalidInput.
The program
The whole lesson is one package in the Examples repository. Its comments explain every step.
// Code that reads and writes bytes is usually written against the `Reader` and `Writer`
// interfaces rather than against `File`, and the Binary lesson's `ReadUint16` and `WriteUint16`
// are written that way. A memory stream is a `Reader` or `Writer` whose bytes live in memory, so
// the same code can run without a file at all:
//
// ByteWriter a `Writer` that keeps everything written in a block it grows as needed;
// `Bytes()` hands the bytes back
// ByteReader a `Reader` over a slice of bytes you already have; at the end it fails
// with `EndOfStream`, like a file
//
// That is how a format is tested without touching the disk, how a message is built in memory
// before it is sent in one piece, and how bytes that arrived some other way — over a network,
// embedded in the program — are read with the code that reads files.
//
// The file this program also writes lives in its package's `Bin/` folder, and is deleted at
// the end.
import Allocator::{ Allocator, SystemAllocator };
import FileSystem::{ DeleteFile, File, OpenOptions };
import Io::{ ByteReader, ByteWriter, Endian, IoError, Print, PrintLine, ReadInt16, ReadUint16,
Reader, WriteInt16, WriteUint16, Writer };
import Path::{ OsString, Path };
import Text::TextError;
// A sensor's reading, in tenths of a degree: 215 is 21.5 °C.
struct Reading {
sensor: uint16;
tenths: int16;
}
// The format: a count, then each reading's sensor and temperature, two bytes each, little endian.
// It takes any `Writer`, so it neither knows nor cares where the bytes go.
func WriteReadings(out: &var Writer, readings: Reading[..]) -> ! IoError {
WriteUint16(out, Endian::Little, readings.length as uint16)?;
for reading in readings {
WriteUint16(out, Endian::Little, reading.sensor)?;
WriteInt16(out, Endian::Little, reading.tenths)?;
}
}
// The other half of the format, from any `Reader`. A stream that ends part-way fails it.
func ReadReadings(input: &var Reader) -> ! IoError {
let count = ReadUint16(input, Endian::Little)?;
for _ in 0..count {
let sensor = ReadUint16(input, Endian::Little)?;
let tenths = ReadInt16(input, Endian::Little)?;
PrintLine(" sensor {} {:.1} C", sensor, tenths as float64 / 10.0);
}
}
func Main() -> ! IoError | TextError {
var system = SystemAllocator();
let allocator: Allocator = system;
let readings: Reading[3] = [Reading { sensor: 1, tenths: 215 },
Reading { sensor: 2, tenths: 198 },
Reading { sensor: 7, tenths: 230 }];
// Into memory. The writer grows its block as bytes arrive.
var memory = ByteWriter::New(allocator);
WriteReadings(memory, readings[..])?;
Print("memory: {} bytes:", memory.Length());
for byte in memory.Bytes() {
Print(" {:02x}", byte as uint8);
}
PrintLine();
// Back out of memory, with the code that would read a file. Reading moves the reader's
// position along, so it must be `var`; a `let` reader is refused:
//
// error: argument 1 to 'ReadReadings' cannot borrow immutable 'reader' as
// '&var rux/io::Reader'
var reader = ByteReader::New(memory.Bytes());
ReadReadings(reader)?;
// The same two functions, given a file.
var holder = OsString::FromText(allocator, "Bin/readings.bin")?;
let path = Path::FromView(holder.View());
var writing = File::Open(allocator, path, OpenOptions::Writing())?;
WriteReadings(writing, readings[..])?;
writing.Close()?;
var reading = File::Open(allocator, path, OpenOptions::Reading())?;
PrintLine("file: {} bytes", reading.Size()?);
ReadReadings(reading)?;
reading.Close()?;
DeleteFile(allocator, path)?;
// Damaged input, made without damaging a file: only the first 9 bytes of the 14.
var cut = ByteReader::New(memory.Bytes()[..9]);
PrintLine("cut: 9 bytes");
ReadReadings(cut) catch {
error if error.IsEnd() => {
PrintLine(" ended inside a reading: refused");
},
error => fail error
};
}
Besides Io, its Rux.toml lists Allocator, FileSystem, Path and Text under [Dependencies].
Run it
cd Examples/Files/MemoryStream
rux run
memory: 14 bytes: 03 00 01 00 d7 00 02 00 c6 00 07 00 e6 00
sensor 1 21.5 C
sensor 2 19.8 C
sensor 7 23.0 C
file: 14 bytes
sensor 1 21.5 C
sensor 2 19.8 C
sensor 7 23.0 C
cut: 9 bytes
sensor 1 21.5 C
ended inside a reading: refused
Common mistakes
Try it yourself
- Write the readings into a
ByteWriter, then write the same count and readings a second time, and read both copies back with oneByteReader. - Give the format a header: write a placeholder count of 0, the readings, then
Seekback to the start and write the real count, as in the seeking example. - Read from a
ByteReaderover only the first two bytes,memory.Bytes()[..2]: a count of 3 with no readings after it. Which call fails, and with what? - Change
Endian::LittletoEndian::Bigin both functions and compare the printed bytes.
Learn more
- Binary —
ReadUint16,WriteUint16and fixed-width records - Buffered I/O — another stream that stands in front of a file
- Interface parameter — taking
&var Writerinstead of a concrete type - Io API —
ByteReader,ByteWriterand the stream interfaces - Reader, Writer and Seeker in the Io API
- Interface values in the Rux Reference
19.10 Temporary file
Create a scratch file with an unguessable name that is deleted when you are done with it.
Overview
Durations, clocks, calendar dates and time zones, repeatable and unguessable random numbers, fast hashes, UUIDs and Nano IDs: eleven lessons on the small standard packages most programs reach for.