Binary
A text file stores numbers as digits: 1200000 is seven characters, 3 is one. A binary file stores the bytes of the values themselves, so a uint16 is always two bytes, a uint32 four and a float64 eight, however large the value. Every record has the same size, and a reader knows exactly how many bytes to expect — which also means it can tell when some are missing.
A fixed-width record
The lesson's record is three fields, fourteen bytes whatever the values are:
| Bytes | Field | Type | Written with | Read with |
|---|---|---|---|---|
| 0–1 | version | uint16 | WriteUint16 | ReadUint16 |
| 2–5 | count | uint32 | WriteUint32 | ReadUint32 |
| 6–13 | average | float64 | WriteFloat64 | ReadFloat64 |
The Io package has a pair of functions like these for every fixed-width type, from uint8 up to int512 and both floats. Each takes an Endian naming the byte order — Endian explains why that must be stated. This format simply says little endian, and both sides must agree:
var writing = try File::Open(allocator, path, OpenOptions::Writing());
try WriteUint16(writing, Endian::Little, 3);
try WriteUint32(writing, Endian::Little, 1200000);
try WriteFloat64(writing, Endian::Little, 2.75);
try writing.Close();
Each literal takes its parameter's type — 3 becomes a uint16, 1200000 a uint32 — the Literal rule at work. Reading is the same three calls in the same order:
func ReadRecord(file: &var File) throws IoError {
let version = try ReadUint16(file, Endian::Little);
let count = try ReadUint32(file, Endian::Little);
let average = try ReadFloat64(file, Endian::Little);
PrintLine(" version {}, count {}, average {}", version, count, average);
}
Nothing in the file says where one field ends and the next begins. The format is the agreement between writer and reader, and the code on each side is its only record.
A value is all of its bytes, or nothing
File showed that File::Read is one attempt that may return fewer bytes than asked. A number cannot be assembled from part of its bytes, so the fixed-width readers are built on Io::ReadExact, which keeps reading until every byte of the value has arrived — and the writers on WriteAll:
flowchart LR
r["`ReadUint32(file, …)`"] --> e["`ReadExact`:<br/>4 bytes wanted"]
e --> rd["`file.Read(…)`"]
rd --> q{"what came back?"}
q -- "some bytes,<br/>more still needed" --> rd
q -- "the last<br/>bytes needed" --> v["assemble the `uint32`<br/>in the stated byte order"]
q -- "the end, before<br/>the first byte" --> s["throw `EndOfStream`"]
q -- "the end, part-way<br/>through the value" --> f["throw `UnexpectedEnd`"]If the stream ends before a value starts, ReadUint32 throws the plain end of the stream, EndOfStream — the error whose IsEnd() is true in File. If it ends part-way through a value, it throws UnexpectedEnd instead of building a number from the bytes it got plus whatever happened to be in memory. A short file is an error to report, never a value to guess.
A record cut short
The second half of the program simulates a crash part-way through writing. It opens the file with plain write permission — OpenOptions() with only write set, so the file is not emptied the way Writing would — and cuts it to ten bytes:
var options = OpenOptions();
options.write = true;
var cutting = try File::Open(allocator, path, options);
try cutting.SetLength(10);
try cutting.Close();
Ten bytes hold the whole version and count, and four of the eight bytes of average. ShowRecord reads the record again and catches the UnexpectedEnd error rather than passing it on:
try ReadRecord(reading) catch {
error if error.kind == IoErrorKind::UnexpectedEnd => {
PrintLine(" the file ends inside a field: record refused");
},
error => throw error
};
The first two fields read perfectly well, but the record as a whole is refused: one field that cannot be read whole throws out of ReadRecord, and the half-read values are never printed.
The program
The whole lesson is one package in the Examples repository. Its comments explain every step.
// A text file stores numbers as digits. A binary file stores the bytes of the values themselves:
// a `uint16` is always two bytes, a `uint32` four and a `float64` eight, however large the
// value. Every record has the same size, so a reader knows exactly how many bytes to expect.
//
// `Io` has a function pair for each fixed-width type, such as `WriteUint32` and `ReadUint32`.
// Each takes an `Endian` naming the byte order; the Endian lesson explains why that must be
// stated. Here the format simply says little endian, on both sides.
//
// The real lesson is about bytes that do not arrive. `File::Read` is one attempt and may return
// fewer bytes than asked; at the end of a file it throws `EndOfStream`. The fixed-width
// functions are built on `ReadExact` and `WriteAll`, which keep calling until every byte of the
// value has moved. If the stream ends before a value starts, they throw `EndOfStream`; if it ends
// part-way through one, they throw `UnexpectedEnd` instead of assembling a number from the bytes
// they got plus whatever was in memory. A short file is an error to report, never a value to
// guess.
//
// The file lives in this package's `Bin/` folder, and the program deletes it at the end.
import Allocator::{ Allocator, SystemAllocator };
import FileSystem::{ DeleteFile, File, OpenOptions };
import Io::{ Endian, IoError, IoErrorKind, PrintLine, ReadFloat64, ReadUint16, ReadUint32, WriteFloat64,
WriteUint16, WriteUint32 };
import Path::{ OsString, Path };
// Reads one record. Any field that cannot be read whole fails the whole record.
func ReadRecord(file: &var File) throws IoError {
let version = try ReadUint16(file, Endian::Little);
let count = try ReadUint32(file, Endian::Little);
let average = try ReadFloat64(file, Endian::Little);
PrintLine(" version {}, count {}, average {}", version, count, average);
}
// Opens the file, reads a record from it, and reports a short file instead of passing it on.
func ShowRecord(allocator: Allocator, path: Path) throws IoError {
var reading = try File::Open(allocator, path, OpenOptions::Reading());
try PrintLine("{} holds {} bytes", path, reading.Size());
try ReadRecord(reading) catch {
error if error.kind == IoErrorKind::UnexpectedEnd => {
PrintLine(" the file ends inside a field: record refused");
},
error => throw error
};
try reading.Close();
}
func Main() throws IoError {
let allocator = SystemAllocator();
var holder = OsString::FromText(allocator, "Bin/record.bin");
let path = Path::FromView(holder.View());
// 2 + 4 + 8 bytes: fourteen, whatever the values are.
var writing = try File::Open(allocator, path, OpenOptions::Writing());
try WriteUint16(writing, Endian::Little, 3);
try WriteUint32(writing, Endian::Little, 1200000);
try WriteFloat64(writing, Endian::Little, 2.75);
try writing.Close();
try ShowRecord(allocator, path);
// Cut the file in the middle of the last field, as a crash part-way through writing might
// have left it. Plain `write` permission, without `Writing`'s emptying of the file.
var options = OpenOptions();
options.write = true;
var cutting = try File::Open(allocator, path, options);
try cutting.SetLength(10);
try cutting.Close();
try ShowRecord(allocator, path);
try DeleteFile(allocator, path);
}
Besides Io, its Rux.toml lists Allocator, FileSystem and Path under [Dependencies].
Run it
cd Examples/Files/Binary
rux run
Bin/record.bin holds 14 bytes
version 3, count 1200000, average 2.75
Bin/record.bin holds 10 bytes
the file ends inside a field: record refused
Common mistakes
Try it yourself
- Add a fourth field, an
int8temperature, to both the writer andReadRecord. How many bytes is the record now? - Cut the file to 6 bytes instead of 10. Which field is the first one that cannot be read?
- Write two records one after the other, and read them back in a loop until the end of the file.
- Write the
uint321200000 withEndian::Bigand read it withEndian::Big. Is the file any different in size?
Learn more
- Endian — byte order, and why a format must state it
- File —
Readas one attempt, and the end of a file as a thrown error - Integer and Float — the widths behind each field's size
- Atomic file — how to avoid leaving a record cut short in the first place