Binary
A text file stores numbers as digits: 1200000 is seven characters, 3 is one. A binary file stores the bytes of the values themselves, so a uint16 is always two bytes, a uint32 four and a float64 eight, however large the value. Every record has the same size, and a reader knows exactly how many bytes to expect — which also means it can tell when some are missing.
A fixed-width record
The lesson's record is three fields, fourteen bytes whatever the values are:
| Bytes | Field | Type | Written with | Read with |
|---|---|---|---|---|
| 0–1 | version | uint16 | WriteUint16 | ReadUint16 |
| 2–5 | count | uint32 | WriteUint32 | ReadUint32 |
| 6–13 | average | float64 | WriteFloat64 | ReadFloat64 |
The Io package has a pair of functions like these for every fixed-width type, from uint8 up to int512 and both floats. Each takes an Endian naming the byte order — Endian explains why that must be stated. This format simply says little endian, and both sides must agree:
var writing = File::Open(allocator, path, OpenOptions::Writing())?;
WriteUint16(writing, Endian::Little, 3)?;
WriteUint32(writing, Endian::Little, 1200000)?;
WriteFloat64(writing, Endian::Little, 2.75)?;
writing.Close()?;
Each literal takes its parameter's type — 3 becomes a uint16, 1200000 a uint32 — the Literal rule at work. Reading is the same three calls in the same order:
func ReadRecord(file: &var File) -> ! IoError {
let version = ReadUint16(file, Endian::Little)?;
let count = ReadUint32(file, Endian::Little)?;
let average = ReadFloat64(file, Endian::Little)?;
PrintLine(" version {}, count {}, average {}", version, count, average);
}
Nothing in the file says where one field ends and the next begins. The format is the agreement between writer and reader, and the code on each side is its only record.
A value is all of its bytes, or nothing
File showed that File::Read is one attempt that may return fewer bytes than asked. A number cannot be assembled from part of its bytes, so the fixed-width readers are built on Io::ReadExact, which keeps reading until every byte of the value has arrived — and the writers on WriteAll:
flowchart LR
r["ReadUint32(file, …)"] --> e["ReadExact:<br/>4 bytes wanted"]
e --> rd["file.Read(…)"]
rd --> q{"what came back?"}
q -- "some bytes,<br/>more still needed" --> rd
q -- "the last<br/>bytes needed" --> v["assemble the uint32<br/>in the stated byte order"]
q -- "the end of the file" --> f["fail with an IoError<br/>whose IsEnd() is true"]If the stream ends part-way through a value, ReadUint32 fails instead of building a number from the bytes it got plus whatever happened to be in memory. A short file is an error to report, never a value to guess.
A record cut short
The second half of the program simulates a crash part-way through writing. It opens the file with plain write permission — OpenOptions() with only write set, so the file is not emptied the way Writing would — and cuts it to ten bytes:
var options = OpenOptions();
options.write = true;
var cutting = File::Open(allocator, path, options)?;
cutting.SetLength(10)?;
cutting.Close()?;
Ten bytes hold the whole version and count, and four of the eight bytes of average. ShowRecord reads the record again and catches the end-of-file failure rather than passing it on:
ReadRecord(reading) catch {
error if error.IsEnd() => {
PrintLine(" the file ends inside a field: record refused");
},
error => fail error
};
The first two fields read perfectly well, but the record as a whole is refused: one field that cannot be read whole fails ReadRecord, and the half-read values are never printed.
The program
The whole lesson is one package in the Examples repository. Its comments explain every step.
// A text file stores numbers as digits. A binary file stores the bytes of the values themselves:
// a `uint16` is always two bytes, a `uint32` four and a `float64` eight, however large the
// value. Every record has the same size, so a reader knows exactly how many bytes to expect.
//
// `Io` has a function pair for each fixed-width type, such as `WriteUint32` and `ReadUint32`.
// Each takes an `Endian` naming the byte order; the Endian lesson explains why that must be
// stated. Here the format simply says little endian, on both sides.
//
// The real lesson is about bytes that do not arrive. `File::Read` is one attempt and may return
// fewer bytes than asked; at the end of a file it fails with `EndOfStream`. The fixed-width
// functions are built on `ReadExact` and `WriteAll`, which keep calling until every byte of the
// value has moved. If the stream ends part-way through a value, `ReadUint32` fails instead of
// assembling a number from the bytes it got plus whatever was in memory. A short file is an
// error to report, never a value to guess.
//
// The file lives in this package's `Bin/` folder, and the program deletes it at the end.
import Allocator::{ Allocator, SystemAllocator };
import FileSystem::{ DeleteFile, File, OpenOptions };
import Io::{ Endian, IoError, PrintLine, ReadFloat64, ReadUint16, ReadUint32, WriteFloat64,
WriteUint16, WriteUint32 };
import Path::{ OsString, Path };
import Text::TextError;
// Reads one record. Any field that cannot be read whole fails the whole record.
func ReadRecord(file: &var File) -> ! IoError {
let version = ReadUint16(file, Endian::Little)?;
let count = ReadUint32(file, Endian::Little)?;
let average = ReadFloat64(file, Endian::Little)?;
PrintLine(" version {}, count {}, average {}", version, count, average);
}
// Opens the file, reads a record from it, and reports a short file instead of passing it on.
func ShowRecord(allocator: Allocator, path: Path) -> ! IoError {
var reading = File::Open(allocator, path, OpenOptions::Reading())?;
PrintLine("{} holds {} bytes", path, reading.Size()?);
ReadRecord(reading) catch {
error if error.IsEnd() => {
PrintLine(" the file ends inside a field: record refused");
},
error => fail error
};
reading.Close()?;
}
func Main() -> ! IoError | TextError {
var system = SystemAllocator();
let allocator: Allocator = system;
var holder = OsString::FromText(allocator, "Bin/record.bin")?;
let path = Path::FromView(holder.View());
// 2 + 4 + 8 bytes: fourteen, whatever the values are.
var writing = File::Open(allocator, path, OpenOptions::Writing())?;
WriteUint16(writing, Endian::Little, 3)?;
WriteUint32(writing, Endian::Little, 1200000)?;
WriteFloat64(writing, Endian::Little, 2.75)?;
writing.Close()?;
ShowRecord(allocator, path)?;
// Cut the file in the middle of the last field, as a crash part-way through writing might
// have left it. Plain `write` permission, without `Writing`'s emptying of the file.
var options = OpenOptions();
options.write = true;
var cutting = File::Open(allocator, path, options)?;
cutting.SetLength(10)?;
cutting.Close()?;
ShowRecord(allocator, path)?;
DeleteFile(allocator, path)?;
}
Besides Io, its Rux.toml lists Allocator, FileSystem, Path and Text under [Dependencies].
Run it
cd Examples/Files/Binary
rux run
Bin/record.bin holds 14 bytes
version 3, count 1200000, average 2.75
Bin/record.bin holds 10 bytes
the file ends inside a field: record refused
Common mistakes
WriteUint16(writing, Endian::Little, 70000) fails with error: argument 3 to 'WriteUint16' has type 'int', but parameter 'value' requires 'uint16'. 70000 needs more than sixteen bits, so the literal cannot become a uint16.int variable.With
let count = 1200000;, WriteUint32(writing, Endian::Little, count) fails with has type 'int', but parameter 'value' requires 'uint32'. A variable already has its type. Declare it let count: uint32 = 1200000;, or convert it with as uint32.Swap the first two reads and the program still runs, printing
version 18, count 1333788675: the bytes are read, just as the wrong fields. Nothing in a binary file can catch this; only the agreement between writer and reader can.Read
count with Endian::Big and it comes out as 2152665600 instead of 1200000 — the same four bytes, assembled backwards. The byte order is part of the format, and both sides must state the same one.Try it yourself
- Add a fourth field, an
int8temperature, to both the writer andReadRecord. How many bytes is the record now? - Cut the file to 6 bytes instead of 10. Which field is the first one that cannot be read?
- Write two records one after the other, and read them back in a loop until the end of the file.
- Write the
uint321200000 withEndian::Bigand read it withEndian::Big. Is the file any different in size?
Learn more
- Endian — byte order, and why a format must state it
- File —
Readas one attempt, and the end of a file as a failure - Integer and Float — the widths behind each field's size
- Atomic file — how to avoid leaving a record cut short in the first place