Base64 and friends
JSON, TOML, URLs, e-mail and HTTP headers all carry text. A key, a hash, a signature or a small image is bytes, any of the 256 values, most of which are not printable characters at all. To send bytes through a text format, you write them with a small alphabet of safe characters, and read them back exactly at the other end. That is a base encoding, and the Encoding package has the four you will meet:
import Encoding::{ Base32Encode, Base32Variant, Base58Encode, Base58EncodedMaxLength,
Base64Decode, Base64Encode, Base64EncodedLength, Base64Variant, HexDecode, HexEncode,
ViewHex };
Encoding in Part 14 is about character encodings — how text is stored as UTF-8 or UTF-16 bytes. This lesson goes the other way: any bytes at all become plain ASCII text. The package happens to be called
Encoding too.Four alphabets
Each scheme splits the bytes into groups of bits and writes each group as one character:
| Scheme | Alphabet | Bits per character | 5 bytes become | Used for |
|---|---|---|---|---|
| Hexadecimal | 0–9, a–f | 4 | 10 characters | hashes, colours, debugging |
| Base32 | A–Z, 2–7 | 5 | 8 characters | one-time-password secrets, file names |
| Base64 | A–Z, a–z, 0–9, two more | 6 | 8 characters | e-mail attachments, data URLs, JWTs |
| Base58 | digits and letters, minus 0OIl | about 5.86 | 7 characters | identifiers people copy by hand |
The more bits a character carries, the shorter the text — and the more kinds of character it needs, some of which a particular format may not like. The program writes the same five bytes, fb ef ff 00 10, in each:
hex fbefff0010
base64 ++//ABA=
base32 7PX76AAQ
base58 VRbE2yM (at most 7)
Base64's variants
Base64 needs 64 characters, and the last two of the standard alphabet, + and /, mean something in a URL or a file name. So there is a URL-safe alphabet that uses - and _ instead. Either may also drop the = padding, which rounds the text up to a multiple of four characters:
Base64Variant | Last two | Padding | The five bytes |
|---|---|---|---|
Standard | + / | yes | ++//ABA= |
StandardUnpadded | + / | no | ++//ABA |
Url | - _ | yes | --__ABA= |
UrlUnpadded | - _ | no | --__ABA |
Standard is what MIME and most protocols mean by "Base64"; UrlUnpadded is what JSON Web Tokens and most URL parameters use. A decoder must be told the same variant the text was written in. Base32 has variants as well: the RFC 4648 Standard, and Crockford, which leaves out I, L, O and U because they are easy to misread.
Buffers you provide
Nothing in the package allocates. Every function writes into a buffer you pass and returns how much of it it wrote, as a uint ! FormatError:
func ShowBase64(label: char8[..], bytes: byte[..], form: Base64Variant) {
var text: char8[64];
let written = Base64Encode(bytes, text[..], form) catch {
else => {
PrintLine("{:<18} does not fit", label);
return;
}
};
PrintLine("{:<18} {}", label, text[..written]);
}
The encoded text is text[..written], the front of the buffer. To size a buffer exactly, each scheme has a length function: Base64EncodedLength(27) is 36. Base58's length depends on the value itself, so it has only a bound, Base58EncodedMaxLength. A buffer that is too short is refused, never filled with what fits, because a cut-off encoding would be a different value rather than a shorter one.
When you only want to print bytes, there is a shortcut that needs no buffer at all: ViewHex(bytes) borrows them and renders them as hexadecimal under {}.
Strict decoding
Decoding goes back from text to bytes, into a var byte[..], and returns uint ! DecodeError. It is strict, so that each value has exactly one encoded form and two programs can never read the same text differently. A DecodeError prints what is wrong and where:
func ReadHex(text: char8[..]) {
var bytes: byte[8];
match HexDecode(text, bytes[..]) {
.Success(count) => PrintLine("hex {:<12} -> {} bytes: {}", text, count,
ViewHex(bytes[..count])),
.Failure(error) => PrintLine("hex {:<12} -> {}", text, error)
}
}
| Text | Decoded as | Result |
|---|---|---|
fbefff0010 | hexadecimal | 5 bytes |
FBEFFF0010 | hexadecimal | 5 bytes — either case is accepted |
fbefff001 | hexadecimal | invalid length at 9 — half a byte |
0xfbefff | hexadecimal | invalid character at 1 — no prefixes |
SGVsbG8= | Base64 | 5 bytes, 48656c6c6f: the text Hello |
SGVsbG9= | Base64 | non-canonical trailing bits at 6 |
SGVsbG8 | Base64, Standard | invalid length at 7 — the padding is needed |
SGVsbG8= | into 3 bytes | destination too short, need 5 bytes |
The non-canonical case is the subtle one. o and 9 differ only in bits that the last character carries but the data does not use, so SGVsbG8= and SGVsbG9= would decode to the same five bytes. Refusing the second keeps decoding one-to-one, which is what makes it safe to compare encoded values as text.
flowchart LR
b(["bytes"]) -- "`HexEncode`, `Base64Encode`, …" --> t(["text"])
t -- "`HexDecode`, `Base64Decode`, …" --> b
t -- "wrong alphabet, length,<br/>padding or bits" --> e["`DecodeError`<br/>with the byte offset"]The program
The whole lesson is one package in the Examples repository. Its comments explain every step.
// Text formats such as JSON, TOML, URLs and e-mail carry text, not raw bytes. A key, a hash or
// an image has to be written as characters before it can travel through them, and read back
// exactly afterwards. A base encoding does that with a small alphabet of safe characters:
//
// hexadecimal 16 characters, 4 bits each: 2 characters per byte
// Base32 32 characters, 5 bits each: 8 characters per 5 bytes
// Base64 64 characters, 6 bits each: 4 characters per 3 bytes
// Base58 58 characters, for identifiers people copy by hand
//
// This is not the character encoding of the Encoding lesson, which stores text as UTF-8 or
// UTF-16. It goes the other way: any bytes at all become plain ASCII text.
//
// Nothing in the Encoding package allocates. Every function writes into a buffer you provide and
// returns how much it wrote, and a length function says how big that buffer must be. Decoding is
// strict, and a `DecodeError` says what is wrong and at which byte.
import Encoding::{ Base32Encode, Base32Variant, Base58Encode, Base58EncodedMaxLength,
Base64Decode, Base64Encode, Base64EncodedLength, Base64Variant, HexDecode, HexEncode,
ViewHex };
import Io::PrintLine;
// Prints `bytes` in one Base64 variant.
func ShowBase64(label: char8[..], bytes: byte[..], form: Base64Variant) {
var text: char8[64];
let written = Base64Encode(bytes, text[..], form) catch {
else => {
PrintLine("{:<18} does not fit", label);
return;
}
};
PrintLine("{:<18} {}", label, text[..written]);
}
// Decodes hexadecimal text, or says why it cannot. Encoding turns bytes into characters and
// decoding turns them back, so the destination here is a `var byte[..]`. A character buffer, or
// an array declared with `let`, is the wrong kind of slice:
//
// HexDecode(text, characters[..])
// error: argument 2 to 'HexDecode' has type 'var char8[..]', but parameter 'into'
// requires 'var uint8[..]'
func ReadHex(text: char8[..]) {
var bytes: byte[8];
match HexDecode(text, bytes[..]) {
.Success(count) => PrintLine("hex {:<12} -> {} bytes: {}", text, count,
ViewHex(bytes[..count])),
.Failure(error) => PrintLine("hex {:<12} -> {}", text, error)
}
}
// Decodes standard Base64 text into a buffer of `room` bytes, or says why it cannot.
func ReadBase64(text: char8[..], room: uint) {
var bytes: byte[8];
match Base64Decode(text, bytes[..room]) {
.Success(count) => PrintLine("b64 {:<12} -> {} bytes: {}", text, count,
ViewHex(bytes[..count])),
.Failure(error) => PrintLine("b64 {:<12} -> {}", text, error)
}
}
func Main() -> int {
// Five bytes that are not text: no text format could hold them as they are.
let key: byte[5] = [0xFB, 0xEF, 0xFF, 0x00, 0x10];
var text: char8[64];
let hex = HexEncode(key[..], text[..]) catch { else => 0 };
PrintLine("{:<18} {}", "hex", text[..hex]);
// Base64 comes in four variants. The standard alphabet ends in `+` and `/`, which mean
// something in a URL, so the URL-safe one uses `-` and `_` instead. Either may drop the `=`
// padding that rounds the text up to a multiple of four characters.
ShowBase64("base64", key[..], Base64Variant::Standard);
ShowBase64("base64, no pad", key[..], Base64Variant::StandardUnpadded);
ShowBase64("base64 url", key[..], Base64Variant::Url);
ShowBase64("base64 url, no pad", key[..], Base64Variant::UrlUnpadded);
// Base32 uses only capitals and digits, so it survives case-insensitive systems. Crockford's
// alphabet leaves out I, L, O and U, which are easy to misread.
let base32 = Base32Encode(key[..], text[..], Base32Variant::Standard) catch { else => 0 };
PrintLine("{:<18} {}", "base32", text[..base32]);
let crockford = Base32Encode(key[..], text[..], Base32Variant::Crockford) catch { else => 0 };
PrintLine("{:<18} {}", "base32 crockford", text[..crockford]);
// Base58 drops 0, O, I, l, + and /, so a person can copy it without mistakes. Its length
// depends on the value, so there is only an upper bound to size the buffer with.
let base58 = Base58Encode(key[..], text[..]) catch { else => 0 };
PrintLine("{:<18} {} (at most {})", "base58", text[..base58], Base58EncodedMaxLength(5));
// The length functions size a buffer exactly. A buffer that is too short is refused, never
// filled with what fits: a cut-off encoding would be a different value. So every encoder
// returns `uint ! FormatError`, the count or the refusal, and never just the count:
//
// let written: uint = Base64Encode(key[..], text[..]);
// error: cannot assign 'uint ! FormatError' to 'uint'
PrintLine("27 bytes in Base64 take {} characters", Base64EncodedLength(27));
var short: char8[4];
match Base64Encode("Many hands make light work.", short[..]) {
.Success(written) => PrintLine("wrote {}", written),
.Failure(_) => PrintLine("a 4-character buffer is refused")
}
let written = Base64Encode("Many hands make light work.", text[..]) catch { else => 0 };
PrintLine("{}", text[..written]);
// Decoding is strict, so every value has exactly one encoded form.
ReadHex("fbefff0010");
ReadHex("FBEFFF0010");
ReadHex("fbefff001");
ReadHex("0xfbefff");
ReadBase64("SGVsbG8=", 8);
ReadBase64("SGVsbG9=", 8);
ReadBase64("SGVsbG8", 8);
ReadBase64("SGVsbG8=", 3);
return 0;
}
Besides Io, its Rux.toml lists Encoding under [Dependencies].
Run it
cd Examples/DataFormats/BaseEncoding
rux run
hex fbefff0010
base64 ++//ABA=
base64, no pad ++//ABA
base64 url --__ABA=
base64 url, no pad --__ABA
base32 7PX76AAQ
base32 crockford ZFQZY00G
base58 VRbE2yM (at most 7)
27 bytes in Base64 take 36 characters
a 4-character buffer is refused
TWFueSBoYW5kcyBtYWtlIGxpZ2h0IHdvcmsu
hex fbefff0010 -> 5 bytes: fbefff0010
hex FBEFFF0010 -> 5 bytes: fbefff0010
hex fbefff001 -> invalid length at 9
hex 0xfbefff -> invalid character at 1
b64 SGVsbG8= -> 5 bytes: 48656c6c6f
b64 SGVsbG9= -> non-canonical trailing bits at 6
b64 SGVsbG8 -> invalid length at 7
b64 SGVsbG8= -> destination too short, need 5 bytes
Common mistakes
Try it yourself
- Encode
"Many hands make light work."withBase64Variant::StandardUnpadded. Does it differ from the padded form? Now drop the full stop and compare again. - Decode
"zfqz-y00g"withBase32Variant::Crockford. Crockford's decoder accepts lower case and skips hyphens; what bytes do you get? - Print the key with
PrintLine("{:X}", ViewHex(key[..])), without any buffer. - Encode the bytes
[0, 0, 1]in Base58. What becomes of the two leading zero bytes?
Learn more
- Encoding API — every scheme, the percent-encoding sets, and the streaming
EncodingWriterandDecodingReader - Encoding — character encodings: how text becomes bytes, the opposite direction
- Hash — hashes and checksums, usually shown in hexadecimal
- URL — percent-encoding, the base encoding of URLs
- Writable slice — a function filling a buffer it was lent
- Slices in the Rux Reference
21.5 Writing TOML
Read a TOML document, add a key, write it back with TomlWriteDocument, and check that writing is stable across a round trip.
21.7 URL
Read a URL with Url::Parse, which normalizes it, take it apart into its components, resolve relative links against it with Resolve, walk and decode its query with Pairs and Decode, and build a query safely with QueryBuilder.