Stream types. Three Morloc types name a stream, each a 16-byte slot in voidstar (schema codes F, I, O):

The slot’s first byte says what follows in its last eight: 0 for a relative pointer to a path (length, then bytes), 1 for the index of an open stream in the run’s stream registry (below). A path can cross between runs; an index means something only inside the run that opened it.

File format. A stream file is a sequence of packets (Packet protocol):

A reader finds the footer from the end of the file through the tail, so random access needs no scan. The status is closed for a stream written to the end, or paused for one a writer stopped and may resume. While a stream is open for writing, a temporary footer carrying only diagnostics is rewritten after every batch, so the file always ends in a valid tail.

Compression. Each batch is compressed separately with Zstandard, so a reader can decompress any batch without the others. A batch is split into independent frames of at most 16 MiB, listed in the batch’s metadata, so they can be decompressed in parallel straight into shared memory. The user’s compression level 0-9 maps to Zstandard levels 0, 1, 3, 6, 9, 12, 15, 19, 21 and 22, with long-distance matching from level 7.

Write-behind. A writer fills a buffer in shared memory (16 MiB by default). When compression is on and the buffer is full, it is sealed and handed to background threads for compression, and the writer continues into a fresh buffer. Compressed batches are appended to the file in the order they were sealed, so compression never reorders elements. The writer waits only when more than a fixed number of sealed batches are outstanding (8 by default), and every flush, close and end of call writes everything sealed so far.

Sharing an open stream. A stream’s identity lives in a registry in shared memory, created with the run’s first volume. Each slot records the file’s path, schema, write cursor, element count and batch index, and a generation number that changes when the slot is opened and when it is released. A stream index carries the generation it was made with, and every operation checks it, so an index to a closed stream can never touch the stream that later reuses its slot. Each process keeps its own descriptor and read-only mapping of the file. A writer holds an exclusive file lock while the stream is open.

Readers. An IFile maps the file and locates an element through the batch index. A batch stored uncompressed and 8-byte aligned is read where it lies in the mapping; any other batch is decompressed or copied into shared memory first. An IStream reads forward from the shared cursor. A reader that opens a stream still being written sees the batches written before it opened, up to the temporary footer; reading past the end yields an empty batch.

Rewriting a file. Opening an OStream on a file that already has data writes a new file and renames it into place, so a reader that still has the old file mapped keeps a consistent view. Where a rename is impossible (the directory is not writable, say) and nothing in the program is reading the file, it is truncated in place instead. The runtime never deletes a stream file; only a half-written temporary file is removed on failure.