Random access and streaming

8.7. Random access and streaming

Three abstract types describe a value that lives in a file rather than in memory:

Type Role

Type	Role
`IFile a`	Random-access reader. The file is open for indexed and pattern access; elements are decoded on demand. `a` is the type of the data in the file.
`OStream a`	Sequential writer. Elements are buffered, compressed, and flushed to disk as sub-packets. The output type is always of type `[a]`.
`IStream a`	Sequential reader. The file is walked forward one sub-packet at a time; element order is preserved. Input is always of type `[a]`.

IFile a

Random-access reader. The file is open for indexed and pattern access; elements are decoded on demand. a is the type of the data in the file.

OStream a

Sequential writer. Elements are buffered, compressed, and flushed to disk as sub-packets. The output type is always of type [a].

IStream a

Sequential reader. The file is walked forward one sub-packet at a time; element order is preserved. Input is always of type [a].

All three are opened with the same @open intrinsic, which returns an integer handle that lives in a shared SHM registry, so the handle can be passed transparently across pool boundaries.

8.7.1. Random access with IFile

@open path opens an existing morloc stream file for random access. The file’s element schema is read out of the header and must match the IFile parameter type; a mismatch errors at open time.

Once open, an IFile a is indexable and sliceable using the bracket patterns documented in Patterns:

module main (lookup, slice)

lookup :: Str -> Int -> <IO> Person
lookup path i = do
  Ok f <- @open path :: <IO> (Try Str (IFile [Person]))
  let p = .[i] f
  @close f
  p

slice :: Str -> <IO> [Person]
slice path = do
  Ok f <- @open path :: <IO> (Try Str (IFile [Person]))
  let xs = .[100:200] f
  @close f
  xs

A slice that spans multiple sub-packets decompresses each sub-packet once, in parallel, and copies just the selected elements into the result. The decompressed sub-packets are cached per handle so a second access to a nearby element is a pointer add rather than another zstd pass. @flen f returns the total element count without scanning the file.

8.7.2. Sequential writing with OStream

The line

@open path :: <IO> (Try Str (OStream a))

creates a new stream file. The OStream serializes elements into a write buffer; once the buffer reaches its cap, the contents flush to disk as one zstd-compressed sub-packet. Element atomicity is preserved: an element larger than the buffer flushes as its own oversize sub-packet rather than splitting across boundaries.

module main (writeMany)

source Py from "compute.py" ("produceBatch")
produceBatch :: Int -> [Person]

writeMany :: Str -> Int -> <IO> ()
writeMany path n = do
  Ok o <- @open path :: <IO> (Try Str (OStream Person))
  @write 3 o (produceBatch n)
  @write 3 o (produceBatch n)
  @flush o
  @write 3 o (produceBatch n)
  @close o

@write level o xs appends xs to o using level as the zstd preset for the resulting sub-packet (0 disables compression; the preset table is the same as for Compression). @flush o forces the buffer to disk as a sub-packet boundary; without it, a partially-filled buffer is held until the next write that overflows it, or until @close. @close writes the final footer (full sub-packet index, element count, end-of-file marker) and releases the slot; the file is now readable by IFile or IStream.

@append path opens a stream file for further writes, creating it if it is not there yet, so an append-only log needs nothing else to start one. The element schema is checked against the file’s recorded schema and a mismatch errors before any byte is written.

@concat paths dest byte-level concatenates a list of compatible stream files into dest using sendfile, with no userspace copy. It builds beside dest and renames onto it, so dest may appear in paths — adding a batch to a log is the ordinary use — and a merge that fails leaves the old dest as it was. Refusing an aliased destination was the alternative; it would have forbidden that use and still left a failed merge free to delete a file it never created.

8.7.3. Sequential reading with IStream

@open path opens a stream file for forward reads. @next s returns the next sub-packet’s elements as a list; when the file is exhausted, @next returns []. The cursor advances under the slot’s futex, so two pools holding the same IStream handle can take turns calling @next and each gets a distinct sub-packet.

module main (drain)

drain :: Str -> <IO> U64
drain path = do
  Ok s <- @open path :: <IO> (Try Str (IStream Int))
  Ok a <- @next s
  Ok b <- @next s
  Ok c <- @next s
  @close s
  size a + size b + size c

@stream f derives a fresh IStream from an open IFile a. The derived stream has its own slot, fd, and cursor, so walking it does not perturb the IFile’s random-access state. The underlying file is the same — closing the IFile invalidates the derived IStream’s next read, which returns a generation-mismatch error.

@open also reads standard input, not just files. Opening the path /dev/stdin — the value a --' @stdin CLI argument takes when it is omitted, and the target of a - argument — routes an IStream to the process’s stdin, exactly like @stdin. Because a pipe is not seekable, @open "/dev/stdin" :: IFile a instead returns an Err arm without reading any bytes, so a reader can attempt fast random access first and fall back to a sequential IStream for stdin (see Reading a stream from standard input for the CLI side). Compressed sub-packets are decompressed on demand at @next; a non-morloc input on stdin is rejected rather than mis-decoded.

8.7.4. Typed standard streams

@stdin, @stdout, and @stderr are nullary intrinsics that expose the process’s standard streams as typed morloc streams. Their element types are fixed by inline ascription at the open site, exactly like @open:

Intrinsic Type

Intrinsic	Type
`@stdin`	`<IO> (Try Str (IStream a))`
`@stdout`	`<IO> (OStream a)`
`@stderr`	`<IO> (OStream a)`

@stdin

<IO> (Try Str (IStream a))

@stdout

<IO> (OStream a)

@stderr

<IO> (OStream a)

Once opened, they support the same @next / @write / @flush / @close API as their file-backed cousins. The point is type-safe stream IO between morloc programs: the compiler checks the connection at the source level, and the runtime validates each sub-packet’s wire schema on arrival. Piped morloc programs can carry structured data without falling back to ad-hoc text formats.

The canonical Unix-filter shape:

module main (producer, doubler)

import root-py

producer :: <IO> ()
producer = do
  o <- @stdout :: <IO> (OStream Int)
  @write 0 o [1, 2, 3]
  @write 0 o [4, 5, 6]
  @close o

doubler :: <IO> ()
doubler = do
  Ok s <- @stdin  :: <IO> (Try Str (IStream Int))
  o     <- @stdout :: <IO> (OStream Int)
  Ok xs <- @next s
  @write 0 o (map (\x -> x * 2) xs)
  Ok ys <- @next s
  @write 0 o (map (\x -> x * 2) ys)
  @close o
  @close s

Connecting them is a shell pipeline, and every stage but the last is asked for morloc’s own framing:

$ ./main -f packet producer | ./main doubler
[2,4,6,8,10,12]

The compiler enforces that both sides agree on element type Int. -f packet is required rather than inferred: a nexus cannot see where its standard output goes, and the choice between morloc framing and a readable rendering belongs to whoever built the pipeline. The same flag appears in Reading a stream from standard input.

@stdin / @stdout / @stderr carry morloc’s binary sub-packet format — the same wire format used on disk for IFile / IStream / OStream. It is not human-readable text. If a sourced foreign function writes to the same standard stream while morloc holds it open (print in Python, std::cout << in C++, cat / message in R), those raw bytes interleave with the morloc packet stream and the reader’s next @next fails with a schema-decode error. Either use file-based IFile / OStream for structured output, or ensure no foreign code writes to a standard stream that morloc has opened.

8.7.5. Environment variables

Variable Effect

Variable	Effect
`MORLOC_REGISTRY_SLOT_COUNT`	Number of concurrent stream handles per nexus invocation. Default `4096`. Each slot occupies 512 bytes of SHM; the table is allocated once at nexus startup.
`MORLOC_WRITE_BUFFER_BYTES`	Per-OStream write buffer cap. Default 16 MiB. Smaller values produce more sub-packets (finer reader granularity, more per-flush overhead); larger values amortise zstd overhead over more elements (coarser granularity, longer end-of-run flush). An element larger than the buffer is always written as its own oversize sub-packet regardless of this setting.
`MORLOC_IFILE_CACHE_BYTES`	Per-handle SHM cache for decompressed IFile sub-packets. Default 256 MiB. The cache uses an approximate clock-hand LRU and is released on `@close`. Set to `0` to disable, in which case every IFile access decompresses anew.

MORLOC_REGISTRY_SLOT_COUNT

Number of concurrent stream handles per nexus invocation. Default 4096. Each slot occupies 512 bytes of SHM; the table is allocated once at nexus startup.

MORLOC_WRITE_BUFFER_BYTES

Per-OStream write buffer cap. Default 16 MiB. Smaller values produce more sub-packets (finer reader granularity, more per-flush overhead); larger values amortise zstd overhead over more elements (coarser granularity, longer end-of-run flush). An element larger than the buffer is always written as its own oversize sub-packet regardless of this setting.

MORLOC_IFILE_CACHE_BYTES

Per-handle SHM cache for decompressed IFile sub-packets. Default 256 MiB. The cache uses an approximate clock-hand LRU and is released on @close. Set to 0 to disable, in which case every IFile access decompresses anew.

The MORLOC_FRAME_WORKERS knob from Compression also applies: an IFile cache miss on a sub-packet with multiple zstd frames decompresses the frames in parallel using the same worker pool.