9.2. morloc-nexus view

Reads a data file (morloc data-packet, morloc stream-packet, .json, .mpk, .arrow, .parquet, or .csv) and re-emits it in a chosen output format. The intended use is ad-hoc inspection (view foo.packet | jq | less), one-shot format conversion (view foo.json -f mpk --schema "as" -o foo.mpk), and slicing large files down to a subset without materialising the whole thing (view foo.packet --pattern ".[100:200]" -f json).

view reuses the same loader chain run uses for argument ingress and the same output emitter run uses for results, so format support and conversion behaviour cannot drift away from a real morloc program run. A file accepted by view is one run would also accept, and vice versa.

$ ./myprog -f packet -o result.packet mycmd
$ morloc-nexus view result.packet -f json
[1, 2, 3]
$ morloc-nexus view result.packet -f json | jq '. | length'
3
$ morloc-nexus view data.json -f mpk --schema "as" -o data.mpk

Before loading, view runs the same classifier morloc-nexus file uses, so a truncated or oversize morloc packet is rejected up-front with a clear error instead of decoding garbage from a partial payload.

9.2.1. Reading from stdin

A single - argument reads from stdin. For stream-packet input (the IFile / OStream / IStream shape), view - iterates sub-packet by sub-packet and emits one element (or one line, for -f jsonl) at a time, so a multi-gigabyte stream can be viewed in constant memory. For data-packet input, or when the selected output form requires a full-value load (e.g. -f arrow), stdin is drained to a $TMPDIR temp file and the normal file path runs. The temp file is deleted on exit.

$ ./producer | morloc-nexus view - -f jsonl | head -5

9.2.2. Schema resolution

view always loads through the typed loader, so a schema is required:

  1. --schema STRING if given.

  2. Otherwise, the schema embedded in a morloc-packet’s metadata (data or stream).

  3. Otherwise, view exits with an error directing the user to --schema.

9.2.3. --pattern: extract a subset

--pattern STR applies a morloc pattern chain to the input before emission. The grammar is the same one morloc source uses for bracket accessors: field access (.foo), tuple/positional index (.0, .1), bracket index and slice (.[i], .[a:b], .[a:b:c]), grouped projection (.(.a;.b)), and broadcast tails after a slice (.[:].name, .[:].[0], .[:].(.name;.age)).

The pattern is parsed and type-checked against the input’s schema before any I/O happens. Mismatches (a .foo on an integer, a slice on a scalar) fail with a diagnostic that points at the offending pattern fragment.

Note that the CLI uses ; as the group separator (.(.a;.b)) so a single argument stays shell-safe; in morloc source the separator is , (.(.a,.b)). Both mean the same thing.

$ morloc-nexus view people.packet --pattern ".[0:10].(.name;.age)" -f json
[["Alice",30],["Bob",25],...]

$ morloc-nexus view timeseries.stream --pattern ".[100000:100010]" -f jsonl
{"t":1.0,"v":3.14}
{"t":1.01,"v":3.15}
...

The pattern dispatches through the same IFile walker IFile a values use in morloc code (mlc_ifile_walk) — for a data packet, a single mmap + slice; for a stream packet with a valid footer, log2(K) seeks over K sub-packets. For footer-less streams, view forward-scans the file to reconstruct the sub-packet index before dispatch (see Footer-less streams below).

9.2.4. Packet-shape output: -s / -d / -p

Three mutually exclusive flags control the packet shape when the output form is a morloc packet:

-s, --stream-packet

Emit a MORLOC_STREAM_PACKET file. The value (or the pattern result) must be a list. Implies -f packet.

-d, --data-packet

Emit a MORLOC_DATA_PACKET file (the default packet shape). Implies -f packet.

-p, --preserve-packet

Mirror the input packet’s shape on output: data-in / data-out, stream-in / stream-out. Errors when the input is not a morloc packet. Implies -f packet.

Bare -f packet (no -s/-d/-p) emits a data packet. Any of -s/-d/-p combined with -f <not packet> is an error.

The four conversion arms all preserve typed semantics through the runtime’s authoritative packet writers:

  • DATA → DATA is a byte-level copy (or a loader re-emit when the --schema differs from the one embedded in the input).

  • DATA → STREAM opens the input as an IFile, chunks the array via bracket-slice, and writes each chunk through an OStream so the output is chunked, compressed, and indexed like any other stream file.

  • STREAM → STREAM drains the input via IStream @next and rewrites via OStream @write, so recompression, schema override, and footer normalisation all flow through the canonical writers. When --compression-level matches the source, a fast path verbatim-copies sub-packet payloads and only rewrites the footer.

  • STREAM → DATA materialises the whole stream to memory via the IFile walker and re-emits as a data packet. Gated by the size guardrail below.

9.2.5. -f jsonl: line-delimited JSON

jsonl output emits one JSON value per input element, one per line. JSON emission is element-by-element in all cases, so peak per-line memory is one element’s JSON body regardless of input size. Input buffering depends on the source:

  • stdin (view - -f jsonl) reads one sub-packet at a time and serialises its elements before reading the next, so a multi-GB producer streams through in constant memory.

  • File-based stream input goes through the buffered loader: the whole list is materialised, then emitted line-by-line. Combine with --pattern .[a:b] (or -f jsonl on stdin) for a constant-memory shape on multi-GB inputs.

9.2.6. Size guardrails and --force

Two guardrails refuse large operations without an explicit --force:

  • Any buffered path (stream input to a non-jsonl text output, stream input to a data packet, single-value binary output) whose projected working set exceeds 1 GiB.

  • Binary output (-f packet, -f mpk, -f voidstar, -f arrow, -f parquet) to a terminal.

--force lifts both. Refusal messages state which guardrail fired and suggest an alternative (-s, -f jsonl, --pattern ".[a:b]", or -o FILE). The buffered-path threshold can be overridden with MORLOC_VIEW_MAX_BUFFER_BYTES.

For footer-less stream input the projected size falls back to the compressed file’s on-disk length as a conservative lower bound, so a partially-written multi-GB stream still hits the guardrail rather than silently OOM’ing the buffered path.

A stream file whose final footer is missing (writer crashed before @close, program > out.stream interrupted, etc.) is still usable through view. The classifier reports the missing footer, and:

  • view -p (or explicit -s) re-emits a well-formed stream with a canonical footer — the runtime’s IStream drains forward regardless of footer state, and OStream writes a fresh footer.

  • view -d (buffered materialisation) succeeds subject to the size guardrail.

  • view --pattern PATTERN forward-scans the file to recover a sub-packet-offset index, then dispatches the walker as normal.

9.2.8. Options

-f, --output-form FORM

Output format: json (default), jsonl, mpk, voidstar, packet, arrow, parquet, csv. Same set as run -f plus jsonl.

-z, --compression-level N

zstd preset 0..=9. Applies to each sub-packet when emitting -s (stream packet).

-o, --output-file PATH

Write to PATH instead of stdout.

-s, --stream-packet

Emit a stream packet (mutex with -d/-p).

-d, --data-packet

Emit a data packet (mutex with -s/-p).

-p, --preserve-packet

Mirror the input packet’s shape (mutex with -s/-d).

--pattern STR

Apply a pattern chain (.[a:b].field, .[:].[0], …​) before emission.

--force

Lift the size and binary-to-tty guardrails.

--schema STRING

Morloc schema string (compact format, e.g. "as" for [Str], "ad8" for [F64]). Overrides the input’s embedded schema.