13.8. Tables and tensors on the wire
Tables are columnar, and every language with a serious table library already agrees on one columnar memory format: Apache Arrow. So a table does not travel as voidstar. It travels as Arrow data laid out in one shared-memory block, which each language imports without copying. This section specifies that block, how a table’s declared columns constrain it, and how tensors, which are ordinary voidstar values, carry their dimensions.
The table block. A table is always the whole value of a data packet, sent by pointer with format Arrow (Packet protocol), even when shared memory is otherwise disabled. Tables cannot be nested inside other values. The block holds a header, then node descriptors, buffer descriptors, a string table, and the data buffers, each buffer aligned to 64 bytes from the start of the block, with zeroed padding.
| Offset | Size | Field | Meaning |
|---|---|---|---|
0 |
4 |
magic |
|
4 |
4 |
version |
2. Any other version is rejected. |
8 |
4 |
columns |
Number of columns; equals the root node’s child count. |
12 |
4 |
nodes |
Number of node descriptors. |
16 |
8 |
rows |
Number of rows; equals the root node’s length. |
24 |
8 |
total size |
Bytes in the block; every offset is checked against it. |
32 |
8 |
metadata offset |
Reserved, 0. |
40 |
4 |
metadata length |
Reserved, 0. |
44 |
4 |
buffers |
Number of buffer descriptors. |
48 |
4 |
nodes offset |
Where the node descriptors start. |
52 |
4 |
buffers offset |
Where the buffer descriptors start. |
56 |
4 |
strings offset |
Where the string table starts. |
60 |
4 |
strings length |
Bytes in the string table. |
A node is one Arrow array: the root is a struct whose children are the columns, and a nested column has nodes of its own. Each 64-byte node descriptor carries what the Arrow C Data Interface needs to rebuild the array: length, null count, flags, the format string and column name (as offsets into the string table), and the index and count of its buffers and children. Each 32-byte buffer descriptor gives a buffer’s offset and size within the block. Slices are compacted and dictionaries decoded before a table is written, so every node starts at offset 0 and has no dictionary.
Zero-copy import. A receiver validates the block (every offset in bounds,
every buffer long enough for its node), then builds Arrow C Data Interface
structures whose buffers point into the block, holding a reference on it for
as long as the table lives. Python receives a pyarrow record batch, R an
arrow record batch, C++ a nanoarrow-based handle, and Rust an
arrow-rs record batch, each without copying the data. Converting further,
to a pandas or R data frame, is the language library’s own copy. Sending a
table writes a new block, except that a pool returning a table it received,
unchanged, passes the same block on.
Declared columns. A table’s type lists columns (T: in its schema;
Voidstar format and schema strings), and that list is a lower bound: a table may have more
columns, which travel after the declared ones. A declared column may be a
boolean, integer, float or string, or an optional of one of these; any Arrow
type may appear among the extra columns. On send, a declared column of
another type is cast if the cast loses nothing, and refused otherwise. On
receive, every declared column must be present with an acceptable type, and
a column not declared optional must have no nulls. The number of rows is
not part of the type and is not checked.
Tables from files. A table argument given as a file is read in the pool,
not the nexus. Arrow IPC and Parquet are recognized by their magic bytes;
otherwise the extension (.csv, .tsv, .json) or the content decides.
CSV columns bind to declared columns by header name. As output, the nexus
writes a table as JSON rows, JSON lines, CSV, TSV, Arrow IPC or Parquet.
Tensors are not a runtime type. A tensor is a pair of its dimensions and
its elements in one flat, row-major array, and travels as that ordinary
voidstar tuple: Matrix 2 2 Real has schema t2t2jja:4f8. Because the
array of numbers is 64-byte aligned in shared memory, Python receives a
read-only numpy view over it with no copy. When the dimensions are known at
compile time, the array’s length is checked against their product.