Journal head output contract

Engineering reference for append-only journal artefacts, produced today by hydrate-protobuf-journal (runners/core-utils/processors/hydrate_protobuf_journal.py, helpers in journal_parts.py) and read by the Python runtime (runtimes/python/heat_runtime/src/heatruntime/journal_head.py) and Core API (core/api/Utilities/NodeOutputJournalHead.cs, NodeOutputJournalComposer.cs). Not for the public docs site.

Why one row per run

The runtime completes a node as soon as any output is registered (HeatRuntimeClient.fail_node refuses to fail once task._has_output is set) and RegisterNodeOutputAsync always makes the newest row the current output. A “register part, then register head” design would leave a bare delta as the current output if the process died between the two. So the part IS the row: its blob is the new data and its configuration is the head. Every row is a complete, self-describing state, and Arbex history mode lists successive cumulative states as it does for snapshot nodes.

Row configuration (flat keys, patch-safe)

OutputIdentifier is journal-p{index:06d}. Registration is idempotent on it, so a retry after a crash gets the earlier row back; the producer overwrites the same key and patches the row when partMd5 differs.

Byte invariants

1. CSV header prefix: a part’s header is a prefix of every later head’s csvHeaders. merge_csv_headers only appends, so this holds by construction. The newest part’s header equals the union.
2. Every part ends with a line terminator. Readers add one defensively.
3. No CR or LF inside a CSV cell (the producer sanitises), so padding is line-wise: ","*(union - partWidth) before the terminator, no CSV parsing.
4. CSV terminator is CRLF (csv.writer default). NDJSON uses \n.
5. Composite CSV = union header line, then each part’s bytes after its first line, each line padded to union width. When csvHeaders is empty (a first run that decoded nothing) the composite is empty with no header line. Composite NDJSON = concatenation.
6. Parts are never heatx and never carry start or len.
7. Readers derive a part’s width from its own first line, not from csvHeaderCount.

Compaction

At publish time, when partCount >= maxParts (default 64, 0 disables), the run composes every live part (through the platform reader on its own current row) plus the new rows into one file under the widest header, uploads it as the next part with artefactParts = [self], bumps journalCompactionCount, and patches every prior row with artefactState: "superseded", retentionEligible: true, supersededAtUtc, supersededReason: "journal-compaction", supersededByOutputId.

Retention and archive

• PruneSupersededOutputsAsync never prunes a row whose S3 key is referenced by the node’s current output’s artefactParts, even on a prunable node or when marked superseded by mistake.
• Declared-superseded rows are due only when (supersededAtUtc ?? CreatedAt) <= cutoff, so compaction-superseded parts survive the minimum age while a consumer that claimed just before may still be streaming them.
• ArchiveSessionInternalAsync keeps rows referenced by the keeper’s configuration. Journal nodes must still declare retention.skipArchive: true (the template lint enforces it).
• Session export writes each part as its own zip entry; parts are the data. The content index records each part’s own md5 and size.

Reader locations

• Python: S3StorageService.get_data and stream_data_to_file dispatch to journal_head.compose_journal_bytes / compose_journal_to_file; BlobStorageService.get_data likewise. This covers fetch_latest_inputs, fetch_latest_input_to_file, fetch_latest_full_input_data, fetch_all_inputs, fetch_blobs_for_output_metas, and incremental_hydrate.stream_input_data_to_file (used by the producer for compaction).
• Core API: NodeInstanceService.GetNodeOutputDataAsync composes when returnRawStoredBytes is false; returnRawStoredBytes: true returns the row’s own part blob. Covers /api/node-instances/outputs/{id}/data, Arbex LoadOutputBytesAsync, and every system processor that reads parent bytes. GetLatestNodeInstanceDataAsync prefers CurrentOutputId.
• NodeOutputBlobLocator callers (session export, content scan, capture discovery, protobuf extract) resolve a journal row to its own part blob, which is what they want.

Facet-filtered listing

iter_parent_outputs_since(client, task, watermark, facet=FacetFilter(...)) calls GET /api/nodes/{id}/output/all?SortOrder=desc&SinceId={watermark}&ProtoType=...&PduType=.... The response carries facetCoverage: { total, complete, maxId } over the SinceId range before exclusion; the runtime client keeps it on client.last_facet_coverage and last_listing_max_id(client) reads maxId. The journal node raises its published watermark to maxId (blobs excluded above the newest returned one are covered by a complete facet) and, when the filtered list is empty, patches the watermark forward without publishing so Core does not schedule a no-op task per tick. Older Core APIs ignore the parameters; older runtime clients without the keywords fall back to the unfiltered listing.