hydrate-protobuf-journal (Transform Node)

The Hydrate Protobuf Journal node is the live-session form of hydrate-protobuf and hydrate-protobuf-v2. It decodes the same protobuf payloads with the same decoders, but instead of re-uploading a cumulative artefact on every run it publishes only the rows decoded on that run as an append-only part. The platform reads every live part back as one artefact, so downstream nodes see exactly what the cumulative node would have published.

Use it when a session streams in over time (Capture ticking every few seconds) and the hydrated artefact is large. A cumulative lane writes runs x artefact size to object storage over a session; a journal lane writes about the data size once.

How it works

• Each run decodes only parent outputs newer than the watermark stored on this node’s current output, writes them to one new part object at a deterministic key, and registers that part as the node’s current output.
• The output configuration is the head: an index of every live part (artefactParts), the union CSV header (csvHeaders), the watermark (incrementalLastParentOutputId), and an md5 that changes exactly when the composed artefact changes, so child scheduling behaves as it does for any other node.
• Readers in the runtime and in Core API detect artefactKind: "journal-head" and compose the parts transparently: one header line, then every part’s rows in order, older rows padded to the union width. Consumers need no change and keep inputMode: "latest".
• When the number of live parts reaches maxParts, the run composes them plus the new rows into one part and marks the old rows superseded. Retention reclaims them after the session pruning policy’s minimum age. Amortised write cost is about twice the data size.
• A run that decodes nothing does not publish. The watermark is advanced on the current output so later runs do not re-scan the same range.

Configuration schema

What consumers see

• CSV: one header row (the union of every part’s columns, in first-seen order) followed by every row. Older parts written before a column existed have empty cells for it. Column order is stable: a new column is only ever appended, so SQL written against an early run keeps working.
• NDJSON: the parts concatenated in publish order. Records are ordered within a part by sortBy; a late-arriving blob lands in a later part, the same as the cumulative NDJSON path.
• Gaze CSV: the same fixed header set as hydrate-protobuf with targetProtoType: GazeComponent, so a tabular-query over biometrics needs no SQL change.
• Arbex mode: "latest" reads the composed artefact. mode: "history" lists successive cumulative states, one per run, as it does for any snapshot node.

Facet acceleration

Every run lists parent outputs above the watermark with a content-facet filter: DIS lanes pass pduTypes, the gaze lane passes GazeComponent, NDJSON lanes pass their extractionTarget. Core excludes a blob only when its content index row is complete and holds none of the requested values; blobs without an index row, or with a partial facet, are always returned and decoded as before, so correctness never depends on the index being warm. The index is populated lazily by the node-output-index-sweep static; on a live session the sweep cadence and batch size decide how much of the acceleration a lane sees. The watermark advances past excluded blobs, so a run that finds nothing relevant completes without publishing and without being rescheduled for the same range.

Button presses

With buttonPressEdgesOnly, the edge filter state (last held sample per origin, entity and button) is persisted on the row as buttonPressEdgeState and seeded on the next run, so a hold that straddles two runs is not re-emitted as a fresh press.

Migration from hydrate-protobuf

Create the journal lane as a new node in a new session template revision. A node that already holds cumulative outputs refuses to start a journal beside them.

Limitations

• SimulationFrame, FolderUploadData and SignalPDU are not journal targets today. Use the existing nodes for those.
• Consumers still read the whole composed artefact on each run, as they did with the cumulative artefact. The journal fixes what is written, not what is read.
• The first run of a lane that decodes nothing publishes an empty part so the watermark has a home. The composed artefact is then empty until rows arrive, which is the same behaviour as the cumulative nodes.

Example configuration

{
  "targetProtoType": "HEAT.OpenDISModule.PDU",
  "journalFormat": "csv",
  "incrementalHydration": true,
  "dataSourceName": "HEAT Managed Object Store",
  "excludePduTypes": [2, 3, 13, 14, 20, 21, 22, 23, 24, 25, 26, 27, 28],
  "csvInclude": ["origin_uuid", "packet_timestamp_iso_utc", "pdu.pduType", "pdu.entityID", "pdu.entityLocation"],
  "maxParts": 64,
  "retention": { "skipArchive": true }
}

Related

• hydrate-protobuf
• hydrate-protobuf-v2
• tabular-query
• Core Utils