Skip to Content
This documentation is provided with the HEAT environment and is relevant for this HEAT instance only.
RunnersCore UtilsHydrate Protobuf Node

hydrate-protobuf (Transform Node)

The Hydrate Protobuf node reads binary or JSON payloads from a HEAT capture bundle (e.g. from an upstream ingest or node that emits DataPacketBundle-style data), extracts a specific protobuf type from the bundle, and writes the hydrated payloads as JSON, CSV, or raw binary.

Use this node when you need to turn opaque protobuf blobs into a structured format (JSON/CSV) for downstream analytics, dashboards, or export. The node supports multiple inputs and can aggregate simulation frames into a single time-ordered stream when configured.

Configuration Schema

PropertyTypeRequiredDescription
targetProtoTypestringYesFully-qualified protobuf type inside the bundle. Supported values depend on the runner implementation (e.g. GazeComponent, FolderUploadData, HEAT.OpenDISModule.PDU).
outputFormat"json" | "csv" | "bin"YesEncoding for the output. json and csv produce human-readable or tabular data; bin writes the raw serialized protobuf bytes. Defaults to json.
lineDelimitedJsonbooleanNoAdditive. When true, valid with targetProtoType SimulationFrame or HEAT.OpenDISModule.PDU, outputFormat json, and includeEvents false. Emits newline-delimited JSON (JSON Lines): each line is one JSON object with the same shape as one element of the legacy wrapper array (simulation_frame or open_dis_pdu), not the wrapped document. Default false preserves the single JSON object with that array. Downstream json-flatten should use jsonInputFormat: "ndjson" and usually empty root. Memory: when lineDelimitedJson is true and incrementalHydration is false, the runner streams decode (SimulationFrame also k-way merges) so full replay stays within cgroup memory. For SimulationFrame only, set env HEAT_RUNTIME_HYDRATE_BUFFER_SIMFRAME=1 to restore the old in-memory path.
aggregateSimulationFramesbooleanNoWhen targetProtoType is a simulation frame type, combine all frames into a single time-ordered stream. Defaults to true.
emitFrameIndexbooleanNoWhen aggregating frames, include a synthetic frame index column/field in CSV/JSON. Defaults to true.
enableUploadDeduplicationbooleanNoCompute an MD5 hash of the output and skip uploading if the same hash already exists for this node instance. Defaults to false.
incrementalHydrationbooleanNoOpt-in. When true, only decode parent outputs newer than the watermark stored on this node’s prior output, and append to a cumulative artefact using streaming I/O. For outputFormat: json, requires lineDelimitedJson: true (NDJSON). For HEAT.OpenDISModule.PDU with outputFormat: csv, appends CSV rows and stores csvHeaders on the output config (no lineDelimitedJson). For GazeComponent with outputFormat: csv (opt-in), appends gaze rows under the fixed gaze header set and skips upload when the cumulative md5 is unchanged; a bio node config without this flag keeps the full replay unchanged. For SignalPDU with outputFormat: bin, appends decoded PCM into a cumulative WAV and skips upload when the cumulative md5 is unchanged so reprocess and non-audio parent batches do not churn object storage or reschedule transcript. Default false still runs a memory-bounded OpenDIS CSV full replay (batched parent fetch, no fetch_all_inputs); prefer true for live Capture so later runs only decode new blobs.
csvIncludestring[]NoAdditive. OpenDIS outputFormat: csv only. Whitelist of dotted column prefixes (same semantics as json-flatten include). A flattened key is kept when it equals an entry or starts with entry + ".". Omitted or empty keeps all columns. When set, also stored on the output config next to csvHeaders.
groupByOriginbooleanNoAdditive, SignalPDU only. Default false keeps one mixed-down WAV. When true, audio is split by capture origin and published as a ZIP (see Per-origin audio). Each origin stream is silence-padded across its own idle gaps, so an offset inside a stream plus that stream’s streamStartUnixMs gives wall clock. With incrementalHydration: true the ZIP holds only the window since the watermark, and downstream transcription merges windows per origin.

Incremental hydration (opt-in)

Incremental hydration decodes only new parent outputs, but it still re-uploads the whole cumulative artefact on every run where the bytes changed. On a live session that writes runs x artefact size to object storage. For live Capture lanes use hydrate-protobuf-journal, which publishes only the new rows as an append-only part and lets the platform compose the parts for consumers.

When incrementalHydration is true (default false):

  • The node processes only new parent NodeOutput records since the last successful run (watermark: incrementalLastParentOutputId on the node’s output configuration).
  • Prior artefact bytes are stream-copied from object storage into a temp file. New rows are written as CSV segments during decode; header growth opens a new segment and at most one merge runs at the end of the run (no full NDJSON value spill). Status heartbeats keep long first-pass hydrates from being reaped by the Core API stale-task timeout. Identical cumulative md5 skips re-upload so latest-only children (for example tabular-query) are not rescheduled for no content change.
  • On every such skip the node still advances its watermark by patching incrementalLastParentOutputId onto its current output’s configuration (PATCH /api/node-instances/outputs/{id}/configuration), so consumed-but-unmatched parent outputs are never re-streamed or re-decoded on later runs. Without this, a lane whose PDU filter matches nothing for a stretch (Engagement/Control are the usual cases on live Capture) would re-scan an ever-growing parent range on every trigger.
  • Each run still produces one cumulative artefact. Downstream nodes should keep inputMode: "latest" (do not switch to all).
  • NDJSON path: targetProtoType SimulationFrame or HEAT.OpenDISModule.PDU, outputFormat: json, and lineDelimitedJson: true.
  • CSV path (OpenDIS): targetProtoType HEAT.OpenDISModule.PDU, outputFormat: csv. Rows use dotted keys compatible with json-flatten DIS CSVs (for example pdu.pduType). Header list is persisted as csvHeaders on the output config. New columns discovered in a run expand the header list with at most one rewrite of the prior cumulative CSV for that run (new rows are spilled during decode, then materialised). Optional csvInclude can narrow columns when a template intentionally wants a subset; omit it to keep the full flattened DIS surface for SQL. Prefer this path when the next step is tabular-query (skip json-flatten).
  • CSV path (GazeComponent): targetProtoType GazeComponent, outputFormat: csv, no lineDelimitedJson. Columns are the fixed gaze header set, identical to the full-replay CSV, so a tabular-query over biometrics needs no SQL change; rows are appended per parent blob with the same cell rendering as full replay. Identical cumulative md5 skips upload and still advances the watermark. gazeDecodeFailures on the output config counts malformed payloads that were skipped.
  • WAV path (SignalPDU): targetProtoType SignalPDU, outputFormat: bin, groupByOrigin false or omitted. Prior cumulative WAV is streamed, new Signal PDUs since the watermark are decoded to PCM and appended, then one cumulative WAV is published. Identical cumulative md5 skips upload (reprocess / telemetry-only parent batches do not create new blobs). Downstream transcript nodes must declare inputMode: "latest" and read only the latest parent artefact.
  • Per-origin ZIP path (SignalPDU + groupByOrigin: true): the only incremental path that is not cumulative. The prior artefact is never read: each run publishes a delta-only ZIP covering just the window since the watermark, so a long capture never rewrites every origin stream on every run. Upload is skipped when the per-origin audio md5 (originAudioMd5 on the output config) is unchanged. History is reassembled downstream by generate-transcript, which merges each window into its prior transcript keyed by originUuid. Downstream nodes still declare inputMode: "latest".
  • Downstream contract: the published artefact is always cumulative (prior rows + new). tabular-query only reads the parent latest output, which is correct for this path. Do not combine all historical hydrate outputs (that would double-count). If the prior cumulative CSV cannot be streamed on a later run, the node fails rather than emitting a delta-only file.
  • Output config also records cumulativeArtefact: true, csvRowCount, and csvRowsAdded for operator visibility.

Supported protobuf types

The runner implements decoders for a fixed set of types. Typical values include:

  • SimulationFrame , Per-component rows from simulation frames; JSON output is typically an object with a simulation_frame array, or NDJSON (lineDelimitedJson) with one row object per line.
  • GazeComponent , Gaze and biometric sensor data; output is JSON with a gaze_component array, or CSV (outputFormat: csv), optionally incremental (incrementalHydration: true, opt-in) for a cumulative tabular artefact. Every row carries origin_uuid, the capture machine that produced it, so two headsets that report under the same entity id stay separable downstream (for example in cognitive-load-metrics).
  • FolderUploadData , Uploaded file/folder content; time-series CSV content is merged into a single artefact.
  • HEAT.OpenDISModule.PDU , OpenDIS PDU records; default JSON is {"open_dis_pdu":[...]}, or NDJSON (lineDelimitedJson) with one flattened PDU record per line, or incremental CSV (outputFormat: csv + incrementalHydration) for a cumulative tabular artefact. Optional excludePduTypes (for example [26] for Signal) drops PDU types during decode. Optional csvInclude narrows CSV columns.
  • SignalPDU , DIS Signal audio merged into one WAV (outputFormat: bin), or one WAV per capture origin inside a ZIP when groupByOrigin: true. Prefer incrementalHydration: true for live Capture so each run only decodes new parent blobs and skips upload when the audio is unchanged.

If you specify a targetProtoType not implemented by the runner, the node fails with an error. Check the runner’s supported types when authoring the configuration.

Per-origin audio (groupByOrigin)

SignalPDU audio is only useful for attribution if you can tell who transmitted and when. A single mixed WAV loses both: bursts from every radio are concatenated, so the timeline is compressed and speakers are interleaved.

With groupByOrigin: true the node reads the origin id and packet timestamp off each decoded PDU wrapper, buckets the audio per origin, and publishes a ZIP:

manifest.json
origins/<originUuid>.wav

manifest.json looks like this:

{
  "version": 1,
  "mode": "delta",
  "groupByOrigin": true,
  "watermarkFrom": 812,
  "watermarkTo": 947,
  "origins": [
    {
      "originUuid": "5f2c...",
      "sampleRate": 8000,
      "sampleWidth": 1,
      "streamStartUnixMs": 1763040000000,
      "streamEndUnixMs": 1763040182500,
      "path": "origins/5f2c....wav",
      "md5": "…",
      "pcmBytes": 1460000
    }
  ]
}

Notes for template authors:

  • mode is "full" on a full replay and "delta" under incrementalHydration, where watermarkFrom / watermarkTo bound the window.
  • Each origin WAV is silence-padded across that origin’s idle gaps, so streamStartUnixMs + offsetSeconds * 1000 is wall clock for that stream. Padding is relative to the first chunk in the artefact, which is why every origin carries its own anchor.
  • Emitters that do not report an origin id are bucketed under unknown rather than dropped.
  • ZIP bytes are written deterministically, so identical audio hashes identically and the upload skip is reliable.
  • Sample formats are reconciled per origin, not globally: two radios at different sample rates both survive.

Example configuration

{
  "targetProtoType": "HEAT.OpenDISModule.PDU",
  "outputFormat": "json",
  "aggregateSimulationFrames": true,
  "emitFrameIndex": true,
  "enableUploadDeduplication": false
}

Inputs and outputs

  • Inputs: The node accepts multiple inputs (e.g. from a single upstream node that emits a list of payloads, or from several nodes). All payloads are collected and decoded according to targetProtoType.
  • Output: One or more artefacts are produced. For outputFormat json or csv, the content is UTF-8 encoded. For bin, the artefact is the raw serialized protobuf bytes.

Integration in a session template

  1. Place a hydrate-protobuf node after a node or ingest that produces HEAT capture bundles (or raw protobuf payloads the runner can decode).
  2. Set targetProtoType to the type you want to extract and outputFormat to the desired encoding.
  3. Connect downstream nodes (e.g. tabular-query, json-template, or dashboard nodes) to consume the hydrated output.
converge-outputsHydrate Protobuf Journal Node