Skip to Content
This documentation is provided with the HEAT environment and is relevant for this HEAT instance only.
RunnersCore UtilsHydrate Protobuf v2 Node

hydrate-protobuf-v2 (Transform Node)

The Hydrate Protobuf v2 node decodes HEAT capture data shaped as DataPacket / DataPacketBundle (see HEAT.Proto.Capture in the protolib), walks every nested packet, and extracts a single kind of simulation payload from each SimulationFrame: the full frame, each EntityState, each EntityDataEvent (from frame.events where the EntityEvent carries data_event), or each TrackingEvent (from tracking_event).

Use this node when you need capture-aware extraction with stable envelope fields (parent input key, packet timestamps, origin_uuid, simulation tick/time) across all upstream parents. Optional projection filters prune components and identity so domain trees avoid multi‑hundred‑MB overhydration. The original hydrate-protobuf node remains for Gaze, FolderUploadData, OpenDIS PDU, and its legacy SimulationFrame component flattening.

Capture hierarchy

Configuration schema

PropertyTypeRequiredDescription
extractionTargetstringYesOne of SimulationFrame, EntityState, EntityDataEvent, TrackingEvent.
outputFormat"json" | "csv"Yesjson: UTF-8 JSON object with a top-level records array (or NDJSON when lineDelimitedJson is true). csv: scalar columns plus record_json (full row JSON) per line.
sortBystringNonone (default), simulation_time_seconds, or packet_timestamp_unix_ms for stable ordering after merging parents.
dataSourceNamestringNoOutput data source; defaults to the runner default_store when omitted.
enableUploadDeduplicationbooleanNoLegacy flag. Streaming full replay and incremental paths always skip upload when the artefact md5 matches the prior output (live anti-baggage). Default false.
lineDelimitedJsonbooleanNoWhen true with outputFormat: json, emit one JSON record object per line (NDJSON) instead of a records wrapper. Required when incrementalHydration is true unless cumulativeRecordsJson is set. Default false.
incrementalHydrationbooleanNoOpt-in. Stream-append only new parent outputs to cumulative NDJSON; requires lineDelimitedJson (or cumulativeRecordsJson). Supported for EntityDataEvent and TrackingEvent. Default false (full replay).
cumulativeRecordsJsonbooleanNoOpt-in. With incrementalHydration, publish the cumulative { "records": [...] } object (the shape Arbex openJsonReader needs) instead of NDJSON. The prior artefact is re-read as the base and only parent outputs above the watermark are decoded. Prior records are held in memory, so use it for filtered lanes (edges-only button presses), not raw telemetry. Cannot be combined with lineDelimitedJson. Default false.
includeComponentTypesstring[]NoAllow-list of Component oneof arm names (for example world_position, dynamics). Mutually exclusive with excludeComponentTypes. Applies to EntityState and SimulationFrame.
excludeComponentTypesstring[]NoDeny-list of component oneof arms. Mutually exclusive with includeComponentTypes.
dropEmptyEntitiesbooleanNoWhen component filters are set, omit entities with zero remaining components. Default true.
filterOriginUuidstringNoExact match on capture origin_uuid.
filterEntityIdintegerNoExact match on entity id (EntityState entity.id or event entity_id).
filterEntityNamestringNoExact case-sensitive match on EntityState entity.name.
entityNameRegexstringNoPython regex fullmatch on EntityState entity.name. AND-combined with other identity filters.
includeDataEventTypesstring[]NoAllow-list of EntityDataEvent event_data oneof arms (for example button_press).
buttonPressEdgesOnlybooleanNoWhen true, keep only button_press edges: emit on first sample, on event_data.state change, or after a silence longer than buttonPressEdgeGapMs. Drops held/polled ticks (common for joystick or 2D axis readouts reported as buttons). Default false.
buttonPressEdgeGapMsintegerNoGap in milliseconds that ends a hold when buttonPressEdgesOnly is true. Default 100.
includeTrackingEventTypesstring[]NoAllow-list of TrackingEventType enum names (for example TRACKING_EVENT_TYPE_FIRED). Applies to TrackingEvent and to EntityDataEvent.event_type when present.

When no projection filters are set, behavior matches the previous full-tree extraction.

Incremental hydration (opt-in)

Incremental hydration still re-uploads the whole cumulative NDJSON on every run where the bytes changed. For live Capture lanes use hydrate-protobuf-journal with extractionTarget and journalFormat: "ndjson", which publishes only the new records as an append-only part.

Same contract as hydrate-protobuf: cumulative NDJSON per run, watermark on output config, downstream inputMode: latest. Supported for EntityDataEvent and TrackingEvent extraction.

Live anti-baggage: when the newly built artefact has the same md5 as the node’s current output, the processor completes without uploading. That matches OpenDIS CSV incremental behaviour and avoids creating a new NodeOutput (and child fan-out) on every agnostic-input tick when nothing in this lane changed. The watermark stays on the prior output until content actually changes.

For button_press streams, combine buttonPressEdgesOnly with incrementalHydration so the cumulative artefact stays small and only grows when real edges appear.

Records-JSON variant (cumulativeRecordsJson: true): same watermark and md5-skip contract, but the artefact stays a records object. With buttonPressEdgesOnly, the edge filter state (last held sample per origin, entity and button) is persisted as buttonPressEdgeState on the output configuration and seeded on the next run, so a hold that straddles two triggers is not re-emitted as a new press. On an md5 skip the watermark and edge state are patched onto the prior output. Samples older than the seeded state (retroactive uploads) fall back to state-change detection only; a full replay remains the order-exact path.

Output shape (JSON)

The artefact is a JSON object:

{
  "extractionTarget": "EntityState",
  "records": [
    {
      "parent_input_key": "upstream-node-name",
      "payload_index": 0,
      "inner_packet_index": 2,
      "bundle_timestamp_unix_ms": null,
      "packet_timestamp_unix_ms": 1710000000000,
      "packet_timestamp_iso_utc": "2024-03-09T12:00:00.000Z",
      "origin_uuid": "…",
      "simulation_tick": 42,
      "simulation_time_seconds": 1.5,
      "capture_time_timestamp_ms": null,
      "capture_time_iso_utc": null,
      "capture_time_timezone_offset_minutes": 0,
      "extraction_target": "EntityState",
      "entity_state_index": 0,
      "entity_state": { }
    }
  ]
}

For EntityDataEvent / TrackingEvent, each record follows the decoder event dictionary layout (event_category, event_type, event_data, …) plus the same envelope fields merged in.

Example configurations

Full EntityState (legacy behavior):

{
  "extractionTarget": "EntityState",
  "outputFormat": "json",
  "sortBy": "simulation_time_seconds",
  "dataSourceName": "my-store"
}

Domain tree (world position only, named entities):

{
  "extractionTarget": "EntityState",
  "outputFormat": "json",
  "includeComponentTypes": ["world_position", "dynamics"],
  "entityNameRegex": "^(Blue|Red)-.*",
  "dropEmptyEntities": true
}

Tracking timeline for one entity:

{
  "extractionTarget": "TrackingEvent",
  "outputFormat": "json",
  "lineDelimitedJson": true,
  "incrementalHydration": true,
  "filterEntityId": 42,
  "includeTrackingEventTypes": ["TRACKING_EVENT_TYPE_FIRED", "TRACKING_EVENT_TYPE_HIT"]
}

Inputs and outputs

  • Inputs: All parents are merged. Each parent contributes a list of binary DataPacket payloads (same wire format as v1).
  • Output: One JSON or CSV artefact. Unlike v1, every parent key in fetch_all_inputs is processed, not only the first.
Hydrate Protobuf Journal NodeJSON Flatten Node