Skip to Content
This documentation is provided with the HEAT environment and is relevant for this HEAT instance only.
InternalRunnershydrate-protobuf incremental watermark loop

hydrate-protobuf incremental watermark loop

Summary

hydrate-protobuf incremental mode stores its parent progress watermark on the node output configuration (incrementalLastParentOutputId).

Before the fix, the incremental path could enter a loop when:

  • incrementalHydration: true
  • enableUploadDeduplication: true
  • the newly built artefact had the same md5 as the node’s current output

In that case the processor logged md5 unchanged, skipped the upload, and completed without writing a new node output. Because the watermark only advances when a new output is registered, the next run saw the same parent outputs again and repeated the same work forever.

Symptom

  • A hydrate node repeatedly processes the same parent outputs.
  • Every emitted artefact has the same md5.
  • Runner slots stay occupied and downstream nodes remain pending.

Mitigation

  • For session templates that should not use incremental replay, set incrementalHydration: false.
  • If testing a live environment before the runner fix is deployed, disabling enableUploadDeduplication also avoids the skip-upload branch.

Fix

The incremental path for OpenDIS CSV / Signal must not enter a tight reprocess loop when skip-upload leaves the watermark unchanged: Core only reschedules when parent outputs are newer and not byte-identical. Empty windows (no matching PDUs) may re-decode the same parent range for CPU, but they must not upload duplicate artefacts.

hydrate-protobuf-v2 incremental and streaming full-replay paths follow the same rule: identical md5 → skip upload (no new NodeOutput, no child fan-out).