hydrate-protobuf incremental watermark loop
Summary
hydrate-protobuf incremental mode stores its parent progress watermark on the node
output configuration (incrementalLastParentOutputId).
Before the fix, the incremental path could enter a loop when:
incrementalHydration: trueenableUploadDeduplication: true- the newly built artefact had the same
md5as the node’s current output
In that case the processor logged md5 unchanged, skipped the upload, and completed
without writing a new node output. Because the watermark only advances when a new
output is registered, the next run saw the same parent outputs again and repeated the
same work forever.
Symptom
- A hydrate node repeatedly processes the same parent outputs.
- Every emitted artefact has the same
md5. - Runner slots stay occupied and downstream nodes remain pending.
Mitigation
- For session templates that should not use incremental replay, set
incrementalHydration: false. - If testing a live environment before the runner fix is deployed, disabling
enableUploadDeduplicationalso avoids the skip-upload branch.
Fix
The incremental path for OpenDIS CSV / Signal must not enter a tight reprocess loop when skip-upload leaves the watermark unchanged: Core only reschedules when parent outputs are newer and not byte-identical. Empty windows (no matching PDUs) may re-decode the same parent range for CPU, but they must not upload duplicate artefacts.
hydrate-protobuf-v2 incremental and streaming full-replay paths follow the same
rule: identical md5 → skip upload (no new NodeOutput, no child fan-out).