Node output content facets

Engineering reference for the per-blob content facets on NodeOutputContentIndexes and the facet-filtered outputs listing. Not for the public docs site.

Why

Every hydrate lane used to fetch and decode every agnostic-input blob above its watermark just to learn whether the blob held any of its payload types. On the OdySSEy Lite data the Engagement, Control and Transmitter lanes match a small fraction of blobs, and the gaze lane only blobs with a GazeComponent. The lazy content index already walks every blob once (md5, size, mime, protobuf census); recording what it saw per blob lets Core answer “which outputs above id N can contain type X” without any runner decoding.

Columns (additive, on NodeOutputContentIndexes)

Populated by NodeOutputContentScanService.ScanAsync using the same ProtoValueProfiler capture-discovery uses per blob, on a bounded tee of the logical bytes taken during the streaming scan. Extraction failures never fail the scan: the facet columns stay null with FacetComplete = false. The sweep processor (node-output-index-sweep static) exposes facetWindowBytes; the on-demand scan endpoint accepts it as a query parameter.

Listing filters (GET /api/nodes/{id}/output/all)

Additive query parameters on NodeOutputQueryParams:

• SinceId: only outputs with Id > SinceId.
• ProtoType (repeatable): any-of on ProtoTypes.
• PduType (repeatable): any-of on DisPduTypes.

Semantics: with a facet filter set, a row is excluded only when its index row exists, FacetComplete is true, and none of the requested values are present. Rows with no index row or an incomplete facet are always included. The response adds facetCoverage: { total, complete, maxId } computed over the SinceId range (after SinceId, date and file-type filters, before facet exclusion), so a caller can log coverage and advance a watermark past excluded rows. An empty range still returns 404 as before; a non-empty range whose rows were all excluded by the facet filter returns 200 with empty data, totalCount 0 and populated facetCoverage, because a 404 there would hide maxId and reintroduce the no-op task per tick.

Translation: the exclusion is an anti-join on the index primary key (NOT IN over rows that are complete and lack every requested value, one = ANY(column) predicate per value), which translates on both Npgsql and the in-memory test provider. The GIN indexes on ProtoTypes and DisPduTypes exist for containment and overlap consumers; the per-node listing does not need them.

Consumers

• Python runtime: HeatRuntimeClient.fetch_all_outputs(node_id, since_id=, proto_types=, pdu_types=) keeps facetCoverage on last_facet_coverage.
• core-utils: incremental_hydrate.FacetFilter, iter_parent_outputs_since(..., facet=), last_listing_max_id(client); hydrate-protobuf-journal derives its filter from pduTypes, targetProtoType: GazeComponent, or extractionTarget.
• Not yet using it: hydrate-protobuf and hydrate-protobuf-v2 (unchanged by design), the video atlas scan (StreamComponent would be the natural filter).

Operating notes

• Coverage on a live session depends on the sweep cadence and batch size (defaults: every 15 minutes, 200 outputs). A lane sees no acceleration for blobs the sweep has not reached; it still decodes them.
• Rows scanned before this change have FacetComplete = false and are never excluded. Enable rescan on the sweep static, or truncate the index, to backfill facets.