External request source header (ingress) troubleshooting

Summary

HEAT services distinguish external requests (from outside the cluster, via ingress) from in-cluster requests using the request header X-HEAT-Request-Source: external. Ingress is responsible for injecting it. When injection fails, browser traffic is classified as in-cluster and token validation is skipped.

This page records a real failure on an nginx-class cluster, the diagnosis, and the fix. It exists because the failure is silent: no error is logged by ingress, and only some endpoints break.

Key finding: nginx.ingress.kubernetes.io/proxy-set-headers is not a valid per-Ingress annotation. In ingress-nginx, proxy-set-headers is a global ConfigMap option, not an annotation. An Ingress carrying it is ignored with no warning.

Who reads the header

Header name and comparison live in HeatRequestSourceConstants (present in v2 API, Core API, and HEAT Auth).

Symptom

Only some v2 dashboard endpoints return 401. Others work normally.

• GET /api/v2/dashboard/dimensions returns 401 (session list page fails).
• GET /api/v2/dashboard/{dimensionId}, /layout-config, /data return 200 (session detail page renders).

The split is not about roles or tokens. It is about whether the action reads AuthenticatedUser:

• Controllers that read it (DashboardCatalogController, DashboardOverviewController, DashboardReportsController, DashboardStaticsController) return a bodyless Unauthorized() when it is null.
• Controllers that do not read it (DashboardController) run normally, because RequiresRoleAttribute already short circuited.

Diagnose from the 401 response body

The body identifies the code path uniquely. Use this before changing any configuration.

The RFC9110 ProblemDetails shape is the tell for a missing header. A token problem never produces it.

The header is injected between ingress and the pod. The browser never sends it and never receives it, so browser devtools cannot confirm or deny its presence. Strict-Transport-Security in the response is unrelated (it is a stock ingress-nginx default).

Diagnosis steps

Run in order. All are read only.

1. Confirm the header is the discriminator

port-forward bypasses ingress, reproducing the no-header case with no extra images (works air gapped):

kubectl port-forward -n heat svc/heat-v2-api 5000:5000

Then compare, without and with the header:

curl -i "http://localhost:5000/api/v2/dashboard/dimensions"

curl -i -H "X-HEAT-Request-Source: external" "http://localhost:5000/api/v2/dashboard/dimensions"

If the body flips from ProblemDetails to Missing or invalid Authorization header, the header is the cause.

2. Check the ingress that serves the host

kubectl get ingress -n heat -o custom-columns='NAME:.metadata.name,HOSTS:.spec.rules[*].host,CLASS:.spec.ingressClassName'

Confirm which Ingress matches the host, and that it routes /api/v2 to heat-v2-api.

3. Check whether the header reaches the generated nginx config

This is the authoritative check.

kubectl exec -n app-routing-system deploy/nginx -- grep -c "X-HEAT-Request-Source" /etc/nginx/nginx.conf

Validate the instrument first. A zero result is only meaningful if the file is the live config:

kubectl exec -n app-routing-system deploy/nginx -- grep -c "proxy_set_header" /etc/nginx/nginx.conf

Expect thousands. If that is also zero, the path or container is wrong and the negative result means nothing.

4. Rule out the plausible-but-wrong causes

Each of these was checked and found not to be the cause. Checking them is still worthwhile, because they produce the same symptom.

The RBAC result was a red herring. The controller cannot read ConfigMaps in heat, which is true but irrelevant: it never attempts the read, because it never processes the annotation. The decisive clue was zero RBAC or reflector errors in the controller logs across 8 days. A controller genuinely failing to read a referenced ConfigMap logs errors continuously.

Beware two artefacts that produce false results:

• kubectl auth can-i --as="...:$sa" with an unset $sa evaluates an empty identity and always returns no. Echo the variable before trusting the verdict.
• PowerShell mangles jsonpath containing {"\n"} or |. Use one jsonpath expression per command.

Root cause

deploy/helm/templates/_helpers.tpl emits, for non-traefik classes:

nginx.ingress.kubernetes.io/proxy-set-headers: "<namespace>/heat-external-request-headers"

This annotation does not exist in ingress-nginx:

• The annotations reference  does not list proxy-set-headers. The only similar entry, auth-proxy-set-headers, applies to external auth services only.
• The ConfigMap reference  lists proxy-set-headers as a global option taking a namespace/name value, set on the controller’s own ConfigMap.

So the annotation is silently ignored, and the header is never injected. This affects any nginx-class HEAT deployment using this config shape, not one environment.

Fix applied

Replace the non-existent annotation with configuration-snippet, which is a real annotation, on every Ingress routing /api/v2.

Preconditions on the cluster (verify first):

kubectl get cm nginx -n app-routing-system -o yaml

• allow-snippet-annotations: "true"
• annotations-risk-level permits the annotation (Critical here)
• annotation-value-word-blocklist does not contain proxy_set_header

Back up before changing:

kubectl get ingress heat-ingress-next -n heat -o yaml > backup-heat-ingress-next.yaml

Confirm no snippet already exists, so nothing is replaced:

kubectl get ingress heat-ingress-next -n heat -o jsonpath='{.metadata.annotations.nginx\.ingress\.kubernetes\.io/configuration-snippet}'

Apply, without --overwrite, so the command fails rather than clobbering if a snippet appeared:

kubectl annotate ingress heat-ingress-next -n heat 'nginx.ingress.kubernetes.io/configuration-snippet=proxy_set_header X-HEAT-Request-Source external;'

Repeat for every Ingress that routes /api/v2. On the affected cluster that was both heat-ingress-next (host next.<env>) and heat-ingress-main (host <env>).

Notes:

• Single quotes are required in PowerShell. The value contains a space and a semicolon.
• kubectl annotate patches only the named key. Other annotations (cert-manager, proxy-body-size, timeouts, Helm metadata) are untouched.
• configuration-snippet is a single string of nginx directives. A second annotate on the same key replaces the value. To add a directive to an existing snippet, read the current value and re-set it with the addition appended.

Verification

Header reaches the generated config (was 0, expect a high count):

kubectl exec -n app-routing-system deploy/nginx -- grep -c "X-HEAT-Request-Source" /etc/nginx/nginx.conf

End to end, through the public host, no token required:

curl -s "https://<host>/api/v2/dashboard/dimensions?from=2026-08-26&to=2026-09-02&orderBy=descending&pageNumber=1"

The body must flip from ProblemDetails to:

{"status":"error","message":"Missing or invalid Authorization header"}

That confirms AuthenticationMiddleware is running. Then load the session list in the browser with a real token.

On the affected cluster the header appeared immediately, with no controller restart. That retroactively confirms the diagnosis: the annotation was never read, so this was never a sync timing problem.

Rollback

The annotation did not exist beforehand, so removing it restores the prior state exactly (trailing dash deletes):

kubectl annotate ingress heat-ingress-next heat-ingress-main -n heat nginx.ingress.kubernetes.io/configuration-snippet-

Known limitations and follow-ups

The annotation is manual and will be lost. These Ingresses are Helm managed (meta.helm.sh/release-name: heat). The next helm upgrade reverts them and the 401 returns. Treat the above as a stopgap.

The chart still needs fixing. _helpers.tpl branches only between traefik and the invalid nginx annotation, with no branch for the AKS app routing add on. Options, neither unconditionally correct:

Choosing between them is an architecture decision. Raise with the HEAT engineering lead.

The failure mode is fail open, not fail closed. With the header absent, AuthenticationMiddleware skips token validation rather than rejecting the request. Endpoints that do not read AuthenticatedUser (the whole DashboardController surface) were therefore served to external traffic without token validation. Core API’s ClusterOnly mode reads the same header and needs the same review. Any change here should consider requiring proof of in-cluster origin rather than inferring trust from an absent header.

An unused RBAC grant may exist. If a Role and RoleBinding named heat-external-request-headers-reader was created in heat during diagnosis, it is inert once the snippet fix is used. It is only needed for the global ConfigMap option.

kubectl delete role,rolebinding heat-external-request-headers-reader -n heat

Related

• Platform configuration for system.core.auth_mode
• deploy/helm/templates/ingress-external-traffic.yaml creates the ConfigMap and traefik middleware
• deploy/k8s/base/ingress/ static ingress manifests