Kubernetes: observe mode

Observe is the default mode and the heart of the Kubernetes integration. One controller watches the cluster, builds a normalized request from administrator-owned state, calls the core evaluator, and reports the result — it changes nothing about scheduling and selects no GPU.

Report-only

The observer does not change scheduling, does not select a GPU, and does not evict. It treats mock inventory as mock evidence, never as attestation. Enforcement is a separate, opt-in path — see enforcement.

What the observer does

1. Resolves a Pod's tenant from administrator-owned TenantBinding objects keyed on the Namespace UID (never a tenant-supplied claim).
2. Builds a core EvaluateRequest from a PlacementPolicy, a NodeSecurityState snapshot, and the Pod's guard.xila.us/* annotations, and POSTs it to the core evaluator.
3. Exports the resulting PlacementDecision as a JSONL record and a bounded, redacted Kubernetes Event.
4. Detects drift — an outcome change or lost evidence across evaluations — and reports it (report-only, no eviction).

For a bound Pod, the observer evaluates its observed node at node scope. For an unbound Pod, candidate evaluation is advisory. The report states what was evaluated and whether device allocation was actually known.

flowchart TD
    POL[PlacementPolicy] --> OBS[Observer]
    TB[TenantBinding<br/>by Namespace UID] --> OBS
    NSS[NodeSecurityState<br/>snapshot] --> OBS
    POD[Pod + guard.xila.us annotations] --> OBS
    OBS --> CORE[core POST /v1alpha1/evaluate]
    CORE --> DEC[PlacementDecision]
    DEC --> JSONL[decisions.jsonl / drift.jsonl]
    DEC --> EV[Kubernetes Events<br/>PlacementAllow / PlacementDeny / PlacementDrift]

Opting a Pod into modeled-placement observation

Add these annotations (CPU-only Pods — do not request nvidia.com/gpu in kind):

metadata:
  annotations:
    guard.xila.us/evaluate: "true"
    guard.xila.us/policy: "<PlacementPolicy name>"
    guard.xila.us/candidate-node-uid: "<nodeUid of a NodeSecurityState>"
    guard.xila.us/candidate-scope: "node"

Controller flags

The lab depends on these stable flags:

Flag	Default	Meaning
--kubeconfig	KUBECONFIG / in-cluster	kubeconfig path (registered by controller-runtime)
--core-url	http://127.0.0.1:8080	core server base URL
--mode	observe	only observe is implemented; others exit with an error
--decisions-file	./xila-guard-decisions.jsonl	JSONL decision records
--drift-file	./xila-guard-drift.jsonl	JSONL drift records
--once	false	evaluate eligible Pods once, write outputs, exit 0

With the opt-in authenticated-inventory feature, the observer can additionally call evaluate-node over mTLS against SVID-bound inventory (--authenticated-inventory=off|prefer|require); see identity.

What a run looks like

From a live make demo-kind observer run (CPU-only Pods, registry.k8s.io/pause; no nvidia.com/gpu requested):

evaluated 3 eligible pod(s), 0 failure(s)
{"outcome":"allow","mode":"observe","source":"mock","reasonCodes":[]}
{"outcome":"deny","mode":"observe","source":"mock","reasonCodes":["PARTITIONING_DISALLOWED","TIME_SLICING_FORBIDDEN"]}
{"outcome":"indeterminate","mode":"observe","source":"mock","reasonCodes":["INVENTORY_INCOMPLETE"]}
drift: {"podName":"pod-allow","previousOutcome":"allow","currentOutcome":"deny",
        "reasonCodes":["TIME_SLICING_FORBIDDEN"],"mode":"observe","source":"mock","action":"report-only"}

Each Kubernetes Event message ends with "observation only, no GPU selected or enforced".

Drift and operational behavior

The observer re-evaluates active observed placements when relevant policy, binding, or inventory changes; a timer also checks expiration. It emits a finding when a previously allowed placement becomes denied or indeterminate, or when required monitoring is unavailable. Initial behavior is alert / report only — no automatic Pod eviction, checkpoint, migration, node drain, GPU reset, or credential revocation.

Every decision record includes the workload UID, tenant reference, candidate scope, effective policy digest, schema / engine version, evidence source references and times, outcome, findings, and mode. Model names, prompts, payloads, secret values, and unneeded process data are excluded. Metrics are bounded (outcomes, latency, source age, collector failures, integration errors) and avoid workload / tenant IDs as labels.