Why you can leave it running
Fail-closed at every layer.
Four guarantees carry the rest. A system that can drain workloads and replace nodes
earns the right to run unattended by refusing more often than it acts.
Configuration selects actions. It never injects commands.
Remediation steps come from a closed, allow-listed enum of typed actions. There is no
field anywhere in the API that accepts a shell command, so a compromised config cannot
become arbitrary code on your GPU nodes.
action: drain-and-reset
approval: required
# TTL 30m
maxConcurrent: 1
Real execution must be asked for
Dry-run is the default and runs the whole ladder on paper. Turning it off needs an
explicit node selector and an exact acknowledgement sentence, and it arms only the
agents on the nodes you named — so the blast radius is declared, never global.
Human authority where it matters
Destructive rungs can require approval from a verified identity, and the approval is
bound to the exact plan the human was shown. Change the plan and the old approval is
refused rather than reused.
Unknown state stops the action
Missing evidence, a stale runtime report, an unresolvable blast radius or a partitioned
GPU all halt the step instead of guessing. Every refusal is counted, so what the
control plane declined to do is as visible as what it did.
Also in the box: mTLS with Pod-bound identity, PostgreSQL HA with leader election, durable
workflow state that survives a controller crash without duplicating a side effect, signed
multi-arch images with SBOMs, and a full audit trail.
Read the safety model →
Clear by design
The important answers.
Can I run this in production today?+
Yes — start with a controlled pilot in dry-run. Kernel XID detection, GPU inventory through
nvidia-smi mounted from the node, incident workflows, approvals with a verified
operator identity and the full audit trail all run against your real fleet while destructive
actions stay disabled. Validated end to end on a real Tesla T4 using an injected XID-79 signal.
Real execution is supported when you are ready for it, behind an explicit node selector, an
acknowledgement sentence and the approval gates you set.
How is this different from DCGM Exporter or node-problem-detector?+
Those tell you something is wrong. KubeNeuron decides what to do about it and carries that
decision through a gated workflow with an audit trail. It consumes dcgm-exporter rather than
replacing it, and it deliberately stays out of general node health — use established tooling
for unrelated disk, network, and kernel problems.
What happens if the controller fails over mid-remediation?+
The workflow store is transactional and the action queue is durable. Leader election is
lease-based, and failover replay reattaches to the same queued action rather than issuing a
second one. This is covered by the integration harness, not just by design intent.
Does it need a service mesh or external PKI?+
No. Agent identity is mTLS over TLS 1.3 plus a projected, Pod-bound ServiceAccount token
verified through TokenReview against the live ServiceAccount UID, Pod owner, node binding,
and DaemonSet. Istio and cert-manager are optional conveniences, not dependencies.
Which accelerators are supported?+
NVIDIA is the one validated on hardware, targeting fleets of roughly 10 to 500 GPU
nodes running the NVIDIA GPU Operator. AMD detection ships — kernel-log and
amd-smi sources feeding the same neutral fault catalog — but it has
never run on AMD silicon, and AMD gets detection, protection and incident closure
rather than device repair: nothing arms and no reset exists. Intel is a seam with no
implementation behind it.
Which fleet you run changes what works, enough that
the
pilot checklist opens with a table of it rather than burying it in a footnote.
Build the safe path first
Run a calmer GPU fleet.
Read the source, run the dry-run ladder, and tell us where the policy model breaks.