New Latest release on GitHub

Open source — for Kubernetes GPU fleets

A GPU fleet reliability control plane.

DetectsProtectsRecoversMeasures

Vendor-neutral by construction: KubeNeuron detects degradation, protects the workloads running on the device, automates safe recovery through a policy ladder with approval gates and a complete audit trail — and measures recovered capacity, so what the fleet got back is a number rather than a feeling.

incident / ip-192-168-31-60 ENABLED
XID 79
GPU has fallen off the bus

nvidia0 · detected in kernel log · hardware reset refused on this instance

  1. 01Observe & correlate
  2. 02Cordon node
  3. 03Drain workloads
  4. 04Approve & replace node
  5. 05Verify & close as replaced
waiting for signal…
curl -sfL https://github.com/kubeneuron/kubeneuron/releases/latest/download/install.sh | bash -s -- --version latest

Dry-run by default: the workflow runs end to end and records exactly what it would do, so you see the decision before anything touches a node.

Runs alongside the stack you already have

  • Kubernetes
  • NVIDIA GPU Operator
  • DCGM Exporter
  • VictoriaMetrics
  • Alertmanager
  • Grafana
  • PagerDuty
  • Slack
  • Helm
  • Kustomize

The 3am problem

A GPU dies mid-run.
Now what, exactly?

One uncorrectable GPU memory error can kill hours or days of training work. Today that means a human paging through kernel logs and DCGM dashboards, then guessing whether to reset the device, drain the node, or file an RMA — under time pressure, without a record.

DetectionScattered across three tools

XIDs live in kernel logs, ECC counters live in DCGM, and node death shows up somewhere else entirely. Teams end up correlating them by hand, across tools.

ResponseTribal, and only partly written down

The right answer for XID 64 is not the right answer for XID 94. That knowledge sits in a few engineers' heads and a stale runbook.

AutomationToo dangerous to switch on

Scripts that reboot nodes on a metric threshold are exactly how one bad signal takes out a fleet. So most teams never trust them enough to turn them on.

What recovery gave back

“Resolved” is not
a capacity number.

An incident closing tells you the workflow finished. It does not tell you how much accelerator capacity came back, how long it took, or how often the fleet healed without waking anybody. KubeNeuron measures all three from its own incident store — exact, not sampled, and reproducible from a database snapshot.

366.5GPU-hours recovered, of 412.8 degraded — 88.8%

27 of 31recoveries that finished without a human — 87.1%

12m22smedian time to resolution (p50, n=31; p90 52m0s)

The report these figures come from
kubeneuronctl report --since 30d
degraded GPU-hours     412.8
recovered GPU-hours    366.5       88.8% of degraded
incidents recovered    31 of 37    83.8%
  without a human      27 of 31    87.1% of recovered
MTTR (resolved, n=31)  p50 12m22s  p90 52m0s
  • Recovered means resolved. An incident parked for a human keeps accruing degraded time until somebody closes it. Nothing is credited to automation that automation did not finish.
  • Degraded, not lost. A degraded GPU may still have been serving. The number is honest about what it counts.
  • Dry-run recovers nothing. A pilot still gets the numbers — reported separately, labelled as the simulation it is, so the headline figure cannot flatter a fleet nothing has touched.

Why not just…

Runbooks are safe but slow.
Scripts are fast but scary.

Runbook and a pager

How most GPU fleets run today

  • A human judges every step
  • Nothing destructive happens by surprise

but…

  • Minutes to hours before anyone acts
  • The right answer lives in a few heads
  • No record of what was done, or why
  • Degraded GPUs keep taking jobs meanwhile

Threshold script

The cron job nobody dares to enable

  • Reacts in seconds
  • Encodes one engineer's rule once

but…

  • One bad signal can take out a fleet
  • Same reaction to XID 94 and XID 64
  • Shell in config: a trust surface on every node
  • No approval, no identity, no audit

Human judgment + automation that earns it. Pick both.

Detection is automatic. Anything destructive runs a ladder you wrote, behind gates you configured, with a named approver on the record — and dry-run shows you the decision before you let it act.

No shell in config. No surprise reboots. No unexplained decisions.

  • Seconds to detect, with a policy-chosen reaction per problem class
  • Destructive rungs held for a verified human identity, bound to the exact plan shown
  • Real execution only inside a node selector you named, never fleet-wide
  • Unknown state halts the step; every refusal is counted
  • Dry-run by default: trust the policy before it acts

Get started

Four ways in. All of them free.

( Free )

Install in one command

CRDs, operator, TLS bootstrap, and a starter config. Dry-run by default.

Install KubeNeuron
( No cluster )

Run the local demo

The real controller against a simulated fleet, on your laptop, in a minute.

Start the demo
( Live )

Open the console

Click through incidents, the fleet map, and playbooks captured from a real EKS run.

Open the console
( Design partners )

Shape it around your fleet

Three slots for teams running production GPU fleets. Direct roadmap influence, direct access.

Read the offer

Latest release

Shipped, signed, and written down.

Every release carries signed multi-arch images, an SBOM, checksums, a single-file install manifest, and a changelog entry that says what was proven and what was not.

Latest release on GitHub · Full changelog

Build the safe path first

Run a calmer GPU fleet.

Read the source, run the dry-run ladder, and tell us where the policy model breaks.