SolProbe runs an edge sidecar on every GPU node, streams signals to a central diagnoser, and writes attestations to Solana — so when a training run fails, you have proof of who, what, and when.
| Node | GPU | Util | Temp |
|---|---|---|---|
| node-a01 | H100 ×8 | 92% | 68°C |
| node-a02 | H100 ×8 | 88% | 71°C |
| node-a03 | H100 ×8 | 64% | 82°C |
| node-a04 | H100 ×8 | 91% | 69°C |
| node-a05 | H100 ×8 | 89% | 70°C |
From DCGM probes on the metal to Solana attestations — every transition is auditable.
Lightweight sidecar on each node samples DCGM, NVML, and DiLoCo state at 1 Hz — XID, ECC, PCIe replays, MFU, gradient norms.
Central service correlates per-node alerts across the cluster, runs an LLM-grounded diagnosis with evidence chain, and proposes a recovery action.
Each diagnosis is hashed, signed, and committed to Solana. Validators stake SOL on health; bad attestations get slashed.
Generic monitoring tools don't know what a straggler is. SolProbe does.
Detects stragglers via worker speed ratio, pseudo-gradient drift, and inner/outer-step desync — not just node liveness.
Edge sidecar fires the alert. Central correlates within 200ms. You see it before NCCL does.
Every alert ships with reasoning, an evidence chain, and a suggested action — not just a number above threshold.
Every metric and alert type is documented. Run our sidecar or write your own — the wire format is stable.
Each diagnosis lands on Solana. Insurance, SLAs, and reproducibility all become provable, not promised.
Next.js app, fully typed, ships with WebSocket streaming, alert taxonomy, and a per-node detail view out of the box.
A small, exhaustive vocabulary covering every failure mode we've seen take down a 64-GPU run.
Full taxonomy →One container per host. Auto-discovers GPUs, registers with central, starts streaming.
SolProbe writes a compact attestation per diagnosis to Solana. Validators stake SOL on cluster health — false attestations get slashed, accurate ones earn yield.
SolProbe is in private beta. Tell us about your cluster and we'll get you running this week.