{"record":{"id":"737575e7ccdcea78","repo":"k3s-io/k3s","slug":"s-disarm-failed-v","errorCode":null,"errorMessage":"%s disarm failed: %v","messagePattern":"(.+?) disarm failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/etcd/etcd.go","lineNumber":1434,"sourceCode":"// Other alarm types are not handled.\nfunc (e *ETCD) clearAlarms(ctx context.Context, memberID uint64) error {\n\tif e.client == nil {\n\t\treturn errors.New(\"etcd client was nil\")\n\t}\n\n\talarmList, err := e.client.AlarmList(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"etcd alarm list failed: %v\", err)\n\t}\n\n\tfor _, alarm := range alarmList.Alarms {\n\t\tif alarm.MemberID != memberID {\n\t\t\t// ignore alarms on other cluster members, they should manage their own problems\n\t\t\tcontinue\n\t\t}\n\t\tif alarm.Alarm == etcdserverpb.AlarmType_NOSPACE {\n\t\t\tif _, err := e.client.AlarmDisarm(ctx, &clientv3.AlarmMember{MemberID: alarm.MemberID, Alarm: alarm.Alarm}); err != nil {\n\t\t\t\treturn fmt.Errorf(\"%s disarm failed: %v\", alarm.Alarm, err)\n\t\t\t}\n\t\t\tlogrus.Infof(\"%s disarmed successfully\", alarm.Alarm)\n\t\t} else {\n\t\t\treturn fmt.Errorf(\"%s alarm must be disarmed manually\", alarm.Alarm)\n\t\t}\n\t}\n\treturn nil\n}\n\n// status returns status using the first etcd endpoint.\nfunc (e *ETCD) status(ctx context.Context) (*clientv3.StatusResponse, error) {\n\tif e.client == nil {\n\t\treturn nil, errors.New(\"etcd client was nil\")\n\t}\n\n\tctx, cancel := context.WithTimeout(ctx, statusTimeout)\n\tdefer cancel()\n","sourceCodeStart":1416,"sourceCodeEnd":1452,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/etcd.go#L1416-L1452","documentation":"When clearAlarms finds a NOSPACE alarm on the local member it attempts AlarmsDisarm to clear it after (per surrounding logic) space has been reclaimed. If the disarm RPC fails, this error reports the alarm type and the clientv3 error. The alarm remains armed and etcd may still reject writes.","triggerScenarios":"e.client.AlarmDisarm(...) returning an error while disarming a NOSPACE alarm on the local member (pkg/etcd/etcd.go:1431-1435): the member went unresponsive mid-call, the backend is still over quota so etcd refuses to disarm, or connection/TLS issues.","commonSituations":"Disk freed but quota-backend-bytes limit still exceeded so disarm is rejected; etcd restarted or crashed between alarm list and disarm; slow disk causing gRPC timeout.","solutions":["Free real space AND check the quota: if --etcd-quota-backend-bytes is set, ensure the backend size is below it (compact/defrag: `etcdctl compact` + `etcdctl defrag`) before retrying.","Disarm manually once the member is healthy: `ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=... --cert=... --key=... alarm disarm`.","If disarm keeps failing, restart the etcd member (restart k3s server) so the alarm sweep re-runs against a fresh connection."],"exampleFix":"# after (manual disarm once healthy)\nETCDCTL_API=3 etcdctl \\\n  --endpoints=https://127.0.0.1:2379 \\\n  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \\\n  --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \\\n  --key=/var/lib/rancher/k3s/server/tls/etcd/client.key \\\n  alarm disarm","handlingStrategy":"retry","validationCode":"// Confirm space was actually reclaimed before disarm: backend size must be under quota:\nst, err := cli.Status(ctx, endpoint)\nif err == nil && quotaBytes > 0 && st.DbSize > quotaBytes {\n    log.Printf(\"backend %d still over quota %d - defrag first\", st.DbSize, quotaBytes)\n}","typeGuard":null,"tryCatchPattern":"// Retry disarm after remediation, give up and surface after N attempts:\nerr := retry(3, 5*time.Second, func() error {\n    _, err := cli.AlarmDisarm(ctx, &clientv3.AlarmMember{MemberID: id, Alarm: etcdserverpb.AlarmType_NOSPACE})\n    return err\n})\nif err != nil { log.Fatalf(\"disarm failed; run etcdctl alarm disarm manually once healthy: %v\", err) }","preventionTips":["Free disk space AND defrag/compact before expecting disarm to succeed.","Automate NOSPACE runbooks: alert at 80% quota, compact+defrag, then disarm.","Keep etcd client certs handy so manual etcdctl intervention is possible."],"tags":["etcd","alarms","nospace","quota","disk"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}