{"record":{"id":"dec910c66477f7b1","repo":"k3s-io/k3s","slug":"s-alarm-must-be-disarmed-manually","errorCode":null,"errorMessage":"%s alarm must be disarmed manually","messagePattern":"(.+?) alarm must be disarmed manually","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/etcd/etcd.go","lineNumber":1438,"sourceCode":"\t}\n\n\talarmList, err := e.client.AlarmList(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"etcd alarm list failed: %v\", err)\n\t}\n\n\tfor _, alarm := range alarmList.Alarms {\n\t\tif alarm.MemberID != memberID {\n\t\t\t// ignore alarms on other cluster members, they should manage their own problems\n\t\t\tcontinue\n\t\t}\n\t\tif alarm.Alarm == etcdserverpb.AlarmType_NOSPACE {\n\t\t\tif _, err := e.client.AlarmDisarm(ctx, &clientv3.AlarmMember{MemberID: alarm.MemberID, Alarm: alarm.Alarm}); err != nil {\n\t\t\t\treturn fmt.Errorf(\"%s disarm failed: %v\", alarm.Alarm, err)\n\t\t\t}\n\t\t\tlogrus.Infof(\"%s disarmed successfully\", alarm.Alarm)\n\t\t} else {\n\t\t\treturn fmt.Errorf(\"%s alarm must be disarmed manually\", alarm.Alarm)\n\t\t}\n\t}\n\treturn nil\n}\n\n// status returns status using the first etcd endpoint.\nfunc (e *ETCD) status(ctx context.Context) (*clientv3.StatusResponse, error) {\n\tif e.client == nil {\n\t\treturn nil, errors.New(\"etcd client was nil\")\n\t}\n\n\tctx, cancel := context.WithTimeout(ctx, statusTimeout)\n\tdefer cancel()\n\n\tendpoints := getEndpoints(e.config)\n\treturn e.client.Status(ctx, endpoints[0])\n}\n","sourceCodeStart":1420,"sourceCodeEnd":1456,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/etcd.go#L1420-L1456","documentation":"clearAlarms only auto-disarms NOSPACE alarms. Any other alarm type on the local member - in practice CORRUPT (backend corruption, typically after disk-full or hardware faults) - returns this error and requires manual intervention; k3s deliberately refuses to auto-clear it because the underlying damage must be fixed first.","triggerScenarios":"AlarmList returns an alarm whose type is not NOSPACE for this memberID (pkg/etcd/etcd.go:1436-1439) - classically AlarmType_CORRUPT raised by etcd after detecting an inconsistent/corrupted backend (bloom filter/hash mismatch).","commonSituations":"etcd data corruption after a disk-full event, hard power loss, or failing storage hardware; member's db left inconsistent after a crashed defrag.","solutions":["Check etcd logs for the corruption event and test the disk (smartctl/dmesg) - do not disarm over failing hardware.","Rebuild the member from known-good data: restore from a snapshot with `k3s server --cluster-reset --cluster-reset-restore-path=<snapshot.db>`, or in a multi-member cluster remove and re-add the broken member so it re-joins fresh.","Only after the member is healthy, clear the alarm with `etcdctl alarm disarm` (via the k3s etcd client certs) - if it recurs, the restore source or disk is still bad."],"exampleFix":"# before (auto-disarm refuses)\nk3s server   # -> CORRUPT alarm must be disarmed manually\n\n# after (rebuild from snapshot, alarm clears with the fresh backend)\nk3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-1723680000-0.db","handlingStrategy":"validation","validationCode":"// Pre-check alarm types; only NOSPACE is safe to auto-handle:\nalarms, err := cli.AlarmList(ctx)\nif err != nil { log.Fatal(err) }\nfor _, a := range alarms.Alarms {\n    if a.MemberID != localID { continue }\n    if a.Alarm != etcdserverpb.AlarmType_NOSPACE {\n        log.Fatalf(\"alarm %v on local member is not auto-clearable - rebuild member from snapshot or remove/re-add it\", a.Alarm)\n    }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat CORRUPT alarms as data-loss events: stop, snapshot what remains, restore from a known-good snapshot.","Use UPS-backed nodes and healthy disks for etcd members to avoid corruption in the first place.","Take regular etcd snapshots (`k3s etcd-snapshot` cron) so recovery never depends on a corrupt backend."],"tags":["etcd","corruption","alarms","disaster-recovery","disk"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}