{"record":{"id":"66ed1ea61ee7361d","repo":"k3s-io/k3s","slug":"etcd-alarm-list-failed-v","errorCode":null,"errorMessage":"etcd alarm list failed: %v","messagePattern":"etcd alarm list failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/etcd/etcd.go","lineNumber":1424,"sourceCode":"\tif err := json.NewEncoder(w).Encode(status); err != nil {\n\t\treturn err\n\t}\n\n\t_, err := e.client.Put(ctx, learnerProgressKey, w.String())\n\treturn err\n}\n\n// clearAlarms checks for any NOSPACE alarms on the local etcd member.\n// If found, they are reported and the alarm state is cleared.\n// Other alarm types are not handled.\nfunc (e *ETCD) clearAlarms(ctx context.Context, memberID uint64) error {\n\tif e.client == nil {\n\t\treturn errors.New(\"etcd client was nil\")\n\t}\n\n\talarmList, err := e.client.AlarmList(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"etcd alarm list failed: %v\", err)\n\t}\n\n\tfor _, alarm := range alarmList.Alarms {\n\t\tif alarm.MemberID != memberID {\n\t\t\t// ignore alarms on other cluster members, they should manage their own problems\n\t\t\tcontinue\n\t\t}\n\t\tif alarm.Alarm == etcdserverpb.AlarmType_NOSPACE {\n\t\t\tif _, err := e.client.AlarmDisarm(ctx, &clientv3.AlarmMember{MemberID: alarm.MemberID, Alarm: alarm.Alarm}); err != nil {\n\t\t\t\treturn fmt.Errorf(\"%s disarm failed: %v\", alarm.Alarm, err)\n\t\t\t}\n\t\t\tlogrus.Infof(\"%s disarmed successfully\", alarm.Alarm)\n\t\t} else {\n\t\t\treturn fmt.Errorf(\"%s alarm must be disarmed manually\", alarm.Alarm)\n\t\t}\n\t}\n\treturn nil\n}","sourceCodeStart":1406,"sourceCodeEnd":1442,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/etcd.go#L1406-L1442","documentation":"clearAlarms queries the local etcd member for active alarms (e.g. NOSPACE raised when the backend quota is exceeded). If the AlarmList RPC itself fails - client cannot reach etcd, timeout, member not serving - this error wraps the clientv3 failure. A nil client is a separate error ('etcd client was nil').","triggerScenarios":"e.client.AlarmList(ctx) failing during the post-start alarm sweep (pkg/etcd/etcd.go:1423-1425): local etcd endpoint down or not yet listening, gRPC deadline exceeded, TLS/auth mismatch, or etcd crashed (often itself a symptom of disk-full NOSPACE).","commonSituations":"Disk full on an etcd node making etcd unresponsive; etcd still starting up when the alarm check ran; wrong client cert/ETCDCTL_ENDPOINTS; frequent during the same incident that raised the alarms.","solutions":["Check etcd member health: `k3s etcd-snapshot check --latest` or etcdctl endpoint status; look at the etcd log lines around the failure.","Address the resource problem first - NOSPACE incidents usually mean the disk/quota is exhausted: free space or raise --etcd-quota-backend-bytes, then restart k3s.","Verify disk space (df -h), inodes (df -i) and the data-dir filesystem before retrying startup."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Verify the local etcd endpoint answers before the operation that clears alarms:\nctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\ndefer cancel()\nif _, err := clientv3.New(clientv3.Config{Endpoints: []string{\"https://127.0.0.1:2379\"}, TLS: tlsCfg}).Status(ctx, \"https://127.0.0.1:2379\"); err != nil {\n    log.Fatalf(\"etcd not ready: %v\", err)\n}","typeGuard":null,"tryCatchPattern":"// Retry the alarm list with backoff while etcd is settling:\nvar alarms *clientv3.AlarmResponse\nerr := retry(5, 2*time.Second, func() error {\n    var err error\n    alarms, err = cli.AlarmList(ctx)\n    return err // retry on transient unavailability; abort on auth errors\n})\nif err != nil { /* etcd still down - investigate health, do not loop forever */ }","preventionTips":["Monitor etcd endpoint status and disk capacity continuously (NOSPACE incidents are predictable).","Set --etcd-quota-backend-bytes with headroom and compact/defrag on a schedule.","Give etcd time to become leader-ready after restarts before running maintenance sweeps."],"tags":["etcd","alarms","nospace","disk","health"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}