{"record":{"id":"b13c4d7837b5e94a","repo":"k3s-io/k3s","slug":"etcd-member-has-status-errors-s","errorCode":null,"errorMessage":"etcd member has status errors: %s","messagePattern":"etcd member has status errors: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/etcd/etcd.go","lineNumber":1313,"sourceCode":"\t// See if it's time to evict yet\n\tif now.Sub(progress.LastProgress.Time) > learnerMaxStallTime {\n\t\tif _, err := e.client.MemberRemove(ctx, member.ID); err != nil {\n\t\t\treturn err\n\t\t}\n\t\tlogrus.Warnf(\"Removed learner %s from etcd cluster\", member.Name)\n\t\treturn nil\n\t}\n\n\treturn e.setLearnerProgress(ctx, progress)\n}\n\nfunc (e *ETCD) getETCDStatus(ctx context.Context, url string) (*clientv3.StatusResponse, error) {\n\tresp, err := e.client.Status(ctx, url)\n\tif err != nil {\n\t\treturn resp, errors.WithMessage(err, \"failed to check etcd member status\")\n\t}\n\tif len(resp.Errors) != 0 {\n\t\treturn resp, errors.New(\"etcd member has status errors: \" + strings.Join(resp.Errors, \",\"))\n\t}\n\treturn resp, nil\n}\n\nfunc (e *ETCD) setEtcdStatusCondition(node *v1.Node, memberName string, memberStatus MemberStatus, message string) error {\n\tvar newCondition v1.NodeCondition\n\tswitch memberStatus {\n\tcase StatusLearner:\n\t\tnewCondition = v1.NodeCondition{\n\t\t\tType:    etcdStatusType,\n\t\t\tStatus:  \"False\",\n\t\t\tReason:  \"MemberIsLearner\",\n\t\t\tMessage: \"Node has not been promoted to voting member of the etcd cluster\",\n\t\t}\n\tcase StatusVoter:\n\t\tnewCondition = v1.NodeCondition{\n\t\t\tType:    etcdStatusType,\n\t\t\tStatus:  \"True\",","sourceCodeStart":1295,"sourceCodeEnd":1331,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/etcd.go#L1295-L1331","documentation":"getETCDStatus calls the etcd client Status API; a successful RPC can still carry resp.Errors, an etcd-reported list of member alarms such as NOSPACE (backend quota exceeded) or CORRUPT (backend corruption detected). When the list is non-empty the member is reachable but operationally broken, so the error string embeds the joined alarm strings.","triggerScenarios":"Calling getETCDStatus (used during join/remove and status reconciliation) on a member whose etcd has active alarms: db size over --quota-backend-bytes (default 2GiB) raising NOSPACE, an unclean shutdown triggering CORRUPT, or a failed defrag leaving an alarm armed.","commonSituations":"Large clusters that never compact/defrag until the 2GiB quota trips; nodes with full disks; power loss or kernel panic corrupting the bbolt backend; monitoring that only checks endpoint health and misses alarms.","solutions":["Inspect the alarm strings in the error itself (e.g. 'NOSPACE - Quota backend buddy is exceeded') and run etcdctl alarm list on the member.","For NOSPACE: compact and defrag the backend, then raise --etcd-quota-backend-bytes / etcdctl alarm disarm.","For CORRUPT: the member data is untrusted - remove the member from the cluster and rejoin it so it receives a fresh copy of the db, or restore the whole cluster from a snapshot with --cluster-reset.","Add monitoring on etcd_disk_backend_bytes / mvcc db size so quota is never silently reached."],"exampleFix":"# before: member reports NOSPACE and reconciliation fails\nETCDCTL_API=3 etcdctl alarm list\n# after: reclaim space and clear the alarm\nETCDCTL_API=3 etcdctl compact $(etcdctl endpoint status -w json | jq .[0].Status.header.revision)\nETCDCTL_API=3 etcdctl defrag\nETCDCTL_API=3 etcdctl alarm disarm","handlingStrategy":"validation","validationCode":"// before promoting/using a member, assert it has no active alarms\nresp, err := e.client.Status(ctx, memberURL)\nif err != nil {\n    return err\n}\nif len(resp.Errors) != 0 {\n    return fmt.Errorf(\"member %s has alarms: %v\", memberURL, resp.Errors)\n}","typeGuard":null,"tryCatchPattern":"resp, err := e.getETCDStatus(ctx, url)\nif err != nil {\n    if strings.Contains(err.Error(), \"etcd member has status errors\") {\n        // inspect alarm list; NOSPACE -> compact/defrag/disarm, CORRUPT -> rejoin member\n    }\n    return err\n}","preventionTips":["Alert on etcd mvcc_db_total_size_in_bytes approaching quota-backend-bytes.","Schedule compaction/defrag maintenance windows for large clusters.","Treat CORRUPT as data loss: remove and re-add the member rather than restarting it."],"tags":["etcd","storage","alarm","corruption"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}