k3s-io/k3s · critical

etcd member has status errors: %s

Error message

etcd member has status errors: %s

What it means

getETCDStatus calls the etcd client Status API; a successful RPC can still carry resp.Errors, an etcd-reported list of member alarms such as NOSPACE (backend quota exceeded) or CORRUPT (backend corruption detected). When the list is non-empty the member is reachable but operationally broken, so the error string embeds the joined alarm strings.

Source

Thrown at pkg/etcd/etcd.go:1313

	// See if it's time to evict yet
	if now.Sub(progress.LastProgress.Time) > learnerMaxStallTime {
		if _, err := e.client.MemberRemove(ctx, member.ID); err != nil {
			return err
		}
		logrus.Warnf("Removed learner %s from etcd cluster", member.Name)
		return nil
	}

	return e.setLearnerProgress(ctx, progress)
}

func (e *ETCD) getETCDStatus(ctx context.Context, url string) (*clientv3.StatusResponse, error) {
	resp, err := e.client.Status(ctx, url)
	if err != nil {
		return resp, errors.WithMessage(err, "failed to check etcd member status")
	}
	if len(resp.Errors) != 0 {
		return resp, errors.New("etcd member has status errors: " + strings.Join(resp.Errors, ","))
	}
	return resp, nil
}

func (e *ETCD) setEtcdStatusCondition(node *v1.Node, memberName string, memberStatus MemberStatus, message string) error {
	var newCondition v1.NodeCondition
	switch memberStatus {
	case StatusLearner:
		newCondition = v1.NodeCondition{
			Type:    etcdStatusType,
			Status:  "False",
			Reason:  "MemberIsLearner",
			Message: "Node has not been promoted to voting member of the etcd cluster",
		}
	case StatusVoter:
		newCondition = v1.NodeCondition{
			Type:    etcdStatusType,
			Status:  "True",

View on GitHub (pinned to 6ba341e396)

Solutions

1. Inspect the alarm strings in the error itself (e.g. 'NOSPACE - Quota backend buddy is exceeded') and run etcdctl alarm list on the member.
2. For NOSPACE: compact and defrag the backend, then raise --etcd-quota-backend-bytes / etcdctl alarm disarm.
3. For CORRUPT: the member data is untrusted - remove the member from the cluster and rejoin it so it receives a fresh copy of the db, or restore the whole cluster from a snapshot with --cluster-reset.
4. Add monitoring on etcd_disk_backend_bytes / mvcc db size so quota is never silently reached.

Example fix

# before: member reports NOSPACE and reconciliation fails
ETCDCTL_API=3 etcdctl alarm list
# after: reclaim space and clear the alarm
ETCDCTL_API=3 etcdctl compact $(etcdctl endpoint status -w json | jq .[0].Status.header.revision)
ETCDCTL_API=3 etcdctl defrag
ETCDCTL_API=3 etcdctl alarm disarm

// before promoting/using a member, assert it has no active alarms
resp, err := e.client.Status(ctx, memberURL)
if err != nil {
    return err
}
if len(resp.Errors) != 0 {
    return fmt.Errorf("member %s has alarms: %v", memberURL, resp.Errors)
}

resp, err := e.getETCDStatus(ctx, url)
if err != nil {
    if strings.Contains(err.Error(), "etcd member has status errors") {
        // inspect alarm list; NOSPACE -> compact/defrag/disarm, CORRUPT -> rejoin member
    }
    return err
}

When it happens

Trigger: Calling getETCDStatus (used during join/remove and status reconciliation) on a member whose etcd has active alarms: db size over --quota-backend-bytes (default 2GiB) raising NOSPACE, an unclean shutdown triggering CORRUPT, or a failed defrag leaving an alarm armed.

Common situations: Large clusters that never compact/defrag until the 2GiB quota trips; nodes with full disks; power loss or kernel panic corrupting the bbolt backend; monitoring that only checks endpoint health and misses alarms.

Related errors

• failed to read consistent index
• no bootstrap data found
• %s alarm must be disarmed manually
• no snapshots given for removal
• invalid output format:

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/b13c4d7837b5e94a. Report an issue: GitHub.