k3s-io/k3s · critical

%s alarm must be disarmed manually

Error message

%s alarm must be disarmed manually

What it means

clearAlarms only auto-disarms NOSPACE alarms. Any other alarm type on the local member - in practice CORRUPT (backend corruption, typically after disk-full or hardware faults) - returns this error and requires manual intervention; k3s deliberately refuses to auto-clear it because the underlying damage must be fixed first.

Source

Thrown at pkg/etcd/etcd.go:1438

	}

	alarmList, err := e.client.AlarmList(ctx)
	if err != nil {
		return fmt.Errorf("etcd alarm list failed: %v", err)
	}

	for _, alarm := range alarmList.Alarms {
		if alarm.MemberID != memberID {
			// ignore alarms on other cluster members, they should manage their own problems
			continue
		}
		if alarm.Alarm == etcdserverpb.AlarmType_NOSPACE {
			if _, err := e.client.AlarmDisarm(ctx, &clientv3.AlarmMember{MemberID: alarm.MemberID, Alarm: alarm.Alarm}); err != nil {
				return fmt.Errorf("%s disarm failed: %v", alarm.Alarm, err)
			}
			logrus.Infof("%s disarmed successfully", alarm.Alarm)
		} else {
			return fmt.Errorf("%s alarm must be disarmed manually", alarm.Alarm)
		}
	}
	return nil
}

// status returns status using the first etcd endpoint.
func (e *ETCD) status(ctx context.Context) (*clientv3.StatusResponse, error) {
	if e.client == nil {
		return nil, errors.New("etcd client was nil")
	}

	ctx, cancel := context.WithTimeout(ctx, statusTimeout)
	defer cancel()

	endpoints := getEndpoints(e.config)
	return e.client.Status(ctx, endpoints[0])
}

View on GitHub (pinned to 6ba341e396)

Solutions

1. Check etcd logs for the corruption event and test the disk (smartctl/dmesg) - do not disarm over failing hardware.
2. Rebuild the member from known-good data: restore from a snapshot with `k3s server --cluster-reset --cluster-reset-restore-path=<snapshot.db>`, or in a multi-member cluster remove and re-add the broken member so it re-joins fresh.
3. Only after the member is healthy, clear the alarm with `etcdctl alarm disarm` (via the k3s etcd client certs) - if it recurs, the restore source or disk is still bad.

Example fix

# before (auto-disarm refuses)
k3s server   # -> CORRUPT alarm must be disarmed manually

# after (rebuild from snapshot, alarm clears with the fresh backend)
k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-1723680000-0.db

// Pre-check alarm types; only NOSPACE is safe to auto-handle:
alarms, err := cli.AlarmList(ctx)
if err != nil { log.Fatal(err) }
for _, a := range alarms.Alarms {
    if a.MemberID != localID { continue }
    if a.Alarm != etcdserverpb.AlarmType_NOSPACE {
        log.Fatalf("alarm %v on local member is not auto-clearable - rebuild member from snapshot or remove/re-add it", a.Alarm)
    }
}

When it happens

Trigger: AlarmList returns an alarm whose type is not NOSPACE for this memberID (pkg/etcd/etcd.go:1436-1439) - classically AlarmType_CORRUPT raised by etcd after detecting an inconsistent/corrupted backend (bloom filter/hash mismatch).

Common situations: etcd data corruption after a disk-full event, hard power loss, or failing storage hardware; member's db left inconsistent after a crashed defrag.

Related errors

• etcd alarm list failed: %v
• %s disarm failed: %v
• etcd member has status errors: %s
• failed to read consistent index
• no snapshots given for removal

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/dec910c66477f7b1. Report an issue: GitHub.