tombstone file has been detected but --server is empty: back

Error message

tombstone file has been detected but --server is empty: backup and delete ${datadir}/server/db to create a new cluster, or set --server to rejoin the cluster

What it means

A 'tombstone' file in the etcd db dir marks a datastore that was deliberately invalidated (quorum-lost reset or removal from cluster). On startup with etcd enabled, k3s refuses to proceed unless a --server join URL is provided, in which case it backs up and clears the db dir and rejoins.

Source

Thrown at pkg/etcd/etcd.go:685

			registerEndpointsHandlers(ctx, e)
			registerMemberHandlers(ctx, e)
			registerSnapshotHandlers(ctx, e)

			// Re-run informer factory startup after core and leader-elected controllers have started.
			// Additional caches may need to start for the newly added OnChange/OnRemove callbacks.
			if err := start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core); err != nil {
				panic(errors.WithMessage(err, "failed to start wrangler controllers"))
			}
		}
	}

	// Tombstone file checking is unnecessary if we're not running etcd.
	if !e.config.DisableETCD {
		tombstoneFile := filepath.Join(dbDir(e.config), "tombstone")
		if _, err := os.Stat(tombstoneFile); err == nil {
			if e.config.JoinURL == "" {
				return nil, errors.New("tombstone file has been detected but --server is empty: backup and delete ${datadir}/server/db to create a new cluster, or set --server to rejoin the cluster")
			}
			logrus.Infof("tombstone file has been detected, removing ${datadir}/server/db to rejoin the cluster")
			if _, err := backupDirWithRetention(dbDir(e.config), maxBackupRetention); err != nil {
				return nil, err
			}
		}

		if err := e.setName(false); err != nil {
			return nil, err
		}
	}

	return e.handler(handler), nil
}

// setName sets a unique name for this cluster member. The first time this is called,
// or if force is set to true, a new name will be generated and written to disk. The persistent
// name is used on subsequent calls.

View on GitHub (pinned to 6ba341e396)

Solutions

To rejoin: start with --server https://<healthy-server>:6443; k3s logs that the db dir is backed up and removed, then rejoins.
To form/join as new: back up and delete /var/lib/rancher/k3s/server/db manually and start per your topology.
Do not simply delete the tombstone file alone: the db state it guards is invalid for the current cluster.

Example fix

# before: plain restart fails on tombstone
systemctl start k3s
# after: rejoin the existing cluster
k3s server --server https://10.0.0.10:6443

Defensive patterns

Strategy: validation

Validate before calling

// Pre-start: if a tombstone exists, require a join URL in config:
func tombstonePolicy(dataDir, joinURL string) error {
	if _, err := os.Stat(filepath.Join(dataDir, "db", "tombstone")); err == nil && joinURL == "" {
		return errors.New("tombstone present: pass --server <url> to rejoin, or wipe db dir")
	}
	return nil
}

Try / catch

if err := e.Start(ctx, wg, info); err != nil {
	if strings.Contains(err.Error(), "tombstone file has been detected but --server is empty") {
		// operator decision: rejoin via --server, or backup+delete db to start fresh
	}
}

Prevention

After any cluster-reset or member removal, immediately restart affected nodes with --server.
Automate detection: alert on presence of db/tombstone on any managed node.
Never delete only the tombstone file; the underlying db is invalid by design.

When it happens

Trigger: Node previously removed via k3s-server-remove / cluster-reset on peers, or a reset that wrote the tombstone; restarting that node without --server.

Common situations: Replacing quorum-lost clusters; nodes that were members of a reset cluster booting again; stateful VM restored from snapshot after cluster reformation.

Related errors

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/0af9f3c595670050. Report an issue: GitHub.