{"record":{"id":"0af9f3c595670050","repo":"k3s-io/k3s","slug":"tombstone-file-has-been-detected-but-server-is-e","errorCode":null,"errorMessage":"tombstone file has been detected but --server is empty: backup and delete ${datadir}/server/db to create a new cluster, or set --server to rejoin the cluster","messagePattern":"tombstone file has been detected but --server is empty: backup and delete (.+?)/server/db to create a new cluster, or set --server to rejoin the cluster","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/etcd/etcd.go","lineNumber":685,"sourceCode":"\n\t\t\tregisterEndpointsHandlers(ctx, e)\n\t\t\tregisterMemberHandlers(ctx, e)\n\t\t\tregisterSnapshotHandlers(ctx, e)\n\n\t\t\t// Re-run informer factory startup after core and leader-elected controllers have started.\n\t\t\t// Additional caches may need to start for the newly added OnChange/OnRemove callbacks.\n\t\t\tif err := start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core); err != nil {\n\t\t\t\tpanic(errors.WithMessage(err, \"failed to start wrangler controllers\"))\n\t\t\t}\n\t\t}\n\t}\n\n\t// Tombstone file checking is unnecessary if we're not running etcd.\n\tif !e.config.DisableETCD {\n\t\ttombstoneFile := filepath.Join(dbDir(e.config), \"tombstone\")\n\t\tif _, err := os.Stat(tombstoneFile); err == nil {\n\t\t\tif e.config.JoinURL == \"\" {\n\t\t\t\treturn nil, errors.New(\"tombstone file has been detected but --server is empty: backup and delete ${datadir}/server/db to create a new cluster, or set --server to rejoin the cluster\")\n\t\t\t}\n\t\t\tlogrus.Infof(\"tombstone file has been detected, removing ${datadir}/server/db to rejoin the cluster\")\n\t\t\tif _, err := backupDirWithRetention(dbDir(e.config), maxBackupRetention); err != nil {\n\t\t\t\treturn nil, err\n\t\t\t}\n\t\t}\n\n\t\tif err := e.setName(false); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t}\n\n\treturn e.handler(handler), nil\n}\n\n// setName sets a unique name for this cluster member. The first time this is called,\n// or if force is set to true, a new name will be generated and written to disk. The persistent\n// name is used on subsequent calls.","sourceCodeStart":667,"sourceCodeEnd":703,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/etcd.go#L667-L703","documentation":"A 'tombstone' file in the etcd db dir marks a datastore that was deliberately invalidated (quorum-lost reset or removal from cluster). On startup with etcd enabled, k3s refuses to proceed unless a --server join URL is provided, in which case it backs up and clears the db dir and rejoins.","triggerScenarios":"Node previously removed via k3s-server-remove / cluster-reset on peers, or a reset that wrote the tombstone; restarting that node without --server.","commonSituations":"Replacing quorum-lost clusters; nodes that were members of a reset cluster booting again; stateful VM restored from snapshot after cluster reformation.","solutions":["To rejoin: start with --server https://<healthy-server>:6443; k3s logs that the db dir is backed up and removed, then rejoins.","To form/join as new: back up and delete /var/lib/rancher/k3s/server/db manually and start per your topology.","Do not simply delete the tombstone file alone: the db state it guards is invalid for the current cluster."],"exampleFix":"# before: plain restart fails on tombstone\nsystemctl start k3s\n# after: rejoin the existing cluster\nk3s server --server https://10.0.0.10:6443","handlingStrategy":"validation","validationCode":"// Pre-start: if a tombstone exists, require a join URL in config:\nfunc tombstonePolicy(dataDir, joinURL string) error {\n\tif _, err := os.Stat(filepath.Join(dataDir, \"db\", \"tombstone\")); err == nil && joinURL == \"\" {\n\t\treturn errors.New(\"tombstone present: pass --server <url> to rejoin, or wipe db dir\")\n\t}\n\treturn nil\n}","typeGuard":null,"tryCatchPattern":"if err := e.Start(ctx, wg, info); err != nil {\n\tif strings.Contains(err.Error(), \"tombstone file has been detected but --server is empty\") {\n\t\t// operator decision: rejoin via --server, or backup+delete db to start fresh\n\t}\n}","preventionTips":["After any cluster-reset or member removal, immediately restart affected nodes with --server.","Automate detection: alert on presence of db/tombstone on any managed node.","Never delete only the tombstone file; the underlying db is invalid by design."],"tags":["k3s","etcd","tombstone","join","recovery"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}