{"record":{"id":"f0c17b456e7476ba","repo":"weaviate/weaviate","slug":"recovery-rename-for-q-main-missing-backup-exist","errorCode":null,"errorMessage":"recovery rename for %q: main missing, backup exists, but ingest dir missing — unrecoverable","messagePattern":"recovery rename for %q: main missing, backup exists, but ingest dir missing — unrecoverable","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"adapters/repos/db/inverted_reindex_task_generic.go","lineNumber":2016,"sourceCode":"\t\tingestExists := dirExists(ingestDir)\n\n\t\tswitch {\n\t\tcase mainExists && !backupExists:\n\t\t\t// Pre-rename state (either swap not started, or post-refactor\n\t\t\t// happy-path deferred-rename). Do the full rename pair.\n\t\t\tif !ingestExists {\n\t\t\t\treturn fmt.Errorf(\"recovery rename for %q: main exists, no backup, but ingest dir missing — unrecoverable\", propName)\n\t\t\t}\n\t\t\tif err := os.Rename(mainDir, backupDir); err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery rename main->backup for %q: %w\", propName, err)\n\t\t\t}\n\t\t\tif err := os.Rename(ingestDir, mainDir); err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery rename ingest->main for %q: %w\", propName, err)\n\t\t\t}\n\t\tcase !mainExists && backupExists:\n\t\t\t// Halfway: main was renamed to backup but ingest not yet to main.\n\t\t\tif !ingestExists {\n\t\t\t\treturn fmt.Errorf(\"recovery rename for %q: main missing, backup exists, but ingest dir missing — unrecoverable\", propName)\n\t\t\t}\n\t\t\tif err := os.Rename(ingestDir, mainDir); err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery rename ingest->main for %q: %w\", propName, err)\n\t\t\t}\n\t\tcase mainExists && backupExists:\n\t\t\t// Both exist — ingest was already renamed to main on a prior\n\t\t\t// recovery pass. Idempotent no-op.\n\t\tdefault:\n\t\t\treturn fmt.Errorf(\"unexpected disk state for prop %q: main=%v backup=%v ingest=%v\",\n\t\t\t\tpropName, mainExists, backupExists, ingestExists)\n\t\t}\n\n\t\t// markSwappedProp creates with O_EXCL; a mid-FINALIZING restart may\n\t\t// have already set the sentinel.\n\t\tif !rt.IsSwappedProp(propName) {\n\t\t\tif err := rt.markSwappedProp(propName); err != nil {\n\t\t\t\treturn fmt.Errorf(\"marking swapped prop %q: %w\", propName, err)\n\t\t\t}","sourceCodeStart":1998,"sourceCodeEnd":2034,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/inverted_reindex_task_generic.go#L1998-L2034","documentation":"This unrecoverable-state error is thrown when crash recovery finds a property's bucket directories in an impossible combination: the canonical main bucket directory is missing and a backup directory exists (proof that the first half of the rename pair ran), but the ingest directory that should be renamed to main is absent. There is no directory left to restore or promote — the newly built data cannot be located — so recovery aborts instead of guessing.","triggerScenarios":"RunSwapOnShard -> recoverRuntimeSwapBuckets evaluates case !mainExists && backupExists for a property and dirExists(ingestDir) returns false, i.e. someone or something deleted the ingest_<gen> directory (or it was never flushed/created on this volume) between the main->backup rename and recovery.","commonSituations":"Manual cleanup of 'stray' ingest_* directories by an operator or script that did not know they were recovery-critical; data loss after an unclean shutdown where the ingest dir's creation was not durably synced; restoring only part of a shard directory from a snapshot; copying shards between nodes while a swap was mid-flight.","solutions":["Inspect the shard's lsm-contents directory to confirm main_<...>, backup_<...> and ingest_<...> names; if the ingest dir was accidentally moved/renamed, restore it to its expected ingest_<gen> name and restart so recovery can complete.","Restore the affected shard from the most recent backup/snapshot taken before the interrupted swap — the new inverted index data is lost and must be rebuilt.","Trigger a reindex/migration from scratch for the affected property (drop and re-add the index config, or re-run the reindex task) so fresh ingest buckets are built.","Audit any cleanup scripts or operator runbooks so ingest_* directories inside lsm-contents are never deleted outside of Weaviate's own FinalizeCompletedMigrations/tidy step."],"exampleFix":"// before: operator cleanup deletes 'leftover' dirs\nrm -rf lsm-contents/ingest_property_text   // shard can no longer recover\n\n// after: never delete ingest_* manually; let recovery finish, then let\n// FinalizeCompletedMigrations/tidyBackupBuckets clean up backup_* dirs","handlingStrategy":"validation","validationCode":"// before restarting a node with interrupted swaps, confirm all three dirs are present per pending prop\nfunc validateSwapDirs(lsmPath string, strategy interface{ SourceBucketName(string) string }, ingestName, backupName string) error {\n\tfor _, d := range []string{strategy.SourceBucketName(\"prop\"), ingestName, backupName} {\n\t\t// main OR (backup AND ingest) must exist; log what's missing before any restart\n\t\tif _, err := os.Stat(filepath.Join(lsmPath, d)); err != nil {\n\t\t\treturn fmt.Errorf(\"expected dir %s missing in %s: %w\", d, lsmPath, err)\n\t\t}\n\t}\n\treturn nil\n}","typeGuard":"func isRecoverableSwapState(mainExists, backupExists, ingestExists bool) bool {\n\t// main present (full pair) or halfway (backup+ingest) are recoverable;\n\t// backup without ingest is not.\n\treturn (mainExists && !backupExists && ingestExists) || (!mainExists && backupExists && ingestExists)\n}","tryCatchPattern":"if err := task.RunSwapOnShard(ctx, shard); err != nil {\n\tif strings.Contains(err.Error(), \"unrecoverable\") {\n\t\t// halt — data must be restored from backup; never attempt manual dir surgery\n\t\tlogger.Errorf(\"shard swap unrecoverable, restore from snapshot: %v\", err)\n\t\treturn errShutdownForRestore\n\t}\n\treturn err\n}","preventionTips":["Never delete ingest_* directories inside lsm-contents manually — they are recovery-critical until FinalizeCompletedMigrations runs","Restore shards from snapshots atomically (whole shard directory), never partially","Use durable shutdown (SIGTERM handling) so rename sequences are not interrupted mid-pair","Run fsck/smart checks if directories disappear without human action"],"tags":["filesystem","data-loss","crash-recovery","lsmkv","unrecoverable"],"backgroundTag":"missing-directory-unrecoverable-state","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}