{"record":{"id":"4b3f0a352265ef86","repo":"weaviate/weaviate","slug":"shard-q-is-not-in-reindexed-state-and-has-no-star","errorCode":null,"errorMessage":"shard %q is not in reindexed state and has no started sentinel — no in-flight migration on disk","messagePattern":"shard %q is not in reindexed state and has no started sentinel — no in-flight migration on disk","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/inverted_reindex_task_generic.go","lineNumber":525,"sourceCode":"\t})\n\n\trt, err := t.newReindexTrackerGuarded(concreteShard)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"creating reindex tracker: %w\", err)\n\t}\n\n\t// MUST stay ahead of the iteration-resume ladder below: merged implies\n\t// the iteration completed, so on a torn sentinel state (IsMerged &&\n\t// !IsReindexed) resuming would re-run the iteration against an\n\t// already-merged migration. Both callers handle merged downstream.\n\tif rt.IsMerged() {\n\t\treturn &dtmPhaseEntry{shard: concreteShard, logger: logger, rt: rt}, nil\n\t}\n\n\tif !rt.IsReindexed() {\n\t\tif !rt.IsStarted() {\n\t\t\t// Shouldn't happen via OnGroupCompleted (units are node-assigned).\n\t\t\treturn nil, fmt.Errorf(\"shard %q is not in reindexed state and has no started sentinel — no in-flight migration on disk\", concreteShard.Name())\n\t\t}\n\t\tlogger.Info(method + \": state not yet reindexed on disk; resuming iteration\")\n\t\tif err := t.RunReindexOnlyOnShard(ctx, shard); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"resume iteration: %w\", err)\n\t\t}\n\t\trt, err = t.newReindexTrackerGuarded(concreteShard)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"creating reindex tracker after iteration resume: %w\", err)\n\t\t}\n\t\tif !rt.IsReindexed() {\n\t\t\treturn nil, fmt.Errorf(\"shard %q: iteration resume returned but IsReindexed still false\", concreteShard.Name())\n\t\t}\n\t}\n\n\treturn &dtmPhaseEntry{shard: concreteShard, logger: logger, rt: rt}, nil\n}\n\n// RunSwapOnShard runs the swap+tidy+OnMigrationComplete phase.","sourceCodeStart":507,"sourceCodeEnd":543,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/inverted_reindex_task_generic.go#L507-L543","documentation":"enterDTMPhase found a reindex tracker whose on-disk sentinel state is neither reindexed nor started: the shard has a migration tracker directory but no migration is actually in flight. Because OnGroupCompleted only schedules units that are node-assigned and terminal, a shard must either be merged, reindexed, or at least past the started sentinel; anything else is an inconsistent on-disk state, so enterDTMPhase refuses to proceed rather than silently re-indexing or marking a migration complete.","triggerScenarios":"RunPrepareOnShard or RunSwapOnShard is entered, rt.IsMerged() is false, rt.IsReindexed() is false, and rt.IsStarted() is false — i.e. the tracker file/migration dir exists but no started.mig sentinel was ever written, typically because the tracker directory was partially created or the sentinel files were deleted/manually cleaned while the migration dir remained.","commonSituations":"An operator manually removed sentinel .mig files (e.g. during cleanup of a stuck migration) but left the tracker directory; a crash between tracker MkdirAll and the first sentinel write; restoring a shard from a partial backup that included the empty migration dir; disk corruption losing sentinel files.","solutions":["Inspect the shard's migration directory and confirm which .mig sentinel files exist; determine which migration this tracker belongs to","If no migration should be in flight (schema is already consistent with the buckets), remove the stale migration tracker directory for that shard so a clean tracker can be created","If a migration should be in flight, restart the node so OnAfterLsmInit/rehydrate can rebuild the tracker state, or re-trigger the schema migration from the cluster level","If this reproduces after backup/restore, check that the restore tool copies or correctly omits the migration directory as a whole, not sentinel files selectively"],"exampleFix":"// before: manually deleting sentinels leaves an inconsistent state\nrm shard_data/reindex*/started.mig\n// after: remove the whole stale migration tracker directory so a fresh one is created\nrm -rf shard_data/<migration-dir>/\n// then restart weaviate and re-trigger the migration","handlingStrategy":"validation","validationCode":"// Go / shell: before relying on the migration, validate the on-disk sentinel state\n// ls the migration dir and require at least one of started.mig, reindexed.mig, merged.mig\nout, err := exec.Command(\"sh\", \"-c\",\n    \"ls \"+shardDir+\"/migration-dir/*.mig 2>/dev/null | wc -l\").Output()\nif err == nil && strings.TrimSpace(string(out)) == \"0\" {\n    return fmt.Errorf(\"stale/empty migration tracker in %s — remove the tracker dir and re-trigger\", shardDir)\n}","typeGuard":"// Go: guard tracker state before phase entry\nfunc migrationInFlight(rt reindexTracker) bool {\n    return rt.IsStarted() || rt.IsReindexed() || rt.IsMerged()\n}\n// call: if !migrationInFlight(rt) { cleanupStaleTracker(shard); return }","tryCatchPattern":null,"preventionTips":["Never delete individual .mig sentinel files by hand; remove the entire migration tracker directory if a reset is needed","Include the migration directory consistently in backup/restore tooling (all-or-nothing, not selective files)","After a crash, restart the node so the rehydrate path rebuilds tracker state instead of cleaning files manually","Audit any automation that prunes files under the shard LSM directory for migration-dir exclusions"],"tags":["go","migration","state-corruption","lsm","weaviate"],"backgroundTag":"stale-migration-state","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}