weaviate/weaviate · error

shard %q is not in reindexed state and has no started sentin

Error message

shard %q is not in reindexed state and has no started sentinel — no in-flight migration on disk

What it means

enterDTMPhase found a reindex tracker whose on-disk sentinel state is neither reindexed nor started: the shard has a migration tracker directory but no migration is actually in flight. Because OnGroupCompleted only schedules units that are node-assigned and terminal, a shard must either be merged, reindexed, or at least past the started sentinel; anything else is an inconsistent on-disk state, so enterDTMPhase refuses to proceed rather than silently re-indexing or marking a migration complete.

Source

Thrown at adapters/repos/db/inverted_reindex_task_generic.go:525

	})

	rt, err := t.newReindexTrackerGuarded(concreteShard)
	if err != nil {
		return nil, fmt.Errorf("creating reindex tracker: %w", err)
	}

	// MUST stay ahead of the iteration-resume ladder below: merged implies
	// the iteration completed, so on a torn sentinel state (IsMerged &&
	// !IsReindexed) resuming would re-run the iteration against an
	// already-merged migration. Both callers handle merged downstream.
	if rt.IsMerged() {
		return &dtmPhaseEntry{shard: concreteShard, logger: logger, rt: rt}, nil
	}

	if !rt.IsReindexed() {
		if !rt.IsStarted() {
			// Shouldn't happen via OnGroupCompleted (units are node-assigned).
			return nil, fmt.Errorf("shard %q is not in reindexed state and has no started sentinel — no in-flight migration on disk", concreteShard.Name())
		}
		logger.Info(method + ": state not yet reindexed on disk; resuming iteration")
		if err := t.RunReindexOnlyOnShard(ctx, shard); err != nil {
			return nil, fmt.Errorf("resume iteration: %w", err)
		}
		rt, err = t.newReindexTrackerGuarded(concreteShard)
		if err != nil {
			return nil, fmt.Errorf("creating reindex tracker after iteration resume: %w", err)
		}
		if !rt.IsReindexed() {
			return nil, fmt.Errorf("shard %q: iteration resume returned but IsReindexed still false", concreteShard.Name())
		}
	}

	return &dtmPhaseEntry{shard: concreteShard, logger: logger, rt: rt}, nil
}

// RunSwapOnShard runs the swap+tidy+OnMigrationComplete phase.

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Inspect the shard's migration directory and confirm which .mig sentinel files exist; determine which migration this tracker belongs to
2. If no migration should be in flight (schema is already consistent with the buckets), remove the stale migration tracker directory for that shard so a clean tracker can be created
3. If a migration should be in flight, restart the node so OnAfterLsmInit/rehydrate can rebuild the tracker state, or re-trigger the schema migration from the cluster level
4. If this reproduces after backup/restore, check that the restore tool copies or correctly omits the migration directory as a whole, not sentinel files selectively

Example fix

// before: manually deleting sentinels leaves an inconsistent state
rm shard_data/reindex*/started.mig
// after: remove the whole stale migration tracker directory so a fresh one is created
rm -rf shard_data/<migration-dir>/
// then restart weaviate and re-trigger the migration

// Go / shell: before relying on the migration, validate the on-disk sentinel state
// ls the migration dir and require at least one of started.mig, reindexed.mig, merged.mig
out, err := exec.Command("sh", "-c",
    "ls "+shardDir+"/migration-dir/*.mig 2>/dev/null | wc -l").Output()
if err == nil && strings.TrimSpace(string(out)) == "0" {
    return fmt.Errorf("stale/empty migration tracker in %s — remove the tracker dir and re-trigger", shardDir)
}

// Go: guard tracker state before phase entry
func migrationInFlight(rt reindexTracker) bool {
    return rt.IsStarted() || rt.IsReindexed() || rt.IsMerged()
}
// call: if !migrationInFlight(rt) { cleanupStaleTracker(shard); return }

When it happens

Trigger: RunPrepareOnShard or RunSwapOnShard is entered, rt.IsMerged() is false, rt.IsReindexed() is false, and rt.IsStarted() is false — i.e. the tracker file/migration dir exists but no started.mig sentinel was ever written, typically because the tracker directory was partially created or the sentinel files were deleted/manually cleaned while the migration dir remained.

Common situations: An operator manually removed sentinel .mig files (e.g. during cleanup of a stuck migration) but left the tracker directory; a crash between tracker MkdirAll and the first sentinel write; restoring a shard from a partial backup that included the empty migration dir; disk corruption losing sentinel files.

Related errors

• creating reindex tracker: %w
• resume iteration: %w
• creating reindex tracker after iteration resume: %w
• shard %q: iteration resume returned but IsReindexed still fa
• create 1 named vector config: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/4b3f0a352265ef86. Report an issue: GitHub.