What causes "failed to iterate over loaded shards: %w" in weaviate/weaviate?

Inspect the wrapped error from ForEachShard for the underlying cause (likely index shutdown or store error).

failed to iterate over loaded shards: %w

Error message

failed to iterate over loaded shards: %w

What it means

updateIndexShards snapshots all currently loaded shards via idx.ForEachShard before reconciling which shards should be loaded vs unloaded. This error means the iteration itself failed — not any individual load/unload — so the migrator cannot build a reliable picture of local shard state and aborts the reconcile rather than risk leaving shards unreconciled.

Source

Thrown at adapters/repos/db/migrator.go:503

	}

	if err := ec.ToError(); err != nil {
		return fmt.Errorf("drop tenant shards %v during update index: %w", toRemove, err)
	}
	return nil
}

func (m *Migrator) updateIndexShards(ctx context.Context, idx *Index,
	incomingSS *sharding.State,
) error {
	requestedShards := incomingSS.AllLocalPhysicalShards()
	existingShards := make(map[string]ShardLike)

	if err := idx.ForEachShard(func(name string, shard ShardLike) error {
		existingShards[name] = shard
		return nil
	}); err != nil {
		return fmt.Errorf("failed to iterate over loaded shards: %w", err)
	}

	// every shard is attempted and every failure reported: a swallowed unload
	// reports a partial reconcile as a success, and stopping at the first failed
	// load leaves every shard behind it unreconciled
	ec := errorcompounder.New()

	// Initialize missing shards and shutdown unneeded ones
	for shardName := range existingShards {
		if !slices.Contains(requestedShards, shardName) {
			ec.AddWrapf(idx.UnloadLocalShard(ctx, shardName),
				"shutdown shard %s during update index", shardName)
		}
	}

	for _, shardName := range requestedShards {
		if _, exists := existingShards[shardName]; !exists {
			ec.AddWrapf(idx.loadLocalShardForReload(ctx, shardName, false /* mustLoad */),

View on GitHub (pinned to 75aa4b6d11)

Solutions

Inspect the wrapped error from ForEachShard for the underlying cause (likely index shutdown or store error).
Check whether the index/collection was concurrently deleted or the node shut down; retry reconciliation if it was transient.
Re-run UpdateIndex once the node is stable; it is safe to re-run.
If it recurs, check disk/store health for the affected index.

Example fix

// before
// retrying blindly in a tight loop
// after
// wait for quiescence, then re-run reconcile
time.Sleep(5 * time.Second)
if err := migrator.UpdateIndex(ctx, class, shardingState); err != nil {
    log.Printf("reconcile retry failed: %v", err)
}

Defensive patterns

Strategy: retry

Try / catch

if err := migrator.UpdateIndex(ctx, class, ss); err != nil {
    if strings.Contains(err.Error(), "failed to iterate over loaded shards") {
        // index likely shutting down; wait and re-run reconcile
        time.Sleep(backoff)
        return migrator.UpdateIndex(ctx, class, ss)
    }
    return err
}

Prevention

Avoid triggering schema reconciliation while the node/index is shutting down
Serialize index drop/shutdown against UpdateIndex calls
Retry reconciliation after transient failures; it is designed to be re-run

When it happens

Trigger: idx.ForEachShard returns a non-nil error, typically because the underlying shard map iteration was interrupted — e.g. the index is being shut down or dropped concurrently, or an internal store error occurred during traversal.

Common situations: Node shutdown/drop racing with a schema UpdateIndex; index closed between GetIndex and ForEachShard; storage backend errors during a rejoin/reconcile after downtime.

Related errors

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/21ac55a1b6609ba0. Report an issue: GitHub.