{"record":{"id":"21ac55a1b6609ba0","repo":"weaviate/weaviate","slug":"failed-to-iterate-over-loaded-shards-w","errorCode":null,"errorMessage":"failed to iterate over loaded shards: %w","messagePattern":"failed to iterate over loaded shards: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/migrator.go","lineNumber":503,"sourceCode":"\t}\n\n\tif err := ec.ToError(); err != nil {\n\t\treturn fmt.Errorf(\"drop tenant shards %v during update index: %w\", toRemove, err)\n\t}\n\treturn nil\n}\n\nfunc (m *Migrator) updateIndexShards(ctx context.Context, idx *Index,\n\tincomingSS *sharding.State,\n) error {\n\trequestedShards := incomingSS.AllLocalPhysicalShards()\n\texistingShards := make(map[string]ShardLike)\n\n\tif err := idx.ForEachShard(func(name string, shard ShardLike) error {\n\t\texistingShards[name] = shard\n\t\treturn nil\n\t}); err != nil {\n\t\treturn fmt.Errorf(\"failed to iterate over loaded shards: %w\", err)\n\t}\n\n\t// every shard is attempted and every failure reported: a swallowed unload\n\t// reports a partial reconcile as a success, and stopping at the first failed\n\t// load leaves every shard behind it unreconciled\n\tec := errorcompounder.New()\n\n\t// Initialize missing shards and shutdown unneeded ones\n\tfor shardName := range existingShards {\n\t\tif !slices.Contains(requestedShards, shardName) {\n\t\t\tec.AddWrapf(idx.UnloadLocalShard(ctx, shardName),\n\t\t\t\t\"shutdown shard %s during update index\", shardName)\n\t\t}\n\t}\n\n\tfor _, shardName := range requestedShards {\n\t\tif _, exists := existingShards[shardName]; !exists {\n\t\t\tec.AddWrapf(idx.loadLocalShardForReload(ctx, shardName, false /* mustLoad */),","sourceCodeStart":485,"sourceCodeEnd":521,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/migrator.go#L485-L521","documentation":"updateIndexShards snapshots all currently loaded shards via idx.ForEachShard before reconciling which shards should be loaded vs unloaded. This error means the iteration itself failed — not any individual load/unload — so the migrator cannot build a reliable picture of local shard state and aborts the reconcile rather than risk leaving shards unreconciled.","triggerScenarios":"idx.ForEachShard returns a non-nil error, typically because the underlying shard map iteration was interrupted — e.g. the index is being shut down or dropped concurrently, or an internal store error occurred during traversal.","commonSituations":"Node shutdown/drop racing with a schema UpdateIndex; index closed between GetIndex and ForEachShard; storage backend errors during a rejoin/reconcile after downtime.","solutions":["Inspect the wrapped error from ForEachShard for the underlying cause (likely index shutdown or store error).","Check whether the index/collection was concurrently deleted or the node shut down; retry reconciliation if it was transient.","Re-run UpdateIndex once the node is stable; it is safe to re-run.","If it recurs, check disk/store health for the affected index."],"exampleFix":"// before\n// retrying blindly in a tight loop\n// after\n// wait for quiescence, then re-run reconcile\ntime.Sleep(5 * time.Second)\nif err := migrator.UpdateIndex(ctx, class, shardingState); err != nil {\n    log.Printf(\"reconcile retry failed: %v\", err)\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := migrator.UpdateIndex(ctx, class, ss); err != nil {\n    if strings.Contains(err.Error(), \"failed to iterate over loaded shards\") {\n        // index likely shutting down; wait and re-run reconcile\n        time.Sleep(backoff)\n        return migrator.UpdateIndex(ctx, class, ss)\n    }\n    return err\n}","preventionTips":["Avoid triggering schema reconciliation while the node/index is shutting down","Serialize index drop/shutdown against UpdateIndex calls","Retry reconciliation after transient failures; it is designed to be re-run"],"tags":["shards","iteration","reconciliation","weaviate"],"backgroundTag":"shard-iteration-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}