{"record":{"id":"16531fada04242d4","repo":"weaviate/weaviate","slug":"load-ingest-buckets-w","errorCode":null,"errorMessage":"load ingest buckets: %w","messagePattern":"load ingest buckets: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/inverted_reindex_task_generic.go","lineNumber":783,"sourceCode":"\t\t\tmissingIngest = append(missingIngest, propName)\n\t\t}\n\t}\n\n\tif len(missingReindex) > 0 {\n\t\tlogger.WithField(\"props\", missingReindex).\n\t\t\tWarn(\"reindex buckets not in store but dirs exist; defensively loading before runtime swap\")\n\t\tif err := t.loadReindexBuckets(ctx, logger, shard, missingReindex); err != nil {\n\t\t\treturn fmt.Errorf(\"load reindex buckets: %w\", err)\n\t\t}\n\t}\n\tif len(missingIngest) > 0 {\n\t\tlogger.WithField(\"props\", missingIngest).\n\t\t\tWarn(\"ingest buckets not in store but dirs exist; defensively loading before runtime swap\")\n\t\t// keepLevelCompaction=false, keepTombstones=false: at this\n\t\t// point (pre-prepend, mid-runtimeSwap) the standard\n\t\t// post-merge ingest options apply.\n\t\tif err := t.loadIngestBuckets(ctx, logger, shard, missingIngest, false, false); err != nil {\n\t\t\treturn fmt.Errorf(\"load ingest buckets: %w\", err)\n\t\t}\n\t}\n\treturn nil\n}\n\n// finalizeMigrationAfterRecovery runs the strategy's OnMigrationComplete\n// hook and trims older on-disk generations. This is the rehydrate-path\n// equivalent of runtimeSwap's final two steps (lines 1103/1124),\n// invoked by the recovery branches in [RunSwapOnShard] which don't go\n// through runtimeSwap.\n//\n// Best-effort on trim — failures are logged, not returned, matching\n// the trim policy at the end of runtimeSwap.\nfunc (t *ShardReindexTaskGeneric) finalizeMigrationAfterRecovery(\n\tctx context.Context, logger logrus.FieldLogger, shard ShardLike,\n\trt reindexTracker, props []string,\n) error {\n\t// Ordering contract: rebuild must run and be checked before","sourceCodeStart":765,"sourceCodeEnd":801,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/inverted_reindex_task_generic.go#L765-L801","documentation":"This error wraps a failure from loadIngestBuckets() inside ensureReindexBucketsLoadedForSwap, which defensively reloads ingest buckets missing from the in-memory store while their directories exist, using the standard post-merge options (keepLevelCompaction=false, keepTombstones=false). It runs immediately before the pre-prepend runtime swap.","triggerScenarios":"RunSwapOnShard (default branch) or RunPrepareOnShard detects a property's ingest bucket is nil in the store while the ingest_<gen> dir exists on disk, and loading it fails — corrupted memtable/segment WAL files, I/O errors, or shutdown racing the load.","commonSituations":"Restart during the prepend window leaves ingest_<gen> dirs unopened; WAL replay failure on a torn write; disk-space or permission problems; concurrent shard drop during recovery.","solutions":["Inspect the wrapped cause for WAL/segment corruption in the ingest_<gen> dir; repair the underlying disk issue first","If the ingest dir is unrecoverable, restore from backup or rebuild the index — dropping a corrupt ingest dir loses only unmerged reindex output that the reindex will regenerate","Retry after a clean restart so the normal OnAfterLsmInit recovery hooks open the buckets instead of the defensive path","Verify the shard is not being dropped/shut down concurrently with the swap","Check disk space and directory permissions on the shard LSM path"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// pre-check ingest dir integrity before swap\nfor _, p := range props {\n    dir := filepath.Join(shard.PathLSM(), task.ingestBucketName(p))\n    if _, err := os.ReadDir(dir); err != nil {\n        return fmt.Errorf(\"prop %q ingest dir unreadable: %w\", p, err)\n    }\n}\n// confirm no shutdown in progress before driving the swap\nif ctx.Err() != nil { return ctx.Err() }","typeGuard":null,"tryCatchPattern":"if err := ensureReindexBucketsLoadedForSwap(ctx, logger, shard, props); err != nil {\n    if errors.Is(err, context.Canceled) {\n        return err // transient\n    }\n    logger.Errorf(\"ingest bucket load failed, shard %q may need restore: %v\", shard.Name(), err)\n    return err\n}","preventionTips":["Monitor WAL replay errors at shard startup — torn WALs predict this failure","Enforce disk free-space alerts on LSM volumes","Quiesce shard drops and store shutdown during migration recovery","Keep backups of shards mid-migration; a corrupt ingest dir may require restore or index rebuild"],"tags":["reindex","lsm","bucket-loading","wal"],"backgroundTag":"lsm-bucket-load-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}