weaviate/weaviate · error

load ingest buckets: %w

Error message

load ingest buckets: %w

What it means

This error wraps a failure from loadIngestBuckets() inside ensureReindexBucketsLoadedForSwap, which defensively reloads ingest buckets missing from the in-memory store while their directories exist, using the standard post-merge options (keepLevelCompaction=false, keepTombstones=false). It runs immediately before the pre-prepend runtime swap.

Source

Thrown at adapters/repos/db/inverted_reindex_task_generic.go:783

			missingIngest = append(missingIngest, propName)
		}
	}

	if len(missingReindex) > 0 {
		logger.WithField("props", missingReindex).
			Warn("reindex buckets not in store but dirs exist; defensively loading before runtime swap")
		if err := t.loadReindexBuckets(ctx, logger, shard, missingReindex); err != nil {
			return fmt.Errorf("load reindex buckets: %w", err)
		}
	}
	if len(missingIngest) > 0 {
		logger.WithField("props", missingIngest).
			Warn("ingest buckets not in store but dirs exist; defensively loading before runtime swap")
		// keepLevelCompaction=false, keepTombstones=false: at this
		// point (pre-prepend, mid-runtimeSwap) the standard
		// post-merge ingest options apply.
		if err := t.loadIngestBuckets(ctx, logger, shard, missingIngest, false, false); err != nil {
			return fmt.Errorf("load ingest buckets: %w", err)
		}
	}
	return nil
}

// finalizeMigrationAfterRecovery runs the strategy's OnMigrationComplete
// hook and trims older on-disk generations. This is the rehydrate-path
// equivalent of runtimeSwap's final two steps (lines 1103/1124),
// invoked by the recovery branches in [RunSwapOnShard] which don't go
// through runtimeSwap.
//
// Best-effort on trim — failures are logged, not returned, matching
// the trim policy at the end of runtimeSwap.
func (t *ShardReindexTaskGeneric) finalizeMigrationAfterRecovery(
	ctx context.Context, logger logrus.FieldLogger, shard ShardLike,
	rt reindexTracker, props []string,
) error {
	// Ordering contract: rebuild must run and be checked before

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Inspect the wrapped cause for WAL/segment corruption in the ingest_<gen> dir; repair the underlying disk issue first
2. If the ingest dir is unrecoverable, restore from backup or rebuild the index — dropping a corrupt ingest dir loses only unmerged reindex output that the reindex will regenerate
3. Retry after a clean restart so the normal OnAfterLsmInit recovery hooks open the buckets instead of the defensive path
4. Verify the shard is not being dropped/shut down concurrently with the swap
5. Check disk space and directory permissions on the shard LSM path

// pre-check ingest dir integrity before swap
for _, p := range props {
    dir := filepath.Join(shard.PathLSM(), task.ingestBucketName(p))
    if _, err := os.ReadDir(dir); err != nil {
        return fmt.Errorf("prop %q ingest dir unreadable: %w", p, err)
    }
}
// confirm no shutdown in progress before driving the swap
if ctx.Err() != nil { return ctx.Err() }

if err := ensureReindexBucketsLoadedForSwap(ctx, logger, shard, props); err != nil {
    if errors.Is(err, context.Canceled) {
        return err // transient
    }
    logger.Errorf("ingest bucket load failed, shard %q may need restore: %v", shard.Name(), err)
    return err
}

When it happens

Trigger: RunSwapOnShard (default branch) or RunPrepareOnShard detects a property's ingest bucket is nil in the store while the ingest_<gen> dir exists on disk, and loading it fails — corrupted memtable/segment WAL files, I/O errors, or shutdown racing the load.

Common situations: Restart during the prepend window leaves ingest_<gen> dirs unopened; WAL replay failure on a torn write; disk-space or permission problems; concurrent shard drop during recovery.

Related errors

• load reindex buckets: %w
• after async LSM init: %w
• ensure buckets loaded: %w
• recovery tidy: %w
• recovery swap: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/16531fada04242d4. Report an issue: GitHub.