weaviate/weaviate · error

read migrations dir for stale-state cleanup: %w

Error message

read migrations dir for stale-state cleanup: %w

What it means

During stale partial-reindex state cleanup, the shard reads the .migrations directory under its LSM path to enumerate stale migration dirs. If os.ReadDir fails for any reason other than the directory not existing, this error wraps and returns it. A missing directory is treated as nothing-to-clean and is not an error.

Source

Thrown at adapters/repos/db/shard_init_properties.go:289

}

// cleanStaleMigrationDirsIn is [cleanStaleMigrationDirsAt] on a caller-built
// scope, so a sweep can share one payload memo with its preserve pass.
//
// A listing it cannot read is returned, not logged: this helper removed
// nothing, so a caller that reports its own outcome would otherwise call a
// sweep finished on a directory it never read. Removal failures stay logged,
// since those leave the rest of the sweep done. A cancelled ctx is returned
// too, for the same reason and so the sweep path can report it as a run that
// stopped rather than a shard that failed ([truncatedByCancellation]).
func cleanStaleMigrationDirsIn(ctx context.Context, scope migrationDirScope, logger logrus.FieldLogger) error {
	migrationsRoot := filepath.Join(scope.lsmPath, ".migrations")
	entries, err := os.ReadDir(migrationsRoot)
	if err != nil {
		if os.IsNotExist(err) {
			return nil
		}
		return fmt.Errorf("read migrations dir for stale-state cleanup: %w", err)
	}
	// Asked before the preserve pass rather than only inside the loop: that
	// pass opens a tracker payload per dir whose name leaves the property
	// open, and nothing interrupts it once it starts.
	if err := ctx.Err(); err != nil {
		return fmt.Errorf("stale-state cleanup stopped before reading %s: %w", migrationsRoot, err)
	}
	preserved := completedMigrationGens(scope)
	for _, entry := range entries {
		if err := ctx.Err(); err != nil {
			return fmt.Errorf("stale-state cleanup stopped partway through %s: %w", migrationsRoot, err)
		}
		if !entry.IsDir() {
			continue
		}
		name := entry.Name()
		if !scope.inScope(name) {
			continue

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check permissions and ownership of the shard's .migrations directory
2. Verify the path is a directory (if it became a file, it is corrupt — remove or restore it with the node stopped)
3. Check storage/disk health (dmesg, mount errors) if EIO appears
4. Re-run the cleanup after fixing; the operation is safe to retry

info, err := os.Stat(filepath.Join(lsmPath, ".migrations"))
if err == nil && !info.IsDir() {
	// path is corrupt: .migrations exists but is not a directory
}
if err != nil && !os.IsNotExist(err) {
	// cleanup will fail: fix permissions/access first
}

if err := shard.CleanStalePartialReindexState(ctx, prop, idxType); err != nil {
	if !errors.Is(err, fs.ErrNotExist) {
		logger.Errorf("cleanup failed: %v", err)
	}
}

When it happens

Trigger: os.ReadDir on <lsmPath>/.migrations fails with a non-ENOENT error — e.g. permission denied, EIO, or the path existing but being a file, while running CleanStalePartialReindexState or cleanStaleMigrationDirsAt.

Common situations: Corrupted shard directory where .migrations was replaced by a regular file, wrong ownership/permissions after container migration, or disk I/O failures on networked storage.

Related errors

• shard %q: %w
• marking swapped prop %q: %w
• mkdir migration dir %q: %w
• recovery tidy: %w
• recovery tidy after swap: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/a33284866af01197. Report an issue: GitHub.