recovery rename for %q: main missing, backup exists, but ing

Error message

recovery rename for %q: main missing, backup exists, but ingest dir missing — unrecoverable

What it means

This unrecoverable-state error is thrown when crash recovery finds a property's bucket directories in an impossible combination: the canonical main bucket directory is missing and a backup directory exists (proof that the first half of the rename pair ran), but the ingest directory that should be renamed to main is absent. There is no directory left to restore or promote — the newly built data cannot be located — so recovery aborts instead of guessing.

Source

Thrown at adapters/repos/db/inverted_reindex_task_generic.go:2016

		ingestExists := dirExists(ingestDir)

		switch {
		case mainExists && !backupExists:
			// Pre-rename state (either swap not started, or post-refactor
			// happy-path deferred-rename). Do the full rename pair.
			if !ingestExists {
				return fmt.Errorf("recovery rename for %q: main exists, no backup, but ingest dir missing — unrecoverable", propName)
			}
			if err := os.Rename(mainDir, backupDir); err != nil {
				return fmt.Errorf("recovery rename main->backup for %q: %w", propName, err)
			}
			if err := os.Rename(ingestDir, mainDir); err != nil {
				return fmt.Errorf("recovery rename ingest->main for %q: %w", propName, err)
			}
		case !mainExists && backupExists:
			// Halfway: main was renamed to backup but ingest not yet to main.
			if !ingestExists {
				return fmt.Errorf("recovery rename for %q: main missing, backup exists, but ingest dir missing — unrecoverable", propName)
			}
			if err := os.Rename(ingestDir, mainDir); err != nil {
				return fmt.Errorf("recovery rename ingest->main for %q: %w", propName, err)
			}
		case mainExists && backupExists:
			// Both exist — ingest was already renamed to main on a prior
			// recovery pass. Idempotent no-op.
		default:
			return fmt.Errorf("unexpected disk state for prop %q: main=%v backup=%v ingest=%v",
				propName, mainExists, backupExists, ingestExists)
		}

		// markSwappedProp creates with O_EXCL; a mid-FINALIZING restart may
		// have already set the sentinel.
		if !rt.IsSwappedProp(propName) {
			if err := rt.markSwappedProp(propName); err != nil {
				return fmt.Errorf("marking swapped prop %q: %w", propName, err)
			}

View on GitHub (pinned to 75aa4b6d11)

Solutions

Inspect the shard's lsm-contents directory to confirm main_<...>, backup_<...> and ingest_<...> names; if the ingest dir was accidentally moved/renamed, restore it to its expected ingest_<gen> name and restart so recovery can complete.
Restore the affected shard from the most recent backup/snapshot taken before the interrupted swap — the new inverted index data is lost and must be rebuilt.
Trigger a reindex/migration from scratch for the affected property (drop and re-add the index config, or re-run the reindex task) so fresh ingest buckets are built.
Audit any cleanup scripts or operator runbooks so ingest_* directories inside lsm-contents are never deleted outside of Weaviate's own FinalizeCompletedMigrations/tidy step.

Example fix

// before: operator cleanup deletes 'leftover' dirs
rm -rf lsm-contents/ingest_property_text   // shard can no longer recover

// after: never delete ingest_* manually; let recovery finish, then let
// FinalizeCompletedMigrations/tidyBackupBuckets clean up backup_* dirs

Defensive patterns

Strategy: validation

Validate before calling

// before restarting a node with interrupted swaps, confirm all three dirs are present per pending prop
func validateSwapDirs(lsmPath string, strategy interface{ SourceBucketName(string) string }, ingestName, backupName string) error {
	for _, d := range []string{strategy.SourceBucketName("prop"), ingestName, backupName} {
		// main OR (backup AND ingest) must exist; log what's missing before any restart
		if _, err := os.Stat(filepath.Join(lsmPath, d)); err != nil {
			return fmt.Errorf("expected dir %s missing in %s: %w", d, lsmPath, err)
		}
	}
	return nil
}

Type guard

func isRecoverableSwapState(mainExists, backupExists, ingestExists bool) bool {
	// main present (full pair) or halfway (backup+ingest) are recoverable;
	// backup without ingest is not.
	return (mainExists && !backupExists && ingestExists) || (!mainExists && backupExists && ingestExists)
}

Try / catch

if err := task.RunSwapOnShard(ctx, shard); err != nil {
	if strings.Contains(err.Error(), "unrecoverable") {
		// halt — data must be restored from backup; never attempt manual dir surgery
		logger.Errorf("shard swap unrecoverable, restore from snapshot: %v", err)
		return errShutdownForRestore
	}
	return err
}

Prevention

Never delete ingest_* directories inside lsm-contents manually — they are recovery-critical until FinalizeCompletedMigrations runs
Restore shards from snapshots atomically (whole shard directory), never partially
Use durable shutdown (SIGTERM handling) so rename sequences are not interrupted mid-pair
Run fsck/smart checks if directories disappear without human action

When it happens

Trigger: RunSwapOnShard -> recoverRuntimeSwapBuckets evaluates case !mainExists && backupExists for a property and dirExists(ingestDir) returns false, i.e. someone or something deleted the ingest_<gen> directory (or it was never flushed/created on this volume) between the main->backup rename and recovery.

Common situations: Manual cleanup of 'stray' ingest_* directories by an operator or script that did not know they were recovery-critical; data loss after an unclean shutdown where the ingest dir's creation was not durably synced; restoring only part of a shard directory from a snapshot; copying shards between nodes while a swap was mid-flight.

Related errors

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/f0c17b456e7476ba. Report an issue: GitHub.