rqlite/rqlite · error

failed to remove temporary upgraded snapshot directory %s: %

Error message

failed to remove temporary upgraded snapshot directory %s: %s

What it means

Upgrade7To8 migrates a v7 snapshot directory to v8. If a temporary directory for the upgraded snapshot already exists (a previous upgrade was interrupted), it removes it first; this error wraps an os.RemoveAll failure on that temp directory and aborts the upgrade.

Source

Thrown at snapshot/upgrader.go:49

			stats.Add(upgradeFail, 1)
		}
	}()
	newTmpDir := tmpName(new)
	defer func() {
		if retErr != nil {
			if err := os.RemoveAll(newTmpDir); err != nil && !os.IsNotExist(err) {
				logger.Printf("failed to remove temporary upgraded snapshot directory at %s due to outer error (%s) cleanup: %s",
					newTmpDir, retErr, err)
			}
		}
	}()

	// If a temporary version of the new snapshot exists, remove it. This implies a
	// previous upgrade attempt was interrupted. We will need to start over.
	if fsutil.DirExists(newTmpDir) {
		logger.Printf("detected temporary upgraded snapshot directory at %s, removing it", newTmpDir)
		if err := os.RemoveAll(newTmpDir); err != nil {
			return fmt.Errorf("failed to remove temporary upgraded snapshot directory %s: %s", newTmpDir, err)
		}
	}

	if !fsutil.DirExists(old) {
		logger.Printf("old v7 snapshot directory does not exist at %s, nothing to upgrade", old)
		return nil
	}

	oldIsEmpty, err := fsutil.DirIsEmpty(old)
	if err != nil {
		return fmt.Errorf("failed to check if old snapshot directory %s is empty: %s", old, err)
	}

	if oldIsEmpty {
		logger.Printf("old snapshot directory %s is empty, nothing to upgrade", old)
		if err := os.RemoveAll(old); err != nil {
			return fmt.Errorf("failed to remove empty old snapshot directory %s: %s", old, err)
		}

View on GitHub (pinned to 7586a4d1bd)

Solutions

1. Fix permissions/ownership so the process can delete the temporary directory
2. Ensure the filesystem is writable (not mounted read-only)
3. Stop the node and remove the temporary upgraded snapshot directory manually, then restart the upgrade
4. Check for processes holding files open in that directory (lsof) and stop them

Example fix

// before: upgrade aborts with 'failed to remove temporary upgraded snapshot directory ...: permission denied'
sudo systemctl stop rqlited
sudo rm -rf /var/lib/rqlite/<snapshot-dir>.tmp-v8
sudo systemctl start rqlited
// after: upgrade restarts cleanly from v7

// before running the upgrade, ensure the temp dir is removable
newTmpDir := old + ".upgrade-tmp" // per library naming
if fsutil.DirExists(newTmpDir) {
    if err := os.RemoveAll(newTmpDir); err != nil {
        return fmt.Errorf("cannot clear stale upgrade temp dir: %w", err)
    }
}

// Go
if err := upgrader.Upgrade7To8(old, new, logger); err != nil {
    if strings.Contains(err.Error(), "failed to remove temporary upgraded snapshot directory") {
        logger.Printf("stale upgrade temp dir not removable (%v); stop node and clear manually", err)
    }
    return err
}

When it happens

Trigger: Opening a store (via Open) where an earlier v7→v8 upgrade was interrupted, leaving newTmpDir behind, and RemoveAll fails due to permissions, open file handles, read-only FS, or I/O errors.

Common situations: Node crashed or was killed during a snapshot format upgrade; upgrade run as a different user than the directory owner; backup software holding files; disk mounted read-only.

Related errors

• failed to move existing FIFO DB to CDC service directory: %w
• failed to upgrade v7 snapshots: %s
• failed to upgrade v8 snapshots: %s
• CRC32 check of %s: %w
• moving WAL %s: %w

AI-assisted analysis of rqlite/rqlite@7586a4d1bd (2026-09-03). Data as JSON: /api/errors/497720748e3c7ea2. Report an issue: GitHub.