{"record":{"id":"497720748e3c7ea2","repo":"rqlite/rqlite","slug":"failed-to-remove-temporary-upgraded-snapshot-direc","errorCode":null,"errorMessage":"failed to remove temporary upgraded snapshot directory %s: %s","messagePattern":"failed to remove temporary upgraded snapshot directory (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"snapshot/upgrader.go","lineNumber":49,"sourceCode":"\t\t\tstats.Add(upgradeFail, 1)\n\t\t}\n\t}()\n\tnewTmpDir := tmpName(new)\n\tdefer func() {\n\t\tif retErr != nil {\n\t\t\tif err := os.RemoveAll(newTmpDir); err != nil && !os.IsNotExist(err) {\n\t\t\t\tlogger.Printf(\"failed to remove temporary upgraded snapshot directory at %s due to outer error (%s) cleanup: %s\",\n\t\t\t\t\tnewTmpDir, retErr, err)\n\t\t\t}\n\t\t}\n\t}()\n\n\t// If a temporary version of the new snapshot exists, remove it. This implies a\n\t// previous upgrade attempt was interrupted. We will need to start over.\n\tif fsutil.DirExists(newTmpDir) {\n\t\tlogger.Printf(\"detected temporary upgraded snapshot directory at %s, removing it\", newTmpDir)\n\t\tif err := os.RemoveAll(newTmpDir); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to remove temporary upgraded snapshot directory %s: %s\", newTmpDir, err)\n\t\t}\n\t}\n\n\tif !fsutil.DirExists(old) {\n\t\tlogger.Printf(\"old v7 snapshot directory does not exist at %s, nothing to upgrade\", old)\n\t\treturn nil\n\t}\n\n\toldIsEmpty, err := fsutil.DirIsEmpty(old)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to check if old snapshot directory %s is empty: %s\", old, err)\n\t}\n\n\tif oldIsEmpty {\n\t\tlogger.Printf(\"old snapshot directory %s is empty, nothing to upgrade\", old)\n\t\tif err := os.RemoveAll(old); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to remove empty old snapshot directory %s: %s\", old, err)\n\t\t}","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/rqlite/rqlite/blob/7586a4d1bdbd9a5a80021664c5a863cd850adb60/snapshot/upgrader.go#L31-L67","documentation":"Upgrade7To8 migrates a v7 snapshot directory to v8. If a temporary directory for the upgraded snapshot already exists (a previous upgrade was interrupted), it removes it first; this error wraps an os.RemoveAll failure on that temp directory and aborts the upgrade.","triggerScenarios":"Opening a store (via Open) where an earlier v7→v8 upgrade was interrupted, leaving newTmpDir behind, and RemoveAll fails due to permissions, open file handles, read-only FS, or I/O errors.","commonSituations":"Node crashed or was killed during a snapshot format upgrade; upgrade run as a different user than the directory owner; backup software holding files; disk mounted read-only.","solutions":["Fix permissions/ownership so the process can delete the temporary directory","Ensure the filesystem is writable (not mounted read-only)","Stop the node and remove the temporary upgraded snapshot directory manually, then restart the upgrade","Check for processes holding files open in that directory (lsof) and stop them"],"exampleFix":"// before: upgrade aborts with 'failed to remove temporary upgraded snapshot directory ...: permission denied'\nsudo systemctl stop rqlited\nsudo rm -rf /var/lib/rqlite/<snapshot-dir>.tmp-v8\nsudo systemctl start rqlited\n// after: upgrade restarts cleanly from v7","handlingStrategy":"fallback","validationCode":"// before running the upgrade, ensure the temp dir is removable\nnewTmpDir := old + \".upgrade-tmp\" // per library naming\nif fsutil.DirExists(newTmpDir) {\n    if err := os.RemoveAll(newTmpDir); err != nil {\n        return fmt.Errorf(\"cannot clear stale upgrade temp dir: %w\", err)\n    }\n}","typeGuard":null,"tryCatchPattern":"// Go\nif err := upgrader.Upgrade7To8(old, new, logger); err != nil {\n    if strings.Contains(err.Error(), \"failed to remove temporary upgraded snapshot directory\") {\n        logger.Printf(\"stale upgrade temp dir not removable (%v); stop node and clear manually\", err)\n    }\n    return err\n}","preventionTips":["Avoid killing nodes during snapshot format upgrades; let them finish","Keep upgrade directories owned by the same user as the process","Exclude data dirs from processes that lock files during upgrades","After a crash, inspect for leftover upgrade temp dirs before restarting"],"tags":["snapshot","upgrade","migration","filesystem"],"backgroundTag":"temp-dir-cleanup-failed","analyzedSha":"7586a4d1bdbd9a5a80021664c5a863cd850adb60","analyzedAt":"2026-09-03T07:03:02.260Z","contentChangedAt":"2026-09-03T07:03:02.260Z","schemaVersion":2},"datasetVersion":"2026-09-10T12:17:11.382Z"}