{"record":{"id":"e50d2ce21b7187bc","repo":"can1357/oh-my-pi","slug":"all-backups-failed-integrity-check","errorCode":null,"errorMessage":"All backups failed integrity check","messagePattern":"All backups failed integrity check","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"critical","filePath":"packages/mnemopi/src/dr/recovery.ts","lineNumber":312,"sourceCode":"\t\t\t\t.filter(name => /^mnemopi_backup_.*\\.db\\.gz$/.test(name))\n\t\t\t\t.sort()\n\t\t\t\t.reverse()\n\t\t\t\t.map(name => join(dir, name))\n\t\t: [];\n\n\tif (backups.length === 0) throw new FileNotFoundError(`No backups found in ${dir}`);\n\n\tlet attempts = 0;\n\tfor (const backup of backups) {\n\t\tattempts += 1;\n\t\ttry {\n\t\t\tconst result = restoreBackup(backup, targetPath);\n\t\t\tif (result.integrity_check) return { restored: true, backup_used: backup, attempts };\n\t\t} catch {\n\t\t\t// Try the next backup, matching the Python recovery behavior.\n\t\t}\n\t}\n\tthrow new Error(\"All backups failed integrity check\");\n}\nexport function verifyIntegrity(dbPath?: string | null): boolean {\n\tconst targetPath = dbPath ?? getDefaultPaths().dbPath;\n\tif (!existsSync(targetPath)) return false;\n\n\tlet db: Database | null = null;\n\ttry {\n\t\tdb = openDatabase(targetPath, { create: false, readwrite: false, pragmas: false });\n\t\tconst row = db.query(\"PRAGMA integrity_check\").get() as { integrity_check: string } | null;\n\t\treturn row?.integrity_check === \"ok\";\n\t} catch {\n\t\treturn false;\n\t} finally {\n\t\tcloseQuietly(db);\n\t}\n}\nexport function listBackups(backupDir?: string | null): BackupInfo[] {\n\tconst dir = backupDir ?? getDefaultPaths().backupDir;","sourceCodeStart":294,"sourceCodeEnd":330,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/mnemopi/src/dr/recovery.ts#L294-L330","documentation":"emergencyRestore() iterates backups newest-first, attempting restoreBackup() on each. Failures are swallowed (matching Python recovery behavior) and the next backup is tried. If every candidate fails — corrupt archives or databases failing integrity checks — it throws this generic error after exhausting the list. The individual per-backup reasons are lost because the catch block discards errors.","triggerScenarios":"Calling emergencyRestore(dir, targetPath) where every mnemopi_backup_*.db.gz in dir either fails to decompress/parse or fails verifyIntegrity during restoreBackup().","commonSituations":"A storage failure that corrupted the whole backup directory, all backups pruned to truncated files, restoring onto failing hardware, or gzip/SQLite version incompatibilities affecting every file.","solutions":["Check disk space and filesystem health; free space or repair the volume, then retry.","Manually test each backup (`gzip -t`, sqlite3 integrity_check) to find which fail and why.","Produce a fresh backup from any surviving replica and place it in the backup dir.","Restore from offsite/external backups, since all local candidates are unusable."],"exampleFix":"// before\nawait emergencyRestore('/var/backups/mnemopi', dbPath);\n// after\ndf -h /var/backups && gzip -t /var/backups/mnemopi_backup_*.db.gz  # diagnose first\ntry {\n  await emergencyRestore('/var/backups/mnemopi', dbPath);\n} catch (err) {\n  console.error('All local backups failed; fetch offsite backup before retrying');\n  throw err;\n}","handlingStrategy":"try-catch","validationCode":"import { readdirSync } from 'node:fs';\nimport { gunzipSync } from 'node:zlib';\nimport { readFileSync } from 'node:fs';\nconst candidates = readdirSync(dir).filter(n => /^mnemopi_backup_.*\\.db\\.gz$/.test(n));\nconst anyValid = candidates.some(n => {\n  try {\n    return gunzipSync(readFileSync(join(dir, n))).subarray(0, 16).toString('utf8').startsWith('SQLite format 3');\n  } catch { return false; }\n});\nif (!anyValid) console.error('All local backups unreadable — fetch offsite copies first');","typeGuard":null,"tryCatchPattern":"try {\n  await emergencyRestore(dir, targetPath);\n} catch (err) {\n  if (err instanceof Error && err.message === 'All backups failed integrity check') {\n    console.error('Every local backup is unusable; restore from offsite backup');\n    // page on-call / start incident response — data recovery now depends on external copies\n  } else throw err;\n}","preventionTips":["Replicate backups offsite; never rely on a single directory","Test restores (not just backup jobs) on a schedule","Monitor disk space — a full disk silently corrupts whole backup sets","Keep per-backup failure logs since emergencyRestore swallows individual errors"],"tags":["backup-restore","disaster-recovery","sqlite","data-corruption"],"backgroundTag":"all-backups-failed","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}