{"record":{"id":"03fccc9f45fe6796","repo":"JuliusBrussee/caveman","slug":"ccr-recovery-storage-changed-restart-the-process-and-restore","errorCode":null,"errorMessage":"ccr: recovery storage changed; restart the process and restore missing recovery files","messagePattern":"ccr: recovery storage changed; restart the process and restore missing recovery files","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"engine/ccr/store.go","lineNumber":38,"sourceCode":"\t\"errors\"\n\t\"fmt\"\n\t\"slices\"\n\t\"strings\"\n\t\"time\"\n)\n\n// ErrNotFound is returned by Get when a handle is unknown. The store never\n// guesses a recovery — an unknown handle is an explicit miss.\nvar ErrNotFound = errors.New(\"ccr: recovery handle not found\")\n\n// ErrBudgetExceeded means a new recovery was refused before publishing lossy\n// bytes because the local store's configured payload budget would be exceeded.\n// Existing handles remain intact and retrievable; callers must pass through.\nvar ErrBudgetExceeded = errors.New(\"ccr: storage budget exceeded\")\n\n// ErrStorageChanged means the recovery database no longer matches the open\n// connection. Callers must preserve the original input and report the error.\nvar ErrStorageChanged = errors.New(\"ccr: recovery storage changed; restart the process and restore missing recovery files\")\n\n// ObjectType is a closed typed-working-memory enum. Unknown values fail closed:\n// adapters may preserve unknown native payloads outside CCR, but may not invent\n// retrieval semantics for them.\ntype ObjectType string\n\nconst (\n\tObjectFileObservation      ObjectType = \"FileObservation\"\n\tObjectSearchResult         ObjectType = \"SearchResult\"\n\tObjectCommandResult        ObjectType = \"CommandResult\"\n\tObjectTestResult           ObjectType = \"TestResult\"\n\tObjectBuildResult          ObjectType = \"BuildResult\"\n\tObjectDiffSnapshot         ObjectType = \"DiffSnapshot\"\n\tObjectTaskContract         ObjectType = \"TaskContract\"\n\tObjectTaskDecision         ObjectType = \"TaskDecision\"\n\tObjectExecutionState       ObjectType = \"ExecutionState\"\n\tObjectDocumentationExcerpt ObjectType = \"DocumentationExcerpt\"\n\tObjectBrowserSnapshot      ObjectType = \"BrowserSnapshot\"","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/3ee70a102609e550bd2e68004bf5990a9341c851/engine/ccr/store.go#L20-L56","documentation":"CCR's sentinel ErrStorageChanged: the recovery database on disk no longer matches the open connection (generation/file inspection failed via inspectSQLiteGeneration/checkGeneration). The store fails closed, telling the caller to restart the process and restore missing recovery files rather than continue on unverified storage.","triggerScenarios":"During Put or generation checks, inspectSQLiteGeneration detects: the database parent dir no longer resolves to itself (symlink/path changed), DB files deleted or replaced mid-operation, or the parent path became unverifiable (os.ErrNotExist).","commonSituations":"External cleanup jobs or tmpfs reaping removed the DB file; another process recreated/moved the recovery database; running in a container with an ephemeral filesystem wiping the store dir; symlinked paths retargeted.","solutions":["Restart the process so the store reopens storage matching the current on-disk generation.","Restore the missing/moved recovery database files to the configured path.","Stop external processes (cleaners, sync tools) from deleting or moving the store directory.","Pin a stable, non-symlinked path for the recovery DB in configuration."],"exampleFix":"// before\nstore, _ := ccr.Open(cfg) // path under volatile /tmp\ndefer store.Close()\n\n// after\n// use a persistent dir and monitor ErrStorageChanged\nstore, _ := ccr.Open(cfgWithPersistentPath)\nif errors.Is(err, ccr.ErrStorageChanged) {\n    return restartAndRestore(cfgWithPersistentPath, backup)\n}","handlingStrategy":"try-catch","validationCode":"if _, err := os.Stat(storePath); errors.Is(err, os.ErrNotExist) { return restoreBackup(storePath) }","typeGuard":null,"tryCatchPattern":"if errors.Is(err, ccr.ErrStorageChanged) {\n    log.Fatalf(\"ccr storage changed: restart process and restore %s\", storePath)\n}\n","preventionTips":["Place the recovery DB on persistent, non-ephemeral storage.","Exclude the store directory from cleaners/sync/tmp-reaping.","Use a real directory, not a symlink, for the DB path.","Keep verified backups to restore after a generation mismatch."],"tags":["ccr","storage","consistency","sentinel-error","fail-closed"],"backgroundTag":"storage-state-changed","analyzedSha":"3ee70a102609e550bd2e68004bf5990a9341c851","analyzedAt":"2026-09-20T15:53:39.229Z","contentChangedAt":"2026-09-20T15:53:39.229Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}