{"record":{"id":"1ef25dd2bc0b70db","repo":"JuliusBrussee/caveman","slug":"w-database-or-journal-removed-or-replaced-connection","errorCode":null,"errorMessage":"%w: database or journal removed or replaced; connection quarantined until process exit","messagePattern":"%w: database or journal removed or replaced; connection quarantined until process exit","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"engine/ccr/store_generation.go","lineNumber":162,"sourceCode":"\tif errors.Is(err, errStorageUnverifiable) {\n\t\t// \"Could not look\" is not \"was replaced\". A momentary stat/permission\n\t\t// failure — an antivirus lock on Windows, EIO on a network home, a\n\t\t// parent directory whose mode was loose for one instant — fails THIS\n\t\t// operation closed, but must not latch the terminal state below and\n\t\t// make already-stored recoveries unreadable for the rest of the process.\n\t\treturn err\n\t}\n\tif err != nil {\n\t\ts.quarantined = err\n\t\treturn s.quarantined\n\t}\n\tif s.db != nil && s.files.same(files) {\n\t\treturn nil\n\t}\n\t// No SQLite calls, including Close, follow invalidation. The public driver\n\t// exposes no descriptor identity or NO_CKPT_ON_CLOSE control. Keeping this\n\t// one connection until exit avoids replaying its obsolete journal.\n\ts.quarantined = fmt.Errorf(\"%w: database or journal removed or replaced; connection quarantined until process exit\", ErrStorageChanged)\n\treturn s.quarantined\n}\n\nfunc withStore[T any](s *Store, fn func() (T, error)) (T, error) {\n\ts.mu.Lock()\n\tdefer s.mu.Unlock()\n\tvar zero T\n\tif err := s.checkGeneration(); err != nil {\n\t\treturn zero, err\n\t}\n\tvalue, err := fn()\n\tif changed := s.checkGeneration(); changed != nil {\n\t\treturn zero, changed\n\t}\n\treturn value, err\n}\n\n// Close releases a valid connection. A quarantined connection is deliberately","sourceCodeStart":144,"sourceCodeEnd":180,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/3ee70a102609e550bd2e68004bf5990a9341c851/engine/ccr/store_generation.go#L144-L180","documentation":"checkGeneration detected that the database or one of its journal files was removed or replaced since the Store opened. The connection is terminally quarantined: the library deliberately keeps (never closes) the stale SQLite connection until process exit, because closing could checkpoint an obsolete WAL into the new main file and corrupt the replacement database. Every subsequent Store operation and Close returns this error.","triggerScenarios":"Any Store method (Put, Get, PutObject, Summary, ...) or Close runs after the on-disk file identity (dev/inode of the main DB or existing journal files) changed — the db file was deleted and recreated, swapped by rename, or restored from a different file while the process kept its old descriptor.","commonSituations":"A deployment/restore job replaces ccr.db while the long-running agent process is alive; a container restart of a sidecar wipes the volume; the database is recreated by a parallel process; snapshot/restore tooling rolls the file back to a different inode.","solutions":["Stop replacing the database file under a running process: perform restores/replacements while the engine is stopped, then start it fresh.","If the replacement was intentional, restart the process so a new Store opens the new database — automatic adoption of a replacement is deliberately unsafe.","Check for concurrent processes creating/renaming ccr.db and serialize them or give each process its own store path.","Restore the original file (same inode) if the swap was accidental — note quarantine is latched regardless; a process restart is still required.","Persist recovery data outside the swapped window or snapshot the whole directory atomically (e.g. via rename of the directory, with the process restarted)."],"exampleFix":"// before\nmv ccr.db ccr.db.bak && cp new/ccr.db ccr.db   # while process running\n// after\nsystemctl stop caveman-agent\nmv ccr.db ccr.db.bak && cp new/ccr.db ccr.db\nsystemctl start caveman-agent","handlingStrategy":"try-catch","validationCode":"// compare inode before/after external operations:\nbefore := inodeOf(dbPath) // os.Stat -> syscall.Stat_t.Ino\n// ...external restore...\nif inodeOf(dbPath) != before { restartEngine() }","typeGuard":null,"tryCatchPattern":"if errors.Is(err, ccr.ErrStorageChanged) {\n    // terminal quarantine: fail the request and restart the process/worker\n    // so a fresh Store opens the current database\n    os.Exit(1) // or supervisor restart\n}","preventionTips":["Stop the engine before any db restore, replace, or rollback","Restore the database atomically only while no process holds it open","Give concurrent processes separate store paths","Use supervisor restarts on ErrStorageChanged instead of retrying the same Store"],"tags":["go","sqlite","storage-changed","quarantine","data-safety"],"backgroundTag":"invalid-state-transition","analyzedSha":"3ee70a102609e550bd2e68004bf5990a9341c851","analyzedAt":"2026-09-20T15:53:39.229Z","contentChangedAt":"2026-09-20T15:53:39.229Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}