{"record":{"id":"0459f273c021a27c","repo":"atuinsh/atuin","slug":"persistence-task-shouldn-t-panic","errorCode":null,"errorMessage":"persistence task shouldn't panic","messagePattern":"persistence task shouldn't panic","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/atuin-daemon/src/output_capture/backend/fjall/mod.rs","lineNumber":240,"sourceCode":"                //\n                // Well the persist won't observe those db-writes.\n                //\n                // The counter-argument is that both db-write and persist acquire the same mutex,\n                // so must be seq-cst-ordered?\n                //\n                // Unsure but would be curious to learn more.\n                //\n                // @taylordotfish mentioned we shouldn't rely on the internal implementation\n                // details.\n                if !inner.dirty.swap(false, Ordering::Acquire) {\n                    continue;\n                }\n\n                let db = inner.db.clone();\n                if let Err(err) =\n                    tokio::task::spawn_blocking(move || db.persist(PersistMode::SyncAll))\n                        .await\n                        .expect(\"persistence task shouldn't panic\")\n                {\n                    error!(?err, \"failed to persist data on disk. will try again...\");\n                    inner.dirty.store(true, Ordering::Relaxed);\n                }\n            }\n        });\n\n        Self { task }\n    }\n}\n\nimpl Drop for Flusher {\n    fn drop(&mut self) {\n        // Stop the background loop once nothing is holding the flusher any more.\n        self.task.abort();\n    }\n}\n","sourceCodeStart":222,"sourceCodeEnd":258,"githubUrl":"https://github.com/atuinsh/atuin/blob/c0c717ab04c881764bcad4b3d169a507e2432643/crates/atuin-daemon/src/output_capture/backend/fjall/mod.rs#L222-L258","documentation":"This `.expect()` in `Flusher::spawn` (crates/atuin-daemon/src/output_capture/backend/fjall/mod.rs:240) unwraps the `JoinHandle` of the blocking task that calls `db.persist(PersistMode::SyncAll)` every 5s when the store is dirty. The expect fires only if the persistence task itself panicked or was cancelled — actual persist failures are handled gracefully (logged, dirty flag re-set for retry). Losing the flusher task means in-memory fjall data is no longer fsynced to disk, risking data loss on crash/power loss.","triggerScenarios":"The periodic flush loop's `spawn_blocking(db.persist(...)).await` resolves to a `JoinError` (panicked payload or cancellation at daemon shutdown) and `.expect(\"persistence task shouldn't panic\")` panics the flusher task; a panic inside fjall's persist path on a failing disk or corrupt WAL also propagates here.","commonSituations":"Daemon shutdown concurrently aborting the flusher while a persist is in flight (cancellation JoinError); hardware/IO faults causing fjall persist to panic; runtime shutdown ordering issues in tests or embedding code that drops the backend while the flusher is mid-cycle.","solutions":["Update atuin-daemon; if this is a shutdown-ordering panic (Drop aborts the task while persist is in flight), a fixed release will handle JoinError::is_cancelled without panicking","Check disk health and filesystem errors around the fjall data directory — persist panics usually trace to I/O faults","Restart the daemon to restore the flusher; verify dirty data is persisted by checking segment file mtimes after a few seconds","In deployments, ensure the daemon is stopped gracefully (SIGTERM handled) so the flusher is not cancelled mid-persist"],"exampleFix":"// before\nif let Err(err) = tokio::task::spawn_blocking(move || db.persist(PersistMode::SyncAll))\n    .await\n    .expect(\"persistence task shouldn't panic\")\n{\n// after: treat cancellation as benign, surface real panics\nmatch tokio::task::spawn_blocking(move || db.persist(PersistMode::SyncAll)).await {\n    Ok(Err(err)) => {\n        error!(?err, \"failed to persist data on disk. will try again...\");\n        inner.dirty.store(true, Ordering::Relaxed);\n    }\n    Err(err) if err.is_cancelled() => break,\n    Err(err) => return Err(err),\n    Ok(Ok(())) => {}\n}","handlingStrategy":"fallback","validationCode":"// Verify the flusher's health periodically; a dead flusher means\n// dirty data is not being fsynced.\nif !flusher_is_alive() {\n    restart_backend_or_alert();\n}","typeGuard":null,"tryCatchPattern":"// Persistence failures are already logged and retried internally;\n// guard the whole backend against task death with a supervisor:\nmatch tokio::spawn(flusher_loop(inner)).await {\n    Ok(()) => {},\n    Err(join_err) if join_err.is_cancelled() => {}, // shutdown: expected\n    Err(join_err) => supervisor_restart(join_err),\n}","preventionTips":["Stop the daemon with SIGTERM and wait for graceful shutdown so the flusher is not aborted mid-persist","Monitor the fjall data directory for recent writes (persist cadence ~5s) to detect a dead flusher","Ensure adequate disk space and healthy storage; persist panics frequently follow I/O errors","Upgrade atuin-daemon so JoinError::cancelled at shutdown is not turned into a panic"],"tags":["panic","storage","fjall","tokio","persistence"],"backgroundTag":"internal-invariant-violation","analyzedSha":"c0c717ab04c881764bcad4b3d169a507e2432643","analyzedAt":"2026-09-12T07:40:01.341Z","contentChangedAt":"2026-09-12T07:40:01.341Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}