atuinsh/atuin · critical

persistence task shouldn't panic

Error message

persistence task shouldn't panic

What it means

This `.expect()` in `Flusher::spawn` (crates/atuin-daemon/src/output_capture/backend/fjall/mod.rs:240) unwraps the `JoinHandle` of the blocking task that calls `db.persist(PersistMode::SyncAll)` every 5s when the store is dirty. The expect fires only if the persistence task itself panicked or was cancelled — actual persist failures are handled gracefully (logged, dirty flag re-set for retry). Losing the flusher task means in-memory fjall data is no longer fsynced to disk, risking data loss on crash/power loss.

Source

Thrown at crates/atuin-daemon/src/output_capture/backend/fjall/mod.rs:240

                //
                // Well the persist won't observe those db-writes.
                //
                // The counter-argument is that both db-write and persist acquire the same mutex,
                // so must be seq-cst-ordered?
                //
                // Unsure but would be curious to learn more.
                //
                // @taylordotfish mentioned we shouldn't rely on the internal implementation
                // details.
                if !inner.dirty.swap(false, Ordering::Acquire) {
                    continue;
                }

                let db = inner.db.clone();
                if let Err(err) =
                    tokio::task::spawn_blocking(move || db.persist(PersistMode::SyncAll))
                        .await
                        .expect("persistence task shouldn't panic")
                {
                    error!(?err, "failed to persist data on disk. will try again...");
                    inner.dirty.store(true, Ordering::Relaxed);
                }
            }
        });

        Self { task }
    }
}

impl Drop for Flusher {
    fn drop(&mut self) {
        // Stop the background loop once nothing is holding the flusher any more.
        self.task.abort();
    }
}

View on GitHub (pinned to c0c717ab04)

Solutions

1. Update atuin-daemon; if this is a shutdown-ordering panic (Drop aborts the task while persist is in flight), a fixed release will handle JoinError::is_cancelled without panicking
2. Check disk health and filesystem errors around the fjall data directory — persist panics usually trace to I/O faults
3. Restart the daemon to restore the flusher; verify dirty data is persisted by checking segment file mtimes after a few seconds
4. In deployments, ensure the daemon is stopped gracefully (SIGTERM handled) so the flusher is not cancelled mid-persist

Example fix

// before
if let Err(err) = tokio::task::spawn_blocking(move || db.persist(PersistMode::SyncAll))
    .await
    .expect("persistence task shouldn't panic")
{
// after: treat cancellation as benign, surface real panics
match tokio::task::spawn_blocking(move || db.persist(PersistMode::SyncAll)).await {
    Ok(Err(err)) => {
        error!(?err, "failed to persist data on disk. will try again...");
        inner.dirty.store(true, Ordering::Relaxed);
    }
    Err(err) if err.is_cancelled() => break,
    Err(err) => return Err(err),
    Ok(Ok(())) => {}
}

// Verify the flusher's health periodically; a dead flusher means
// dirty data is not being fsynced.
if !flusher_is_alive() {
    restart_backend_or_alert();
}

// Persistence failures are already logged and retried internally;
// guard the whole backend against task death with a supervisor:
match tokio::spawn(flusher_loop(inner)).await {
    Ok(()) => {},
    Err(join_err) if join_err.is_cancelled() => {}, // shutdown: expected
    Err(join_err) => supervisor_restart(join_err),
}

When it happens

Trigger: The periodic flush loop's `spawn_blocking(db.persist(...)).await` resolves to a `JoinError` (panicked payload or cancellation at daemon shutdown) and `.expect("persistence task shouldn't panic")` panics the flusher task; a panic inside fjall's persist path on a failing disk or corrupt WAL also propagates here.

Common situations: Daemon shutdown concurrently aborting the flusher while a persist is in flight (cancellation JoinError); hardware/IO faults causing fjall persist to panic; runtime shutdown ordering issues in tests or embedding code that drops the backend while the flusher is mid-cycle.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• output-capture read task panicked
• output-capture delete task panicked
• output-capture reclaim task panicked
• output-capture write task panicked
• failed to register sigterm handler

AI-assisted analysis of atuinsh/atuin@c0c717ab04 (2026-09-12). Data as JSON: /api/errors/0459f273c021a27c. Report an issue: GitHub.