clockworklabs/SpacetimeDB · error

error syncing data to disk

Error message

error syncing data to disk

What it means

Panic in `DatabaseLogger::tail`/stream logic: when a caller requests a bounded tail (`n = Some(>0)`) of the database log, the implementation locks the log file and calls `inner.sync_data().expect("error syncing data to disk")` inside a blocking thread before snapshotting the last lines. If `sync_data` fails (I/O error, file closed), the panic fires inside `asyncify`, killing the task that requested the log stream.

Solutions

1. Treat it as a storage-fault symptom: check `dmesg`/disk SMART, free space, and mount health on the database directory, then restart the node.
2. Retry the log tail after remediating storage — the panic is per-request, so unlike the commit log poison flag it does not permanently corrupt state.
3. If you only need live logs, request the unbounded/follow-only variant (`n = None`) which skips the `sync_data` path entirely.
4. For embedded use, catch the panic boundary (`futures::FutureExt::catch_unwind`) around log-tail streams so a sync failure degrades to stale logs instead of a crashed task.

Example fix

// before: bounded tail triggers sync_data and can panic
logger.tail(Some(100), true).await

// after: follow-only stream avoids the sync_data path
logger.tail(None, true).await

// Avoid the sync path entirely when live-following: pass unbounded n
let stream = db_logger.tail(None, true).await?; // skips sync_data entirely

let fut = logger.tail(Some(n), false);
let r = future::FutureExt::catch_unwind(std::pin::pin!(fut));
if r.await.is_err() { /* fall back to None (follow-only) which never syncs */ }

When it happens

Trigger: Calling `tail(Some(n), ..)` / subscribing to a bounded database-log tail while the underlying log file is on failing storage or was closed/truncated externally. With `None` or `Some(0)` the sync path is skipped entirely, so only bounded tails hit it.

Common situations: Reading recent database logs (e.g. `spacetime logs --last N` or energy/metrics dashboards tailing logs) right as a disk error or container volume issue occurs; file descriptor invalidated by a container restart; quota exhaustion during sync. Usually co-occurs with errors 604/605 from the same storage fault.

Related errors

• fsync failed
• log worker panicked
• Duration overflows i64 microseconds
• Duration since Unix epoch overflows i64 microseconds
• expected ModuleDef to contain key, but it does not

AI-assisted analysis of clockworklabs/SpacetimeDB@6dee26c6ef (2026-08-20). Data as JSON: /api/errors/44a4001a74a8f632. Report an issue: GitHub.

Appendix: source

Thrown at crates/core/src/database_logger.rs:518

        // If following isn't requested, we can stream the data.
        if !follow {
            return self.logger.lock().await.tail_stream(n);
        }
        match n {
            // If we don't need to access the disk,
            // locking and spawning can be avoided.
            None | Some(0) => self.subscribe().map(Ok).boxed(),

            // Otherwise, we need to hold the lock to prevent writes
            // while we gather a snapshot of the persistent tail.
            Some(n) => {
                // Cap reading the tail into memory at a few hundred KiB.
                let n = n.min(2500);
                let (tail, more) = {
                    let inner = self.logger.clone().lock_owned().await;
                    let more = self.subscribe();
                    asyncify(move || {
                        inner.sync_data().expect("error syncing data to disk");
                        (inner.tail(n), more)
                    })
                }
                .await;

                stream::once(future::ready(tail)).chain(more.map(Ok)).boxed()
            }
        }
    }

    fn subscribe(&self) -> impl Stream<Item = Bytes> + use<T> {
        BroadcastStream::new(self.broadcast.subscribe()).filter_map(move |x| {
            future::ready(match x {
                Ok(chunk) => Some(chunk),
                Err(BroadcastStreamRecvError::Lagged(skipped)) => {
                    log::trace!("skipped {skipped} lines in module log");
                    None
                }

View on GitHub (pinned to 6dee26c6ef)