clockworklabs/SpacetimeDB · critical

error syncing data to disk

Error message

error syncing data to disk

What it means

When `DatabaseLogger::tail` is asked for the last `n` lines, it takes the file lock and — inside a blocking thread — calls `sync_data().expect("error syncing data to disk")` so the snapshot is durable before streaming. This expect fires when the OS rejects the fsync on the log file: disk full, I/O error, or the file living on a failed/unplugged filesystem.

Source

Thrown at crates/core/src/database_logger.rs:518

        // If following isn't requested, we can stream the data.
        if !follow {
            return self.logger.lock().await.tail_stream(n);
        }
        match n {
            // If we don't need to access the disk,
            // locking and spawning can be avoided.
            None | Some(0) => self.subscribe().map(Ok).boxed(),

            // Otherwise, we need to hold the lock to prevent writes
            // while we gather a snapshot of the persistent tail.
            Some(n) => {
                // Cap reading the tail into memory at a few hundred KiB.
                let n = n.min(2500);
                let (tail, more) = {
                    let inner = self.logger.clone().lock_owned().await;
                    let more = self.subscribe();
                    asyncify(move || {
                        inner.sync_data().expect("error syncing data to disk");
                        (inner.tail(n), more)
                    })
                }
                .await;

                stream::once(future::ready(tail)).chain(more.map(Ok)).boxed()
            }
        }
    }

    fn subscribe(&self) -> impl Stream<Item = Bytes> + use<T> {
        BroadcastStream::new(self.broadcast.subscribe()).filter_map(move |x| {
            future::ready(match x {
                Ok(chunk) => Some(chunk),
                Err(BroadcastStreamRecvError::Lagged(skipped)) => {
                    log::trace!("skipped {skipped} lines in module log");
                    None
                }

View on GitHub (pinned to 524b4487d9)

Solutions

1. Free or grow space on the volume holding the database's log files.
2. Check dmesg/journalctl for hardware or filesystem errors; run fsck if corruption is suspected.
3. Retry the tail request after remediation; the log file itself may be incomplete.
4. Keep log/data volumes on healthy local storage.

Example fix

// before
inner.sync_data().expect("error syncing data to disk");

// after: stream the (possibly not-yet-durable) tail rather than aborting
if let Err(e) = inner.sync_data() {
    log::warn!("tail: fsync failed, streaming unsynced tail: {}", e);
}

When it happens

Trigger: Subscribing with `tail(n, follow)` where n > 0 (e.g. `spacetime logs` or the HTTP log endpoint) while the log file's volume returns an fsync error such as ENOSPC or EIO.

Common situations: Operators tailing a database's logs when the node's disk has filled or the device is failing; log directory on a network mount with broken fsync semantics.

Related errors

• failed to flush segment upon rotation
• fsync failed
• failed to fsync file {}: {}
• log worker panicked
• never types are not yet supported in C# output

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/44a4001a74a8f632. Report an issue: GitHub.