clockworklabs/SpacetimeDB · critical

log worker panicked

Error message

log worker panicked

What it means

Panic in SpacetimeDB's `DatabaseLogger` when sending a log record to the background log worker. `Cmd::Append` is pushed over an mpsc channel and `.expect("log worker panicked")` fires when the receiver has been dropped — i.e. the dedicated task that owns and writes the database log file has exited (almost always because it itself panicked) or the runtime shut down out of order.

Solutions

1. Look EARLIER in the panic output/logs: the real failure is the log worker's original panic (usually disk I/O) — fix that root cause (disk space, hardware, mount).
2. For shutdown races in tests/hosts, ensure all actors that can log are dropped before the runtime shuts down, or wrap logging in `std::panic::catch_unwind` at test boundaries.
3. Check disk health and free space on the database log directory, then restart the node.
4. If you embed SpacetimeDB as a library, consider using `request_snapshot_ignore_closed`-style tolerant send sites (ignore closed channels) for non-critical logging paths in your own code.

Example fix

// before: sending to a possibly-dead worker and panicking
self.cmd.send(Cmd::Append { .. }).expect("log worker panicked");

// after (defensive variant for non-critical paths): tolerate a closed worker
let _ = self.cmd.send(Cmd::Append { ts: record.ts, record: buf });

// If you wrap DatabaseLogger in your own facade, tolerate a dead worker
fn safe_write(&self, rec: impl Serialize) {
    let buf = serde_json::to_vec(&rec).unwrap_or_default();
    let _ = self.cmd.send(Cmd::Append { ts: rec_ts(&buf), record: Bytes::from(buf) }); // no expect
}

When it happens

Trigger: Any module/host code writing a log record (`logger.write`, logging from a reducer) after the log worker task has died. Typical root causes: the worker hit the 'fsync failed' panic (error 604) or a disk I/O panic while appending; or Tokio runtime shutdown dropped the worker task while loggers were still alive.

Common situations: Disk failure causing the worker's internal write/sync to panic; tests dropping the Tokio runtime before database actors; node shutdown races where reducers still log after the worker is cancelled. The panic then spreads to whichever task logs next, so the process dies with a confusing secondary panic.

Related errors

• snapshot worker panicked
• error syncing data to disk
• fsync failed
• Duration overflows i64 microseconds
• Duration since Unix epoch overflows i64 microseconds

AI-assisted analysis of clockworklabs/SpacetimeDB@6dee26c6ef (2026-08-20). Data as JSON: /api/errors/b60939dcbaa0d329. Report an issue: GitHub.

Appendix: source

Thrown at crates/core/src/database_logger.rs:380

            LogLevel::Info => LogEvent::Info(record),
            LogLevel::Debug => LogEvent::Debug(record),
            LogLevel::Trace => LogEvent::Trace(record),
            LogLevel::Panic => {
                trace = bt.capture();
                frames = trace.frames();
                LogEvent::Panic { record, trace: &frames }
            }
        };
        // TODO(perf): Reuse serialization buffer.
        let mut buf = serde_json::to_string(&event).unwrap();
        buf.push('\n');
        let buf = Bytes::from(buf);
        self.cmd
            .send(Cmd::Append {
                ts: record.ts,
                record: buf,
            })
            .expect("log worker panicked");
    }

    /// Stream the contents of this logger.
    ///
    /// If `n` is `Some`, only yield up to the last `n` lines in the log.
    /// If `follow` is `true`, the stream waits for new records to be appended
    /// to the log (via [Self::write]) and yields them as they become available.
    pub async fn tail(&self, n: Option<u32>, follow: bool) -> Result<LogStream, LoggerPanicked> {
        let (tx, rx) = oneshot::channel();
        self.cmd.send(Cmd::Tail { n, follow, reply: tx })?;
        Ok(rx.await?)
    }

    /// Read the most recent logs in `logs_dir`, up to `num_lines`.
    ///
    /// Note that this only reads from the most recent log file, even if it
    /// contains less than `num_lines` lines.
    ///

View on GitHub (pinned to 6dee26c6ef)