clockworklabs/SpacetimeDB · error

log worker panicked

Error message

log worker panicked

What it means

`DatabaseLogger::write` forwards each serialized log record over an mpsc channel to a dedicated worker task; `.expect("log worker panicked")` fires when `send` fails, which only happens once the receiving worker has exited. In practice the worker itself panicked earlier (look for the first panic in stderr) or the logger is being torn down while a write is still in flight.

Source

Thrown at crates/core/src/database_logger.rs:380

            LogLevel::Info => LogEvent::Info(record),
            LogLevel::Debug => LogEvent::Debug(record),
            LogLevel::Trace => LogEvent::Trace(record),
            LogLevel::Panic => {
                trace = bt.capture();
                frames = trace.frames();
                LogEvent::Panic { record, trace: &frames }
            }
        };
        // TODO(perf): Reuse serialization buffer.
        let mut buf = serde_json::to_string(&event).unwrap();
        buf.push('\n');
        let buf = Bytes::from(buf);
        self.cmd
            .send(Cmd::Append {
                ts: record.ts,
                record: buf,
            })
            .expect("log worker panicked");
    }

    /// Stream the contents of this logger.
    ///
    /// If `n` is `Some`, only yield up to the last `n` lines in the log.
    /// If `follow` is `true`, the stream waits for new records to be appended
    /// to the log (via [Self::write]) and yields them as they become available.
    pub async fn tail(&self, n: Option<u32>, follow: bool) -> Result<LogStream, LoggerPanicked> {
        let (tx, rx) = oneshot::channel();
        self.cmd.send(Cmd::Tail { n, follow, reply: tx })?;
        Ok(rx.await?)
    }

    /// Read the most recent logs in `logs_dir`, up to `num_lines`.
    ///
    /// Note that this only reads from the most recent log file, even if it
    /// contains less than `num_lines` lines.
    ///

View on GitHub (pinned to 524b4487d9)

Solutions

1. Find the FIRST panic in process output — the worker's original error explains why the channel closed; this expect is only the downstream symptom.
2. Fix that root cause (commonly the fsync/disk error or serialization failure that killed the worker).
3. Ensure module hosts stop producing log records before the logger/runtime is dropped during shutdown.
4. Upgrade spacetimedb if a known worker-panic bug exists for your version.

Example fix

// before
self.cmd.send(Cmd::Append { ts: record.ts, record: buf }).expect("log worker panicked");

// after: tolerate a dead worker during teardown instead of aborting
if self.cmd.send(Cmd::Append { ts: record.ts, record: buf }).is_err() {
    eprintln!("[database_logger] worker unavailable: {}", String::from_utf8_lossy(&buf));
}

let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| logger.write(record))); if r.is_err() { /* worker died: find the original panic in stderr, stop logging to this sink */ }

When it happens

Trigger: Calling `logger.write(...)` (or any module log path) after the worker task terminated — the worker hit an fsync or serialization panic, or the DatabaseLogger/runtime is dropped concurrently with a write during shutdown.

Common situations: Shutdown races while stopping spacetimedb where module log writes race logger teardown; an earlier disk error killed the worker; tests dropping the tokio runtime while modules still emit logs.

Related errors

• error syncing data to disk
• never types are not yet supported in C# output
• unions not supported
• must specify table accessor, e.g. `#[spacetimedb::table(acce
• a direct index must be paired with a `#[unique] constraint

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/b60939dcbaa0d329. Report an issue: GitHub.