{"record":{"id":"b60939dcbaa0d329","repo":"clockworklabs/SpacetimeDB","slug":"log-worker-panicked","errorCode":null,"errorMessage":"log worker panicked","messagePattern":"log worker panicked","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/core/src/database_logger.rs","lineNumber":380,"sourceCode":"            LogLevel::Info => LogEvent::Info(record),\n            LogLevel::Debug => LogEvent::Debug(record),\n            LogLevel::Trace => LogEvent::Trace(record),\n            LogLevel::Panic => {\n                trace = bt.capture();\n                frames = trace.frames();\n                LogEvent::Panic { record, trace: &frames }\n            }\n        };\n        // TODO(perf): Reuse serialization buffer.\n        let mut buf = serde_json::to_string(&event).unwrap();\n        buf.push('\\n');\n        let buf = Bytes::from(buf);\n        self.cmd\n            .send(Cmd::Append {\n                ts: record.ts,\n                record: buf,\n            })\n            .expect(\"log worker panicked\");\n    }\n\n    /// Stream the contents of this logger.\n    ///\n    /// If `n` is `Some`, only yield up to the last `n` lines in the log.\n    /// If `follow` is `true`, the stream waits for new records to be appended\n    /// to the log (via [Self::write]) and yields them as they become available.\n    pub async fn tail(&self, n: Option<u32>, follow: bool) -> Result<LogStream, LoggerPanicked> {\n        let (tx, rx) = oneshot::channel();\n        self.cmd.send(Cmd::Tail { n, follow, reply: tx })?;\n        Ok(rx.await?)\n    }\n\n    /// Read the most recent logs in `logs_dir`, up to `num_lines`.\n    ///\n    /// Note that this only reads from the most recent log file, even if it\n    /// contains less than `num_lines` lines.\n    ///","sourceCodeStart":362,"sourceCodeEnd":398,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/6dee26c6efc2856793e12b148a59742964f5d783/crates/core/src/database_logger.rs#L362-L398","documentation":"Panic in SpacetimeDB's `DatabaseLogger` when sending a log record to the background log worker. `Cmd::Append` is pushed over an mpsc channel and `.expect(\"log worker panicked\")` fires when the receiver has been dropped — i.e. the dedicated task that owns and writes the database log file has exited (almost always because it itself panicked) or the runtime shut down out of order.","triggerScenarios":"Any module/host code writing a log record (`logger.write`, logging from a reducer) after the log worker task has died. Typical root causes: the worker hit the 'fsync failed' panic (error 604) or a disk I/O panic while appending; or Tokio runtime shutdown dropped the worker task while loggers were still alive.","commonSituations":"Disk failure causing the worker's internal write/sync to panic; tests dropping the Tokio runtime before database actors; node shutdown races where reducers still log after the worker is cancelled. The panic then spreads to whichever task logs next, so the process dies with a confusing secondary panic.","solutions":["Look EARLIER in the panic output/logs: the real failure is the log worker's original panic (usually disk I/O) — fix that root cause (disk space, hardware, mount).","For shutdown races in tests/hosts, ensure all actors that can log are dropped before the runtime shuts down, or wrap logging in `std::panic::catch_unwind` at test boundaries.","Check disk health and free space on the database log directory, then restart the node.","If you embed SpacetimeDB as a library, consider using `request_snapshot_ignore_closed`-style tolerant send sites (ignore closed channels) for non-critical logging paths in your own code."],"exampleFix":"// before: sending to a possibly-dead worker and panicking\nself.cmd.send(Cmd::Append { .. }).expect(\"log worker panicked\");\n\n// after (defensive variant for non-critical paths): tolerate a closed worker\nlet _ = self.cmd.send(Cmd::Append { ts: record.ts, record: buf });","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// If you wrap DatabaseLogger in your own facade, tolerate a dead worker\nfn safe_write(&self, rec: impl Serialize) {\n    let buf = serde_json::to_vec(&rec).unwrap_or_default();\n    let _ = self.cmd.send(Cmd::Append { ts: rec_ts(&buf), record: Bytes::from(buf) }); // no expect\n}","preventionTips":["Hunt for the worker's ORIGINAL panic earlier in logs — this error is always secondary.","In tests, drop all actors that log before shutting down the Tokio runtime.","Keep the log directory healthy (space/permissions); most worker deaths are I/O panics."],"tags":["rust","spacetimedb","logging","channel-closed","worker-panic","tokio","panic"],"backgroundTag":"background-worker-crashed","analyzedSha":"6dee26c6efc2856793e12b148a59742964f5d783","analyzedAt":"2026-08-20T06:08:37.179Z","contentChangedAt":"2026-08-20T06:08:37.179Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}