{"record":{"id":"50f00f5224136152","repo":"databendlabs/databend","slug":"failed-to-push-log-element","errorCode":null,"errorMessage":"Failed to push log element: {}","messagePattern":"Failed to push log element: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/common/tracing/src/remote_log.rs","lineNumber":246,"sourceCode":"            target,\n            cluster_id: self.cluster_id.clone(),\n            node_id: self.node_id.clone(),\n            warehouse_id,\n            query_id,\n            log_level,\n            message,\n            fields,\n        }\n    }\n}\n\nimpl Append for RemoteLog {\n    fn append(&self, record: &Record, _diagnostics: &[Diagnostic]) -> anyhow::Result<()> {\n        // fsync wal\n        let log_element = self.prepare_log_element(record);\n        self.buffer\n            .log(log_element)\n            .map_err(|e| anyhow::anyhow!(\"Failed to push log element: {}\", e))?;\n\n        Ok(())\n    }\n\n    fn flush(&self) {\n        self.buffer.force_collect();\n    }\n}\n\nimpl LogBuffer {\n    const MAX_BUFFER_SIZE: usize = 5000;\n\n    pub fn new(sender: Sender<LogMessage>, interval: u64) -> Self {\n        Self {\n            queue: ConcurrentQueue::unbounded(),\n            last_collect: AtomicU64::new(Timestamp::now().as_microsecond() as u64),\n            sender,\n            interval,","sourceCodeStart":228,"sourceCodeEnd":264,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/common/tracing/src/remote_log.rs#L228-L264","documentation":"RemoteLog implements tracing_appender::Append; append() prepares a log element and pushes it into the shared in-memory buffer (buffer.log). If the buffer's channel rejects the element (send failure), the error is wrapped with this message. This means a log record could not be enqueued for remote shipping.","triggerScenarios":"buffer.log fails when the internal async channel to the collector task is closed or full/unavailable — e.g. the remote log worker task has been dropped/shut down while logging continues, or the sender disconnected.","commonSituations":"Application shutdown ordering: the remote log buffer/worker stopped before the tracing subscriber; panic in the collector task; logging emitted during process teardown.","solutions":["Ensure the RemoteLog buffer and its background task outlive the tracing subscriber (drop the subscriber/guard first at shutdown).","Inspect the wrapped {} error to confirm channel-closed vs other cause.","Wrap or filter remote-logging failures so a dead sink doesn't fail every append (log-and-continue).","Restart the process if the collector task panicked."],"exampleFix":"// before\nself.buffer.log(log_element)\n    .map_err(|e| anyhow::anyhow!(\"Failed to push log element: {}\", e))?;\n// after\nif let Err(e) = self.buffer.log(log_element) {\n    eprintln!(\"remote log dropped: {}\", e); // do not fail the append\n}","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Never fail record append on sink errors; degrade to stderr\nif let Err(e) = self.buffer.log(log_element) {\n    eprintln!(\"remote log unavailable, record dropped: {e}\");\n}","preventionTips":["Keep the remote-log worker task alive for the whole subscriber lifetime","Shut down the tracing subscriber/guard before the log buffer","Monitor worker-task panics so a dead sink is detected early"],"tags":["rust","tracing","logging"],"backgroundTag":"broken-pipe","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}