databendlabs/databend · error

Failed to push log element

Error message

Failed to push log element: {}

What it means

RemoteLog implements tracing_appender::Append; append() prepares a log element and pushes it into the shared in-memory buffer (buffer.log). If the buffer's channel rejects the element (send failure), the error is wrapped with this message. This means a log record could not be enqueued for remote shipping.

Solutions

1. Ensure the RemoteLog buffer and its background task outlive the tracing subscriber (drop the subscriber/guard first at shutdown).
2. Inspect the wrapped {} error to confirm channel-closed vs other cause.
3. Wrap or filter remote-logging failures so a dead sink doesn't fail every append (log-and-continue).
4. Restart the process if the collector task panicked.

Example fix

// before
self.buffer.log(log_element)
    .map_err(|e| anyhow::anyhow!("Failed to push log element: {}", e))?;
// after
if let Err(e) = self.buffer.log(log_element) {
    eprintln!("remote log dropped: {}", e); // do not fail the append
}

// Never fail record append on sink errors; degrade to stderr
if let Err(e) = self.buffer.log(log_element) {
    eprintln!("remote log unavailable, record dropped: {e}");
}

When it happens

Trigger: buffer.log fails when the internal async channel to the collector task is closed or full/unavailable — e.g. the remote log worker task has been dropped/shut down while logging continues, or the sender disconnected.

Common situations: Application shutdown ordering: the remote log buffer/worker stopped before the tracing subscriber; panic in the collector task; logging emitted during process teardown.

Related errors

• Failed to send log element
• Access denied: is outside allowed directories
• Addition resulted in NaN
• AggregateMeta does not support exchanging between multiple…
• async crash me panic

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/50f00f5224136152. Report an issue: GitHub.

Appendix: source

Thrown at src/common/tracing/src/remote_log.rs:246

            target,
            cluster_id: self.cluster_id.clone(),
            node_id: self.node_id.clone(),
            warehouse_id,
            query_id,
            log_level,
            message,
            fields,
        }
    }
}

impl Append for RemoteLog {
    fn append(&self, record: &Record, _diagnostics: &[Diagnostic]) -> anyhow::Result<()> {
        // fsync wal
        let log_element = self.prepare_log_element(record);
        self.buffer
            .log(log_element)
            .map_err(|e| anyhow::anyhow!("Failed to push log element: {}", e))?;

        Ok(())
    }

    fn flush(&self) {
        self.buffer.force_collect();
    }
}

impl LogBuffer {
    const MAX_BUFFER_SIZE: usize = 5000;

    pub fn new(sender: Sender<LogMessage>, interval: u64) -> Self {
        Self {
            queue: ConcurrentQueue::unbounded(),
            last_collect: AtomicU64::new(Timestamp::now().as_microsecond() as u64),
            sender,
            interval,

View on GitHub (pinned to 288d84d76e)