What causes "Failed to send log element: {}" in databendlabs/databend?

Check the wrapped error to confirm the channel is disconnected (SendError).

Failed to send log element

Error message

Failed to send log element: {}

What it means

collect() drains the queued log elements and sends a LogMessage::Flush over sender using send_blocking. If the receiver end (the remote log flusher task) is gone or the blocking send fails, this error is returned. It indicates accumulated log elements could not be delivered for remote storage.

Solutions

Check the wrapped error to confirm the channel is disconnected (SendError).
Keep the consumer task alive until after a final collect()/force_collect() during graceful shutdown.
Use force_collect() with its completion channel to await the flush and detect shutdown ordering issues.
Drop the log pipeline before or after all producers deterministically, matching the lifecycle of the subscriber.

Example fix

// before
self.sender.send_blocking(LogMessage::Flush(...))
    .map_err(|e| anyhow::anyhow!("Failed to send log element: {}", e))?;
// after
match self.sender.send_blocking(LogMessage::Flush(...)) {
    Ok(()) => (),
    Err(e) if e.is_disconnected() => eprintln!("remote log consumer gone; dropping {} elements", self.queue.len()),
    Err(e) => return Err(anyhow::anyhow!("Failed to send log element: {}", e)),
}

Defensive patterns

Strategy: try-catch

Try / catch

match collector.collect() {
    Ok(()) => (),
    Err(e) => eprintln!("flush failed, {} elements lost: {e}", queue_len),
}

Prevention

Perform a final force_collect()/collect() during graceful shutdown before dropping the consumer
Join the flusher task before process exit
Bound the queue and alert on persistent send failures

When it happens

Trigger: Calling collect() (directly or via log -> force_collect path) after the receiving task was dropped or its channel closed; send_blocking failing during shutdown or when the runtime worker panicked.

Common situations: Tearing down the tracing/remote-log pipeline while flush is still in flight; background task aborted due to runtime shutdown; queue backpressure combined with a dead consumer.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/2eaa445028221689. Report an issue: GitHub.

Appendix: source

Thrown at src/common/tracing/src/remote_log.rs:309

                    }
                    self.collect()?;
                    break;
                }
                Err(last_collect) => {
                    if now - self.interval < last_collect {
                        break;
                    }
                    current_last_collect = last_collect;
                }
            }
        }
        Ok(())
    }

    pub fn collect(&self) -> anyhow::Result<()> {
        self.sender
            .send_blocking(LogMessage::Flush(Vec::from_iter(self.queue.try_iter())))
            .map_err(|e| anyhow::anyhow!("Failed to send log element: {}", e))?;
        Ok(())
    }

    pub fn force_collect(&self) {
        let (tx, rx) = channel();
        if self
            .sender
            .send_blocking(LogMessage::ForceFlush(
                Vec::from_iter(self.queue.try_iter()),
                tx,
            ))
            .is_ok()
        {
            let _ = rx.recv_timeout(Duration::from_secs(3));
        }
    }
}

View on GitHub (pinned to 288d84d76e)