BoundaryML/baml · error

Flush timed out after {:?}

Error message

Flush timed out after {:?}

What it means

Raised in flush (public) when waiting for the trace publisher worker's completion ack (ack_rx) exceeds timeout_duration. flush() sends a sentinel and waits for all queued batches to be published; this error means the worker did not finish draining the queue in time, so some trace events may not have reached the collector.

Source

Thrown at engine/baml-runtime/src/tracingv2/publisher/publisher.rs:1163

    log::debug!("Flushing traces [rust]");
    // Set a timeout to avoid waiting indefinitely.
    let timeout_duration = Duration::from_secs(30);

    // First try to flush the trace publisher (which should also flush blobs internally)
    let mut publisher_result: Option<anyhow::Result<()>> = None;
    if let Some(channel) = ensure_publisher_started() {
        let (ack_tx, ack_rx) = tokio::sync::oneshot::channel();
        let send_res = channel
            .send(PublisherMessage::Flush(ack_tx))
            .await
            .map_err(|e| anyhow::anyhow!(e.to_string()));
        if let Err(e) = send_res {
            publisher_result = Some(Err(e));
        } else {
            publisher_result = Some(match timeout(timeout_duration, ack_rx).await {
                Ok(Ok(())) => Ok(()),
                Ok(Err(e)) => Err(e.into()),
                Err(_) => Err(anyhow::anyhow!(
                    "Flush timed out after {:?}",
                    timeout_duration
                )),
            });
        }
    } else {
        log::debug!("No publish channel found [rust]");
    }

    // Always flush the blob uploader explicitly as well to guarantee no leftovers
    log::debug!("Flushing blob uploader [rust]");
    let blob_result = flush_blob_uploader_channel(timeout_duration).await;
    log::debug!("Flushing blob uploader [rust] completed");
    // Prefer reporting blob uploader errors if any; otherwise propagate publisher errors
    blob_result?;

    if let Some(Err(e)) = publisher_result {
        return Err(e);

View on GitHub (pinned to bd85ce9dee)

Solutions

1. Increase the flush timeout passed to flush().
2. Flush periodically during execution instead of once with a big backlog at exit.
3. Verify collector latency/availability; retry flush after a backoff.
4. Reduce trace volume (BAML_TRACE_BATCH_SIZE / fewer events) if backlog is chronic.

Example fix

// before
await publisher.flush(5)
// after
await publisher.flush(60)  # allow queued batches to drain on slow links

// estimate needed timeout from backlog
const BATCH: usize = 50;
let est = Duration::from_secs((publisher.pending_events() / BATCH + 2) as u64);
let timeout = est.max(Duration::from_secs(30));

fn is_flush_timeout(e: &anyhow::Error) -> bool {
    e.to_string().starts_with("Flush timed out")
}

match publisher.flush(Duration::from_secs(60)).await {
    Err(e) if e.to_string().starts_with("Flush timed out") => {
        log::warn!("flush incomplete; some trace events may be lost");
        // optionally retry once with a longer timeout
    }
    Err(e) => log::error!("flush failed: {e}"),
    Ok(_) => {}
}

When it happens

Trigger: flush() called while the publisher still holds up to 4 full batches of events and the network upload is slower than the timeout: slow collector responses, high trace volume, or a timeout set too low (e.g. seconds before process exit).

Common situations: Flushing at process shutdown with a large accumulated trace backlog, slow/unresponsive BAML collector, brief network outage during flush, or a very short flush timeout in serverless/Lambda environments.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Blob flush timed out after {:?}
• Request timed out after {0:?}
• Trace publisher channel is closed
• baml.panics.Cancelled
• timeout: {message}

AI-assisted analysis of BoundaryML/baml@bd85ce9dee (2026-09-12). Data as JSON: /api/errors/79c75db46d2d26eb. Report an issue: GitHub.