What causes "Flush timed out after {:?}" in BoundaryML/baml?

Increase the flush timeout passed to flush().

Flush timed out after

Error message

Flush timed out after {:?}

What it means

Raised in flush (public) when waiting for the trace publisher worker's completion ack (ack_rx) exceeds timeout_duration. flush() sends a sentinel and waits for all queued batches to be published; this error means the worker did not finish draining the queue in time, so some trace events may not have reached the collector.

Solutions

Increase the flush timeout passed to flush().
Flush periodically during execution instead of once with a big backlog at exit.
Verify collector latency/availability; retry flush after a backoff.
Reduce trace volume (BAML_TRACE_BATCH_SIZE / fewer events) if backlog is chronic.

Example fix

// before
await publisher.flush(5)
// after
await publisher.flush(60)  # allow queued batches to drain on slow links

Defensive patterns

Strategy: try-catch

Validate before calling

// estimate needed timeout from backlog
const BATCH: usize = 50;
let est = Duration::from_secs((publisher.pending_events() / BATCH + 2) as u64);
let timeout = est.max(Duration::from_secs(30));

Type guard

fn is_flush_timeout(e: &anyhow::Error) -> bool {
    e.to_string().starts_with("Flush timed out")
}

Try / catch

match publisher.flush(Duration::from_secs(60)).await {
    Err(e) if e.to_string().starts_with("Flush timed out") => {
        log::warn!("flush incomplete; some trace events may be lost");
        // optionally retry once with a longer timeout
    }
    Err(e) => log::error!("flush failed: {e}"),
    Ok(_) => {}
}

Prevention

Use generous timeouts (>=30s) for shutdown flushes
Flush periodically during long-running jobs to limit backlog
Tune BAML_TRACE_BATCH_SIZE for your trace volume
Monitor collector latency; retry flush on timeout

When it happens

Trigger: flush() called while the publisher still holds up to 4 full batches of events and the network upload is slower than the timeout: slow collector responses, high trace volume, or a timeout set too low (e.g. seconds before process exit).

Common situations: Flushing at process shutdown with a large accumulated trace backlog, slow/unresponsive BAML collector, brief network outage during flush, or a very short flush timeout in serverless/Lambda environments.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of BoundaryML/baml@bd85ce9dee (2026-09-12). Data as JSON: /api/errors/79c75db46d2d26eb. Report an issue: GitHub.

Appendix: source

Thrown at engine/baml-runtime/src/tracingv2/publisher/publisher.rs:1163

    log::debug!("Flushing traces [rust]");
    // Set a timeout to avoid waiting indefinitely.
    let timeout_duration = Duration::from_secs(30);

    // First try to flush the trace publisher (which should also flush blobs internally)
    let mut publisher_result: Option<anyhow::Result<()>> = None;
    if let Some(channel) = ensure_publisher_started() {
        let (ack_tx, ack_rx) = tokio::sync::oneshot::channel();
        let send_res = channel
            .send(PublisherMessage::Flush(ack_tx))
            .await
            .map_err(|e| anyhow::anyhow!(e.to_string()));
        if let Err(e) = send_res {
            publisher_result = Some(Err(e));
        } else {
            publisher_result = Some(match timeout(timeout_duration, ack_rx).await {
                Ok(Ok(())) => Ok(()),
                Ok(Err(e)) => Err(e.into()),
                Err(_) => Err(anyhow::anyhow!(
                    "Flush timed out after {:?}",
                    timeout_duration
                )),
            });
        }
    } else {
        log::debug!("No publish channel found [rust]");
    }

    // Always flush the blob uploader explicitly as well to guarantee no leftovers
    log::debug!("Flushing blob uploader [rust]");
    let blob_result = flush_blob_uploader_channel(timeout_duration).await;
    log::debug!("Flushing blob uploader [rust] completed");
    // Prefer reporting blob uploader errors if any; otherwise propagate publisher errors
    blob_result?;

    if let Some(Err(e)) = publisher_result {
        return Err(e);

View on GitHub (pinned to bd85ce9dee)