BoundaryML/baml · error

Blob flush timed out after {:?}

Error message

Blob flush timed out after {:?}

What it means

Raised in flush_blob_uploader_channel (called by flush) when waiting for the blob uploader's acknowledgment (blob_ack_rx) exceeds timeout_duration. The publisher flushes binary blobs (images/files attached to trace events) through a dedicated channel; this error means the blob uploader did not confirm completion within the flush timeout. The flush is abandoned, so those blobs may not have been uploaded.

Source

Thrown at engine/baml-runtime/src/tracingv2/publisher/publisher.rs:1115

async fn flush_blob_uploader_channel(timeout_duration: Duration) -> anyhow::Result<()> {
    let Some(blob_tx) = BLOB_UPLOADER_CHANNEL.get() else {
        return Ok(());
    };

    let (blob_ack_tx, blob_ack_rx) = tokio::sync::oneshot::channel();
    blob_tx
        .send(BlobUploaderMessage::Flush(blob_ack_tx))
        .await
        .map_err(|e| anyhow::anyhow!(e.to_string()))?;

    match timeout(timeout_duration, blob_ack_rx).await {
        Ok(Ok(())) => {
            log::debug!("Flush blob uploader completed");
            Ok(())
        }
        Ok(Err(e)) => Err(e.into()),
        Err(_) => Err(anyhow::anyhow!(
            "Blob flush timed out after {:?}",
            timeout_duration
        )),
    }
}

pub fn publish_trace_event(event: Arc<TraceEventWithMeta>) -> anyhow::Result<()> {
    let Some(channel) = ensure_publisher_started() else {
        return Ok(());
    };
    match channel.try_send(PublisherMessage::Trace(event)) {
        Ok(()) => Ok(()),
        Err(mpsc::error::TrySendError::Full(_)) => {
            log::warn!(
                "Trace event queue is full (max 4 batches). Dropping trace event. \
                Consider increasing BAML_TRACE_BATCH_SIZE or reducing trace volume."
            );
            Ok(())

View on GitHub (pinned to bd85ce9dee)

Solutions

1. Increase the flush timeout duration.
2. Reduce blob backlog before flushing (publish blobs earlier, or fewer/smaller attachments).
3. Check logs for the blob uploader worker crashing (Ok(Err(e)) branch shows its error).
4. Verify network connectivity to blob storage (S3).

Example fix

// before
publisher.flush(Duration::from_secs(5)).await?;
// after
publisher.flush(Duration::from_secs(60)).await?; // allow backlog to drain

// before flushing, ensure blob backlog is manageable
if publisher.pending_blob_count() > 100 {
    log::warn!("large blob backlog; extending flush timeout");
}
let timeout = Duration::from_secs(max(60, publisher.pending_blob_count() as u64));

fn is_blob_flush_timeout(e: &anyhow::Error) -> bool {
    e.to_string().starts_with("Blob flush timed out")
}

match publisher.flush(Duration::from_secs(60)).await {
    Err(e) if e.to_string().contains("Blob flush timed out") => {
        log::warn!("blob flush incomplete; blobs may be lost");
    }
    Err(e) => log::error!("flush failed: {e}"),
    Ok(_) => {}
}

When it happens

Trigger: flush() was called and the blob uploader worker did not send its ack within the timeout: the uploader is stuck on a slow/hung S3 PUT, the channel has a large backlog of blobs, or the worker task has died.

Common situations: Many image attachments queued during a long session flushed at process exit, slow network to blob storage, blob uploader task crashed earlier leaving no ack, or flush timeout too short for the backlog.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Flush timed out after {:?}
• Request timed out after {0:?}
• Trace publisher channel is closed
• baml.panics.Cancelled
• timeout: {message}

AI-assisted analysis of BoundaryML/baml@bd85ce9dee (2026-09-12). Data as JSON: /api/errors/b3306674e8827617. Report an issue: GitHub.