{"record":{"id":"b3306674e8827617","repo":"BoundaryML/baml","slug":"blob-flush-timed-out-after","errorCode":null,"errorMessage":"Blob flush timed out after {:?}","messagePattern":"Blob flush timed out after (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"engine/baml-runtime/src/tracingv2/publisher/publisher.rs","lineNumber":1115,"sourceCode":"\nasync fn flush_blob_uploader_channel(timeout_duration: Duration) -> anyhow::Result<()> {\n    let Some(blob_tx) = BLOB_UPLOADER_CHANNEL.get() else {\n        return Ok(());\n    };\n\n    let (blob_ack_tx, blob_ack_rx) = tokio::sync::oneshot::channel();\n    blob_tx\n        .send(BlobUploaderMessage::Flush(blob_ack_tx))\n        .await\n        .map_err(|e| anyhow::anyhow!(e.to_string()))?;\n\n    match timeout(timeout_duration, blob_ack_rx).await {\n        Ok(Ok(())) => {\n            log::debug!(\"Flush blob uploader completed\");\n            Ok(())\n        }\n        Ok(Err(e)) => Err(e.into()),\n        Err(_) => Err(anyhow::anyhow!(\n            \"Blob flush timed out after {:?}\",\n            timeout_duration\n        )),\n    }\n}\n\npub fn publish_trace_event(event: Arc<TraceEventWithMeta>) -> anyhow::Result<()> {\n    let Some(channel) = ensure_publisher_started() else {\n        return Ok(());\n    };\n    match channel.try_send(PublisherMessage::Trace(event)) {\n        Ok(()) => Ok(()),\n        Err(mpsc::error::TrySendError::Full(_)) => {\n            log::warn!(\n                \"Trace event queue is full (max 4 batches). Dropping trace event. \\\n                Consider increasing BAML_TRACE_BATCH_SIZE or reducing trace volume.\"\n            );\n            Ok(())","sourceCodeStart":1097,"sourceCodeEnd":1133,"githubUrl":"https://github.com/BoundaryML/baml/blob/bd85ce9dee1463ff04d27efd20531013a4ff46c1/engine/baml-runtime/src/tracingv2/publisher/publisher.rs#L1097-L1133","documentation":"Raised in flush_blob_uploader_channel (called by flush) when waiting for the blob uploader's acknowledgment (blob_ack_rx) exceeds timeout_duration. The publisher flushes binary blobs (images/files attached to trace events) through a dedicated channel; this error means the blob uploader did not confirm completion within the flush timeout. The flush is abandoned, so those blobs may not have been uploaded.","triggerScenarios":"flush() was called and the blob uploader worker did not send its ack within the timeout: the uploader is stuck on a slow/hung S3 PUT, the channel has a large backlog of blobs, or the worker task has died.","commonSituations":"Many image attachments queued during a long session flushed at process exit, slow network to blob storage, blob uploader task crashed earlier leaving no ack, or flush timeout too short for the backlog.","solutions":["Increase the flush timeout duration.","Reduce blob backlog before flushing (publish blobs earlier, or fewer/smaller attachments).","Check logs for the blob uploader worker crashing (Ok(Err(e)) branch shows its error).","Verify network connectivity to blob storage (S3)."],"exampleFix":"// before\npublisher.flush(Duration::from_secs(5)).await?;\n// after\npublisher.flush(Duration::from_secs(60)).await?; // allow backlog to drain","handlingStrategy":"retry","validationCode":"// before flushing, ensure blob backlog is manageable\nif publisher.pending_blob_count() > 100 {\n    log::warn!(\"large blob backlog; extending flush timeout\");\n}\nlet timeout = Duration::from_secs(max(60, publisher.pending_blob_count() as u64));","typeGuard":"fn is_blob_flush_timeout(e: &anyhow::Error) -> bool {\n    e.to_string().starts_with(\"Blob flush timed out\")\n}","tryCatchPattern":"match publisher.flush(Duration::from_secs(60)).await {\n    Err(e) if e.to_string().contains(\"Blob flush timed out\") => {\n        log::warn!(\"blob flush incomplete; blobs may be lost\");\n    }\n    Err(e) => log::error!(\"flush failed: {e}\"),\n    Ok(_) => {}\n}","preventionTips":["Publish blobs incrementally instead of deferring to one final flush","Scale flush timeout with the size of the blob backlog","Monitor the blob uploader task health in logs","Verify network throughput to blob storage from CI/prod"],"tags":["timeout","async","baml","tracing"],"backgroundTag":"request-timeout","analyzedSha":"bd85ce9dee1463ff04d27efd20531013a4ff46c1","analyzedAt":"2026-09-12T03:38:25.718Z","contentChangedAt":"2026-09-12T03:38:25.718Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}