BoundaryML/baml · error

BatchProcessor worker thread did not finish in time

Error message

BatchProcessor worker thread did not finish in time

What it means

In `BatchProcessor::flush` (threaded_tracer.rs:213), the worker thread did not drain the batch queue within the timeout loop (polling every 100ms). BAML bails instead of blocking indefinitely, which means buffered trace events may not have been submitted.

Solutions

  1. Check network reachability and latency of the trace collector endpoint; fix connectivity or increase its capacity.
  2. Ensure the log event callback is fast and non-blocking; move slow work off the worker thread.
  3. Retry flush after the endpoint recovers; some buffered events may still drain.
  4. Reduce event volume or batch sizes so the worker can drain within the timeout.
  5. Investigate deadlocks: no callback should acquire locks held elsewhere while flush is waiting.

Example fix

// before: blocking callback stalls the worker
let callback = |event| { reqwest::blocking::post(url).body(event).send().unwrap(); };
// after: enqueue locally, submit asynchronously
let callback = |event| { queue.lock().unwrap().push(event); };
Defensive patterns

Strategy: retry

Validate before calling

// preflight: collector must be reachable before relying on flush
let reachable = std::net::TcpStream::connect_timeout(
    &collector_addr, Duration::from_secs(2)).is_ok();
if !reachable { log::warn!("trace collector unreachable; flush may time out"); }

Try / catch

if let Err(e) = batch_processor.flush() {
    if e.to_string().contains("did not finish in time") {
        log::warn!("trace flush timed out; events may be lost: {e}");
        // optionally retry once after a delay
    } else {
        return Err(e);
    }
}

Prevention

When it happens

Trigger: Calling `flush` on the threaded tracer when the worker thread is stalled: the event callback or HTTP submission is blocked on a slow/hung network call to the collector, the queue keeps receiving events faster than they drain, or the worker thread panicked/deadlocked on a lock.

Common situations: Process shutdown with an unreachable or very slow BAML log endpoint; large burst of trace events at exit exceeding drain time; a log_event_callback that blocks; lock contention on log_event_callback or the queue.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of BoundaryML/baml@bd85ce9dee (2026-09-12). Data as JSON: /api/errors/03f7cad5397eafcf. Report an issue: GitHub.

Appendix: source

Thrown at engine/baml-runtime/src/tracing/threaded_tracer.rs:213

        let flush_start = Instant::now();

        while flush_start.elapsed() < Duration::from_secs(60) {
            {
                match *self.stop_rx.borrow() {
                    ProcessorStatus::Active => {}
                    ProcessorStatus::Done(r_id) if r_id >= id => {
                        return Ok(());
                    }
                    ProcessorStatus::Done(id) => {
                        // Old flush, ignore
                    }
                }
            }
            std::thread::sleep(Duration::from_millis(100));
        }

        anyhow::bail!("BatchProcessor worker thread did not finish in time")
    }

    pub fn set_log_event_callback(&self, log_event_callback: Option<LogEventCallbackSync>) {
        // Get a mutable lock on the log_event_callback
        let mut callback_lock = self.log_event_callback.lock().unwrap();

        *callback_lock = log_event_callback;
    }

    pub fn submit(&self, mut event: LogSchema) -> Result<()> {
        let callback = self.log_event_callback.lock().unwrap();
        if let Some(ref callback) = *callback {
            let event = event.clone();
            let llm_output_model = event.metadata.as_ref().and_then(|m| match m {
                MetadataType::Single(llm_event) => Some(llm_event),
                // take the last element in the vector
                MetadataType::Multi(llm_events) => llm_events.last(),
            });

View on GitHub (pinned to bd85ce9dee)