tracel-ai/burn · error

Event store thread crashed: {err:?}

Error message

Event store thread crashed: {err:?}

What it means

`EventStoreClient::find_epoch` sends a `FindEpoch` message to the background event-store thread and blocks on a reply channel; if `receiver.recv()` returns `Err`, the worker thread has already terminated (its channel dropped), so Burn panics. This means the event store thread died (e.g. panicked handling a prior message) before answering this query — typically invoked from checkpointing logic deciding whether to keep the best epoch.

Source

Thrown at crates/burn-train/src/metric/store/client.rs:71

        name: &str,
        aggregate: Aggregate,
        direction: Direction,
        split: &Split,
    ) -> Option<usize> {
        let (sender, receiver) = mpsc::sync_channel(1);
        self.sender
            .send(Message::FindEpoch(
                name.to_string(),
                aggregate,
                direction,
                split.clone(),
                sender,
            ))
            .expect("Can send event to event store thread.");

        match receiver.recv() {
            Ok(value) => value,
            Err(err) => panic!("Event store thread crashed: {err:?}"),
        }
    }

    /// Find the metric value for the current epoch following the given criteria.
    pub fn find_metric(
        &self,
        name: &str,
        epoch: usize,
        aggregate: Aggregate,
        split: &Split,
    ) -> Option<f64> {
        let (sender, receiver) = mpsc::sync_channel(1);
        self.sender
            .send(Message::FindMetric(
                name.to_string(),
                epoch,
                aggregate,
                split.clone(),

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Look for an earlier panic logged from the event-store thread (its panic output precedes this message) and fix that root cause
2. Check your custom `EventStore`/metric implementations for panics on missing metric names or non-finite values
3. Ensure `find_epoch` is only called while the training run (and thus the store thread) is alive, not after learner teardown
4. Test with the default `InMemoryMetricRepository` to confirm the failure comes from a custom store
5. Update burn; if the crash is reproducible with defaults, file an issue with a minimal repro

Example fix

// before
store.find_metric("loss", Aggregate::Mean, &Split::Train); // name typo panics custom store, later find_epoch crashes
// after
store.find_metric("Loss", Aggregate::Mean, &Split::Train); // match the registered metric name exactly

// before calling learner APIs that query the store, validate the metric was actually logged
fn metric_registered(logged: &[String], name: &str) -> bool { logged.iter().any(|n| n == name) }

let epoch = std::panic::catch_unwind(std::panic::AssertUnwindSafe(||
    client.find_epoch("Loss", Aggregate::Mean, Direction::Low, &Split::Valid)
));
match epoch {
    Ok(value) => value,
    Err(_) => {
        eprintln!("event store thread crashed; treating as no epoch found");
        None
    }
}

When it happens

Trigger: Calling `find_epoch` (directly or via the checkpointer's `CheckpointingStrategy` evaluation) after the `EventStoreClient`'s worker thread has exited: the thread panicked while processing a metric event, or the client was used after `End`/drop lifecycle misuse. Note `receiver.recv()` returning `Err(RecvError)` here means the reply `SyncSender` was dropped without a response.

Common situations: A custom `EventStore` implementation (or a metric aggregator feeding it) that panics on a NaN/missing metric name, killing the worker thread; querying an epoch/metric name that triggers an unwrap in a user store; using the client from DDP secondary workers where the store lives on the main worker.

Related errors

• Gradient sync server disconnected.
• Can receive callback
• Autobatcher should be able to send resulting actions.
• Autobatcher should be able to send resulting probabilities.
• SVD fallback failed: {err}

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/16634cdac6049fa0. Report an issue: GitHub.