{"record":{"id":"16634cdac6049fa0","repo":"tracel-ai/burn","slug":"event-store-thread-crashed-err","errorCode":null,"errorMessage":"Event store thread crashed: {err:?}","messagePattern":"Event store thread crashed: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/burn-train/src/metric/store/client.rs","lineNumber":71,"sourceCode":"        name: &str,\n        aggregate: Aggregate,\n        direction: Direction,\n        split: &Split,\n    ) -> Option<usize> {\n        let (sender, receiver) = mpsc::sync_channel(1);\n        self.sender\n            .send(Message::FindEpoch(\n                name.to_string(),\n                aggregate,\n                direction,\n                split.clone(),\n                sender,\n            ))\n            .expect(\"Can send event to event store thread.\");\n\n        match receiver.recv() {\n            Ok(value) => value,\n            Err(err) => panic!(\"Event store thread crashed: {err:?}\"),\n        }\n    }\n\n    /// Find the metric value for the current epoch following the given criteria.\n    pub fn find_metric(\n        &self,\n        name: &str,\n        epoch: usize,\n        aggregate: Aggregate,\n        split: &Split,\n    ) -> Option<f64> {\n        let (sender, receiver) = mpsc::sync_channel(1);\n        self.sender\n            .send(Message::FindMetric(\n                name.to_string(),\n                epoch,\n                aggregate,\n                split.clone(),","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-train/src/metric/store/client.rs#L53-L89","documentation":"`EventStoreClient::find_epoch` sends a `FindEpoch` message to the background event-store thread and blocks on a reply channel; if `receiver.recv()` returns `Err`, the worker thread has already terminated (its channel dropped), so Burn panics. This means the event store thread died (e.g. panicked handling a prior message) before answering this query — typically invoked from checkpointing logic deciding whether to keep the best epoch.","triggerScenarios":"Calling `find_epoch` (directly or via the checkpointer's `CheckpointingStrategy` evaluation) after the `EventStoreClient`'s worker thread has exited: the thread panicked while processing a metric event, or the client was used after `End`/drop lifecycle misuse. Note `receiver.recv()` returning `Err(RecvError)` here means the reply `SyncSender` was dropped without a response.","commonSituations":"A custom `EventStore` implementation (or a metric aggregator feeding it) that panics on a NaN/missing metric name, killing the worker thread; querying an epoch/metric name that triggers an unwrap in a user store; using the client from DDP secondary workers where the store lives on the main worker.","solutions":["Look for an earlier panic logged from the event-store thread (its panic output precedes this message) and fix that root cause","Check your custom `EventStore`/metric implementations for panics on missing metric names or non-finite values","Ensure `find_epoch` is only called while the training run (and thus the store thread) is alive, not after learner teardown","Test with the default `InMemoryMetricRepository` to confirm the failure comes from a custom store","Update burn; if the crash is reproducible with defaults, file an issue with a minimal repro"],"exampleFix":"// before\nstore.find_metric(\"loss\", Aggregate::Mean, &Split::Train); // name typo panics custom store, later find_epoch crashes\n// after\nstore.find_metric(\"Loss\", Aggregate::Mean, &Split::Train); // match the registered metric name exactly","handlingStrategy":"try-catch","validationCode":"// before calling learner APIs that query the store, validate the metric was actually logged\nfn metric_registered(logged: &[String], name: &str) -> bool { logged.iter().any(|n| n == name) }","typeGuard":null,"tryCatchPattern":"let epoch = std::panic::catch_unwind(std::panic::AssertUnwindSafe(||\n    client.find_epoch(\"Loss\", Aggregate::Mean, Direction::Low, &Split::Valid)\n));\nmatch epoch {\n    Ok(value) => value,\n    Err(_) => {\n        eprintln!(\"event store thread crashed; treating as no epoch found\");\n        None\n    }\n}","preventionTips":["Never panic/unwrap inside custom `EventStore` implementations — return `None` for missing metrics or epochs","Use the exact metric names registered by the metrics struct (check the generated key names, e.g. \"Loss\" vs \"loss\")","Ensure the store client is only queried while the learner/training run is active","Stress-test custom aggregators with NaN, empty series and out-of-range epochs","Keep an eye on earlier thread-panic output — this error is always secondary to a root panic in the store thread"],"tags":["concurrency","channels","panic","metric-store"],"backgroundTag":"channel-disconnected","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}