{"record":{"id":"d1ef01c31d56b329","repo":"nautechsystems/nautilus_trader","slug":"failed-to-terminate-databento-tasks-e","errorCode":null,"errorMessage":"Failed to terminate Databento tasks: {e}","messagePattern":"Failed to terminate Databento tasks: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/databento/src/data.rs","lineNumber":500,"sourceCode":"        self.clear_feed_channels();\n        self.abort_active_tasks();\n        self.is_connected.store(false, Ordering::Relaxed);\n        Ok(())\n    }\n\n    fn dispose(&mut self) -> anyhow::Result<()> {\n        log::debug!(\"Disposing\");\n        self.stop()\n    }\n\n    async fn connect(&mut self) -> anyhow::Result<()> {\n        log::debug!(\"Connecting...\");\n\n        if !self.task_handles.is_open() {\n            self.task_handles\n                .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n                .await\n                .map_err(|e| anyhow::anyhow!(\"Failed to terminate Databento tasks: {e}\"))?;\n            self.task_handles\n                .start_generation()\n                .map_err(|e| anyhow::anyhow!(\"Failed to start Databento task generation: {e}\"))?;\n            self.cancellation_token = self.task_handles.cancellation_token();\n        }\n\n        self.is_connected.store(true, Ordering::Relaxed);\n\n        log::info!(\"Connected\");\n        Ok(())\n    }\n\n    async fn disconnect(&mut self) -> anyhow::Result<()> {\n        log::debug!(\"Disconnecting...\");\n\n        self.send_close_to_active_feeds();\n        self.clear_feed_channels();\n        self.task_handles.begin_shutdown();","sourceCodeStart":482,"sourceCodeEnd":518,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/databento/src/data.rs#L482-L518","documentation":"During `connect`, any leftover task generation from a previous session must be shut down via `task_handles.finish_shutdown` before a new generation starts. If terminating the old tasks fails (tasks not finishing within the given timeouts), connect aborts with this error.","triggerScenarios":"Calling connect() while prior Databento tasks are still alive and fail to terminate within 1s graceful / 2s total shutdown windows.","commonSituations":"Rapid reconnect cycles where tasks are blocked on slow network I/O, wedged live-stream tasks stuck in a read, or calling connect() concurrently from two callers.","solutions":["Increase the shutdown timeout durations before reconnecting","Ensure disconnect() is called before a subsequent connect() so shutdown starts cleanly","Investigate why old tasks hang (e.g. blocked network reads) and add cancellation points","Retry connect() after a short delay once tasks have drained"],"exampleFix":"// before\nclient.connect().await?; // immediately after a previous session\n// after\nclient.disconnect().await?;\ntokio::time::sleep(Duration::from_secs(1)).await;\nclient.connect().await?;","handlingStrategy":"retry","validationCode":"async fn clean_reconnect(client: &DatabentoDataClient) -> anyhow::Result<()> {\n    client.disconnect().await.ok();\n    tokio::time::sleep(Duration::from_millis(500)).await;\n    client.connect().await\n}","typeGuard":null,"tryCatchPattern":"for attempt in 1..=3 {\n    match client.connect().await {\n        Ok(()) => break,\n        Err(e) if e.to_string().contains(\"Failed to terminate Databento tasks\") && attempt < 3 => {\n            tokio::time::sleep(Duration::from_secs(attempt)).await;\n        }\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Call disconnect() before reconnecting to begin shutdown early","Serialize connect()/disconnect() calls; never run them concurrently","Allow cool-down time between reconnect cycles so old tasks can drain"],"tags":["databento","connect","task-shutdown","async"],"backgroundTag":"request-timeout","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}