{"record":{"id":"3fcf90cf28af6051","repo":"databendlabs/databend","slug":"create-client-future-must-be-joined-successfully","errorCode":null,"errorMessage":"create client future must be joined successfully","messagePattern":"create client future must be joined successfully","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"src/query/service/src/servers/flight/v1/exchange/exchange_manager.rs","lineNumber":164,"sourceCode":"                FlightOperation::Connect,\n                &local_node_id,\n                &remote_node_id,\n            )\n        })?;\n\n        Ok(FlightClient::new(\n            FlightServiceClient::new(channel),\n            local_node_id,\n            remote_node_id,\n        ))\n    };\n    if use_current_rt {\n        task.await\n    } else {\n        GlobalIORuntime::instance()\n            .spawn(task)\n            .await\n            .expect(\"create client future must be joined successfully\")\n    }\n}\n\npub struct DataExchangeManager {\n    queries_coordinator: ReentrantMutex<SyncUnsafeCell<HashMap<String, QueryCoordinator>>>,\n}\n\nimpl DataExchangeManager {\n    pub fn init() -> Result<()> {\n        GlobalInstance::set(Arc::new(DataExchangeManager {\n            queries_coordinator: ReentrantMutex::new(SyncUnsafeCell::new(HashMap::new())),\n        }));\n\n        Ok(())\n    }\n\n    pub fn instance() -> Arc<DataExchangeManager> {\n        GlobalInstance::get()","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/query/service/src/servers/flight/v1/exchange/exchange_manager.rs#L146-L182","documentation":"In `create_flight_client`, a flight-client-creation task is either awaited on the current runtime or spawned on `GlobalIORuntime` and joined; the `expect(\"create client future must be joined successfully\")` panics if the JoinHandle resolves to a JoinError, meaning the spawned task was cancelled or panicked before producing a Result.","triggerScenarios":"Calling `init_query_env` -> `create_flight_client` with `use_current_rt == false` while the GlobalIORuntime is shutting down or its worker dropped the task, or the client-building future itself panicked (panic then surfaces as this expect at the join point).","commonSituations":"Node shutdown concurrent with query env initialization; GlobalIORuntime misconfigured/too-small and killed; a panic inside flight client construction (bad flight address config) being masked as a join failure.","solutions":["Check logs above the panic for a flight-client panic or runtime shutdown message; fix the root cause first.","Ensure GlobalIORuntime is initialized and not being torn down while queries start (check shutdown ordering).","Validate the flight/API address configuration (`flight_api_address`) — malformed addresses can make client construction panic.","Patch to convert JoinError into ErrorCode instead of expecting, so shutdown races yield a query error rather than an abort."],"exampleFix":"// before\n.expect(\"create client future must be joined successfully\")\n// after\n.map_err(|e| ErrorCode::Internal(format!(\"create client future join failed: {e}\")))??","handlingStrategy":"try-catch","validationCode":"// before init: ensure runtime is alive\nlet _ = GlobalIORuntime::instance(); // panics early if runtime not initialized","typeGuard":"fn runtime_reachable() -> bool { GlobalIORuntime::instance().inner.try_lock().is_ok() }","tryCatchPattern":"// wrap join result instead of expect\nmatch handle.await {\n    Ok(res) => res,\n    Err(e) => return Err(ErrorCode::Internal(format!(\"flight client task join failed: {e}\"))),\n}","preventionTips":["Do not initialize query envs during node shutdown","Validate flight_api_address config before startup","Check logs for panics inside client-construction futures"],"tags":["rust","tokio","flight","panic","runtime-shutdown"],"backgroundTag":"internal-invariant-violation","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}