What causes "create client future must be joined successfully" in databendlabs/databend?

Check logs above the panic for a flight-client panic or runtime shutdown message; fix the root cause first.

create client future must be joined successfully

Error message

create client future must be joined successfully

What it means

In `create_flight_client`, a flight-client-creation task is either awaited on the current runtime or spawned on `GlobalIORuntime` and joined; the `expect("create client future must be joined successfully")` panics if the JoinHandle resolves to a JoinError, meaning the spawned task was cancelled or panicked before producing a Result.

Solutions

Check logs above the panic for a flight-client panic or runtime shutdown message; fix the root cause first.
Ensure GlobalIORuntime is initialized and not being torn down while queries start (check shutdown ordering).
Validate the flight/API address configuration (`flight_api_address`) — malformed addresses can make client construction panic.
Patch to convert JoinError into ErrorCode instead of expecting, so shutdown races yield a query error rather than an abort.

Example fix

// before
.expect("create client future must be joined successfully")
// after
.map_err(|e| ErrorCode::Internal(format!("create client future join failed: {e}")))??

Defensive patterns

Strategy: try-catch

Validate before calling

// before init: ensure runtime is alive
let _ = GlobalIORuntime::instance(); // panics early if runtime not initialized

Type guard

fn runtime_reachable() -> bool { GlobalIORuntime::instance().inner.try_lock().is_ok() }

Try / catch

// wrap join result instead of expect
match handle.await {
    Ok(res) => res,
    Err(e) => return Err(ErrorCode::Internal(format!("flight client task join failed: {e}"))),
}

Prevention

Do not initialize query envs during node shutdown
Validate flight_api_address config before startup
Check logs for panics inside client-construction futures

When it happens

Trigger: Calling `init_query_env` -> `create_flight_client` with `use_current_rt == false` while the GlobalIORuntime is shutting down or its worker dropped the task, or the client-building future itself panicked (panic then surfaces as this expect at the join point).

Common situations: Node shutdown concurrent with query env initialization; GlobalIORuntime misconfigured/too-small and killed; a panic inside flight client construction (bad flight address config) being masked as a join failure.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/3fcf90cf28af6051. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/servers/flight/v1/exchange/exchange_manager.rs:164

                FlightOperation::Connect,
                &local_node_id,
                &remote_node_id,
            )
        })?;

        Ok(FlightClient::new(
            FlightServiceClient::new(channel),
            local_node_id,
            remote_node_id,
        ))
    };
    if use_current_rt {
        task.await
    } else {
        GlobalIORuntime::instance()
            .spawn(task)
            .await
            .expect("create client future must be joined successfully")
    }
}

pub struct DataExchangeManager {
    queries_coordinator: ReentrantMutex<SyncUnsafeCell<HashMap<String, QueryCoordinator>>>,
}

impl DataExchangeManager {
    pub fn init() -> Result<()> {
        GlobalInstance::set(Arc::new(DataExchangeManager {
            queries_coordinator: ReentrantMutex::new(SyncUnsafeCell::new(HashMap::new())),
        }));

        Ok(())
    }

    pub fn instance() -> Arc<DataExchangeManager> {
        GlobalInstance::get()

View on GitHub (pinned to 288d84d76e)