quickwit-oss/quickwit · critical

tasks running the gRPC server should not panic or be cancell

Error message

tasks running the gRPC server should not panic or be cancelled

What it means

The gRPC server is spawned via spawn_named_task and awaited; the returned JoinHandle is expected to resolve to Ok because the server task must run for the node's lifetime. If the task panics or is cancelled, the JoinError surfaces here as a panic with a message pointing to the gRPC server task.

Source

Thrown at quickwit/quickwit-serve/src/lib.rs:1048

        "node_readiness_reporting",
    );

    let shutdown_handle = tokio::spawn(shutdown_signal_handler(
        shutdown_signal,
        universe,
        ingester_opt,
        ingester_decommission_timeout,
        compactor_supervisor_opt,
        compactor_decommission_timeout,
        grpc_shutdown_trigger_tx,
        rest_shutdown_trigger_tx,
        health_shutdown_trigger_tx_opt,
        cluster.clone(),
    ));
    let grpc_join_handle = async move {
        spawn_named_task(grpc_server, "grpc_server")
            .await
            .expect("tasks running the gRPC server should not panic or be cancelled")
            .context("gRPC server failed")
    };

    let rest_join_handle = async move {
        spawn_named_task(rest_server, "rest_server")
            .await
            .expect("tasks running the REST server should not panic or be cancelled")
            .context("REST server failed")
    };

    let health_join_handle = async move {
        spawn_named_task(health_server_fut, "health_server")
            .await
            .expect("tasks running the health check server should not panic or be cancelled")
            .context("health check server failed")
    };

    let chitchat_server_handle = cluster.chitchat_server_termination_watcher().await;

View on GitHub (pinned to a39730c5cd)

Solutions

1. Inspect logs for the underlying panic in the grpc_server task (this expect only re-raises it)
2. Verify gRPC listen address is not already bound (address-already-in-use)
3. Check TLS certificate/key paths in the server config
4. Ensure the runtime is not shutting down before the gRPC server completes startup

Example fix

// before
.expect("tasks running the gRPC server should not panic or be cancelled")
.context("gRPC server failed")
// after
match spawn_named_task(grpc_server, "grpc_server").await {
    Ok(Ok(())) => Ok(()),
    Ok(Err(e)) => Err(e).context("gRPC server failed"),
    Err(join_err) => Err(anyhow::Error::new(join_err).context("gRPC server task panicked or was cancelled")),
}

// Before startup, verify the gRPC port is free and TLS files exist:
std::net::TcpListener::bind(("0.0.0.0", grpc_port))?; // fails fast if address already in use
for p in [&tls_cert_path, &tls_key_path] { std::fs::metadata(p)?; }

// Supervise the server task:
tokio::select! {
    res = grpc_join_handle => log::error!("gRPC server ended: {res:?}"),
    _ = shutdown_rx => info!("shutting down gRPC server"),
}

When it happens

Trigger: The grpc_server task panics during startup (port already in use handled elsewhere, but e.g. TLS setup failure, service initialization panic) or its tokio task is cancelled; serve_quickwit's async block then expects on the join result.

Common situations: Bad gRPC/TLS configuration crashing the server task at boot; runtime shutdown racing server startup; OOM or panic inside a tonic service handler setup.

Related errors

• `index_uid` should be a required field
• node not found in pending
• OTP logs or traces do not support VRL transforms
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/e65f2d60b93ac8c1. Report an issue: GitHub.