{"record":{"id":"e65f2d60b93ac8c1","repo":"quickwit-oss/quickwit","slug":"tasks-running-the-grpc-server-should-not-panic-or","errorCode":null,"errorMessage":"tasks running the gRPC server should not panic or be cancelled","messagePattern":"tasks running the gRPC server should not panic or be cancelled","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"quickwit/quickwit-serve/src/lib.rs","lineNumber":1048,"sourceCode":"        \"node_readiness_reporting\",\n    );\n\n    let shutdown_handle = tokio::spawn(shutdown_signal_handler(\n        shutdown_signal,\n        universe,\n        ingester_opt,\n        ingester_decommission_timeout,\n        compactor_supervisor_opt,\n        compactor_decommission_timeout,\n        grpc_shutdown_trigger_tx,\n        rest_shutdown_trigger_tx,\n        health_shutdown_trigger_tx_opt,\n        cluster.clone(),\n    ));\n    let grpc_join_handle = async move {\n        spawn_named_task(grpc_server, \"grpc_server\")\n            .await\n            .expect(\"tasks running the gRPC server should not panic or be cancelled\")\n            .context(\"gRPC server failed\")\n    };\n\n    let rest_join_handle = async move {\n        spawn_named_task(rest_server, \"rest_server\")\n            .await\n            .expect(\"tasks running the REST server should not panic or be cancelled\")\n            .context(\"REST server failed\")\n    };\n\n    let health_join_handle = async move {\n        spawn_named_task(health_server_fut, \"health_server\")\n            .await\n            .expect(\"tasks running the health check server should not panic or be cancelled\")\n            .context(\"health check server failed\")\n    };\n\n    let chitchat_server_handle = cluster.chitchat_server_termination_watcher().await;","sourceCodeStart":1030,"sourceCodeEnd":1066,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-serve/src/lib.rs#L1030-L1066","documentation":"The gRPC server is spawned via spawn_named_task and awaited; the returned JoinHandle is expected to resolve to Ok because the server task must run for the node's lifetime. If the task panics or is cancelled, the JoinError surfaces here as a panic with a message pointing to the gRPC server task.","triggerScenarios":"The grpc_server task panics during startup (port already in use handled elsewhere, but e.g. TLS setup failure, service initialization panic) or its tokio task is cancelled; serve_quickwit's async block then expects on the join result.","commonSituations":"Bad gRPC/TLS configuration crashing the server task at boot; runtime shutdown racing server startup; OOM or panic inside a tonic service handler setup.","solutions":["Inspect logs for the underlying panic in the grpc_server task (this expect only re-raises it)","Verify gRPC listen address is not already bound (address-already-in-use)","Check TLS certificate/key paths in the server config","Ensure the runtime is not shutting down before the gRPC server completes startup"],"exampleFix":"// before\n.expect(\"tasks running the gRPC server should not panic or be cancelled\")\n.context(\"gRPC server failed\")\n// after\nmatch spawn_named_task(grpc_server, \"grpc_server\").await {\n    Ok(Ok(())) => Ok(()),\n    Ok(Err(e)) => Err(e).context(\"gRPC server failed\"),\n    Err(join_err) => Err(anyhow::Error::new(join_err).context(\"gRPC server task panicked or was cancelled\")),\n}","handlingStrategy":"try-catch","validationCode":"// Before startup, verify the gRPC port is free and TLS files exist:\nstd::net::TcpListener::bind((\"0.0.0.0\", grpc_port))?; // fails fast if address already in use\nfor p in [&tls_cert_path, &tls_key_path] { std::fs::metadata(p)?; }","typeGuard":null,"tryCatchPattern":"// Supervise the server task:\ntokio::select! {\n    res = grpc_join_handle => log::error!(\"gRPC server ended: {res:?}\"),\n    _ = shutdown_rx => info!(\"shutting down gRPC server\"),\n}","preventionTips":["Pre-check listen address availability and TLS file paths before serve_quickwit","Watch node logs for panics in the grpc_server task, not just the final expect","Ensure the tokio runtime outlives the gRPC server task (no premature shutdown)","Smoke-test gRPC startup in CI with the same config as production"],"tags":["rust","grpc","panic","server","task-join"],"backgroundTag":"task-panicked","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}