{"record":{"id":"2c435377c9782eea","repo":"xai-org/grok-build","slug":"failed-to-build-shared-http-client","errorCode":null,"errorMessage":"failed to build shared HTTP client","messagePattern":"failed to build shared HTTP client","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-grok-http/src/lib.rs","lineNumber":300,"sourceCode":"/// Without the health checks reqwest reuses it for new streams, so every retry fails identically and a reachable server looks unreachable.\n/// Idle and TCP eviction drops connections before the upstream idle window (~60-100s; 30s is a conservative default) closes them.\n/// The HTTP/2 keepalive ping detects a dead connection so the pool stops handing it out.\npub fn shared_client() -> reqwest::Client {\n    static CLIENT: OnceLock<reqwest::Client> = OnceLock::new();\n    CLIENT\n        .get_or_init(|| {\n            let _timer = startup_timer!(\"startup.http_client_build\");\n            xai_grok_extra_ca::build_reqwest_client(|builder| {\n                builder\n                    .connect_timeout(std::time::Duration::from_secs(30))\n                    .user_agent(process_user_agent_string())\n                    .pool_idle_timeout(std::time::Duration::from_secs(30))\n                    .http2_keep_alive_interval(std::time::Duration::from_secs(20))\n                    .http2_keep_alive_timeout(std::time::Duration::from_secs(10))\n                    .http2_keep_alive_while_idle(true)\n                    .tcp_keepalive(std::time::Duration::from_secs(30))\n            })\n            .expect(\"failed to build shared HTTP client\")\n        })\n        .clone()\n}\n\n/// Wrap a raw client with [`AuthRetryMiddleware`] for automatic 401 retry.\npub fn with_auth_retry(\n    client: reqwest::Client,\n    credentials: std::sync::Arc<dyn xai_grok_auth::AuthCredentialProvider>,\n) -> reqwest_middleware::ClientWithMiddleware {\n    reqwest_middleware::ClientBuilder::new(client)\n        .with(xai_grok_auth::AuthRetryMiddleware::new(credentials, 1))\n        .build()\n}\n\n/// Returns a shared [`reqwest::Client`] for GCS uploads, creating it on first call.\n///\n/// Unlike `shared_client()`, this client has aggressive connection pool eviction to avoid reusing stale or poisoned connections during retry loops.\n/// When uploads fail and trigger exponential backoff (1s, 2s, 4s...), idle connections may be closed by the server, Cloudflare, or load balancers.","sourceCodeStart":282,"sourceCodeEnd":318,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-grok-http/src/lib.rs#L282-L318","documentation":"shared_client lazily builds the process-wide reqwest blocking Client (keepalive, pool tuning) inside a cached initializer and .expect()s the build. A panic here means reqwest could not construct the client - nearly always TLS backend initialization or provider/certificate setup failing, not a transient condition.","triggerScenarios":"First call to shared_client() (e.g. via send_with_retry_escaping_pool) when the rustls/aws-lc-rs provider fails to init, the extra root CA material cannot be loaded, or the reqwest builder rejects the configuration.","commonSituations":"Inconsistent rustls/crypto-provider feature flags; deployment image missing crypto prerequisites; a previously installed global CryptoProvider conflicting with this build; static/musl builds with a broken TLS backend.","solutions":["Verify xai-grok-extra-ca and rustls/aws-lc-rs features are consistent workspace-wide and rebuild (cargo clean if needed)","Check for a conflicting rustls::crypto::CryptoProvider::install_default earlier in the process","Pre-flight build the client once at startup and surface a clear error instead of relying on the expect","If this reproduces only in one environment, diff its TLS/crypto library availability against a working one"],"exampleFix":"// before\nlet resp = send_with_retry_escaping_pool(&req); // panics if shared_client build fails\n// after\nlet client = xai_grok_http::shared_client(); // pre-warm at startup with clear panic context\ntracing::info!(\"shared HTTP client ready\");\nlet resp = send_with_retry_escaping_pool(&req);","handlingStrategy":"try-catch","validationCode":"// pre-warm and convert the panic into a checkable failure\nlet client = std::panic::catch_unwind(xai_grok_http::shared_client)\n    .map_err(|_| anyhow!(\"shared HTTP client build failed (TLS init?)\"))?;","typeGuard":"null","tryCatchPattern":"std::panic::catch_unwind(|| send_with_retry_escaping_pool(&req))\n    .map_err(|_| anyhow!(\"HTTP stack unavailable: shared client build failed\"))?","preventionTips":["Pre-warm shared_client() at startup so failures surface immediately","Keep TLS/crypto feature flags uniform workspace-wide","Avoid installing a conflicting global rustls CryptoProvider","Test client construction in a minimal binary on the target image"],"tags":["http","tls","panics","reqwest","rust"],"backgroundTag":"tls-client-build-failed","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}