{"record":{"id":"46cb5cc18e007eec","repo":"xai-org/x-algorithm","slug":"failed-to-create-native-mh-client-for-served-histo","errorCode":null,"errorMessage":"Failed to create native MH client for served history: {e}","messagePattern":"Failed to create native MH client for served history: (.+?)","errorType":"error_code","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"home-mixer/clients/served_history_client.rs","lineNumber":97,"sourceCode":"    pub async fn new(datacenter: &str) -> Result<Self> {\n        let tenant = Tenant {\n            cluster: \"omega\".to_string(),\n            app_id: \"timeline_persistence\".to_string(),\n            dataset: \"timeline_response_batches_v5\".to_string(),\n        };\n        let s2s = S2sConfig {\n            client_cert_path: S2S_CRT_PATH.clone(),\n            client_key_path: S2S_KEY_PATH.clone(),\n            ca_cert_path: S2S_CHAIN_PATH.clone(),\n        };\n        let client = Arc::new(\n            NativeManhattanClient::builder_from_tenant_s2s(&tenant, datacenter, s2s)\n                .timeout(MH_TIMEOUT)\n                .early_terminate_empty_scan(true)\n                .build()\n                .await\n                .map_err(|e| {\n                    anyhow::anyhow!(\"Failed to create native MH client for served history: {e}\")\n                })?,\n        );\n        Ok(Self { client, tenant })\n    }\n}\n\n#[async_trait]\nimpl ServedHistoryClient for ProdServedHistoryClient {\n    async fn get_recent(\n        &self,\n        user_id: u64,\n        timeline_type: TimelineType,\n        client_platform: i32,\n    ) -> Result<Vec<ServedHistory>> {\n        let pkey = timeline_pkey(timeline_type, user_id, client_platform);\n        let range = LkeySelector::Range {\n            from: None,\n            to: None,","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/home-mixer/clients/served_history_client.rs#L79-L115","documentation":"Raised in ServedHistoryClient::new when NativeManhattanClient::builder_from_tenant_s2s(...).build().await fails. It means the native Manhattan (MH) key-value store client could not be constructed for the given tenant/datacenter/S2S credentials, so the served-history store is unusable before any request is made.","triggerScenarios":"Calling ServedHistoryClient::new with an invalid or unknown tenant, wrong datacenter name, missing/invalid S2S (service-to-service) auth credentials, or when the MH backend endpoints are unreachable/misconfigured so the builder's initial handshake or config resolution fails.","commonSituations":"Deploying to a new datacenter without MH tenant whitelisting; expired or missing S2S certs; typo'd tenant string in config; local dev without MH access.","solutions":["Verify the tenant name and datacenter values passed to new() against the MH service catalog","Check S2S credentials/certs are present and valid in the environment","Confirm network reachability of the MH backend from the host","Retry construction (transient backend/config-service failures) with backoff before giving up"],"exampleFix":"// before\nlet client = ServedHistoryClient::new(tenant.clone(), dc, s2s).await?;\n\n// after\nlet client = tokio::time::timeout(\n    Duration::from_secs(10),\n    ServedHistoryClient::new(tenant.clone(), dc, s2s),\n).await\n    .map_err(|_| anyhow::anyhow!(\"timed out creating MH client\"))?\n    .map_err(|e| anyhow::anyhow!(\"MH client init failed (tenant={tenant}, dc={dc}): {e}\"))?;","handlingStrategy":"retry","validationCode":"fn mh_env_reachable(dc: &str) -> bool { /* resolve + TCP probe MH endpoints for dc */ true }","typeGuard":null,"tryCatchPattern":"let client = match ServedHistoryClient::new(t, dc, s2s).await { Ok(c) => c, Err(e) if retryable(&e) => retry_new(...).await?, Err(e) => return Err(e) };","preventionTips":["Validate tenant/datacenter names at config load time","Run a startup readiness probe that constructs the MH client before serving traffic","Alert on repeated client-construction failures"],"tags":["manhattan","kv-store","client-init","rust","backend-unavailable"],"backgroundTag":"backend-client-init-failed","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}