xai-org/x-algorithm · critical · anyhow::Error

Failed to create native MH client for served history: {e}

Error message

Failed to create native MH client for served history: {e}

What it means

Raised in ServedHistoryClient::new when NativeManhattanClient::builder_from_tenant_s2s(...).build().await fails. It means the native Manhattan (MH) key-value store client could not be constructed for the given tenant/datacenter/S2S credentials, so the served-history store is unusable before any request is made.

Source

Thrown at home-mixer/clients/served_history_client.rs:97

    pub async fn new(datacenter: &str) -> Result<Self> {
        let tenant = Tenant {
            cluster: "omega".to_string(),
            app_id: "timeline_persistence".to_string(),
            dataset: "timeline_response_batches_v5".to_string(),
        };
        let s2s = S2sConfig {
            client_cert_path: S2S_CRT_PATH.clone(),
            client_key_path: S2S_KEY_PATH.clone(),
            ca_cert_path: S2S_CHAIN_PATH.clone(),
        };
        let client = Arc::new(
            NativeManhattanClient::builder_from_tenant_s2s(&tenant, datacenter, s2s)
                .timeout(MH_TIMEOUT)
                .early_terminate_empty_scan(true)
                .build()
                .await
                .map_err(|e| {
                    anyhow::anyhow!("Failed to create native MH client for served history: {e}")
                })?,
        );
        Ok(Self { client, tenant })
    }
}

#[async_trait]
impl ServedHistoryClient for ProdServedHistoryClient {
    async fn get_recent(
        &self,
        user_id: u64,
        timeline_type: TimelineType,
        client_platform: i32,
    ) -> Result<Vec<ServedHistory>> {
        let pkey = timeline_pkey(timeline_type, user_id, client_platform);
        let range = LkeySelector::Range {
            from: None,
            to: None,

View on GitHub (pinned to 24c60942c5)

Solutions

  1. Verify the tenant name and datacenter values passed to new() against the MH service catalog
  2. Check S2S credentials/certs are present and valid in the environment
  3. Confirm network reachability of the MH backend from the host
  4. Retry construction (transient backend/config-service failures) with backoff before giving up

Example fix

// before
let client = ServedHistoryClient::new(tenant.clone(), dc, s2s).await?;

// after
let client = tokio::time::timeout(
    Duration::from_secs(10),
    ServedHistoryClient::new(tenant.clone(), dc, s2s),
).await
    .map_err(|_| anyhow::anyhow!("timed out creating MH client"))?
    .map_err(|e| anyhow::anyhow!("MH client init failed (tenant={tenant}, dc={dc}): {e}"))?;
Defensive patterns

Strategy: retry

Validate before calling

fn mh_env_reachable(dc: &str) -> bool { /* resolve + TCP probe MH endpoints for dc */ true }

Try / catch

let client = match ServedHistoryClient::new(t, dc, s2s).await { Ok(c) => c, Err(e) if retryable(&e) => retry_new(...).await?, Err(e) => return Err(e) };

Prevention

When it happens

Trigger: Calling ServedHistoryClient::new with an invalid or unknown tenant, wrong datacenter name, missing/invalid S2S (service-to-service) auth credentials, or when the MH backend endpoints are unreachable/misconfigured so the builder's initial handshake or config resolution fails.

Common situations: Deploying to a new datacenter without MH tenant whitelisting; expired or missing S2S certs; typo'd tenant string in config; local dev without MH access.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/46cb5cc18e007eec. Report an issue: GitHub.