xai-org/x-algorithm · error · anyhow::Error

MH scan failed: {e}

Error message

MH scan failed: {e}

What it means

Raised in get_recent when the Manhattan scan operation fails at the transport/backend level. The request against the served-history table (pkey + range with SCAN_LIMIT, SoftDcReadMyWrites consistency) errored before returning rows, so recent served history cannot be read.

Source

Thrown at home-mixer/clients/served_history_client.rs:128

        client_platform: i32,
    ) -> Result<Vec<ServedHistory>> {
        let pkey = timeline_pkey(timeline_type, user_id, client_platform);
        let range = LkeySelector::Range {
            from: None,
            to: None,
        };

        let items = self
            .client
            .scan_with_consistency(
                self.tenant.clone(),
                pkey,
                range,
                SCAN_LIMIT,
                ConsistencyLevel::SoftDcReadMyWrites,
            )
            .await
            .map_err(|e| anyhow::anyhow!("MH scan failed: {e}"))?;

        let mut entries = Vec::with_capacity(items.len());
        for item in &items {
            let value = item.value();
            let bytes = value.as_bytes();
            match xai_x_thrift::deserialize_compact::<ServedHistory>(bytes) {
                Ok(mut history) => {
                    let lkey_parts = item.lkey().into_byte_vecs();
                    if let Some(part) = lkey_parts.first()
                        && part.len() >= 8
                    {
                        let inverted = i64::from_be_bytes(part[..8].try_into().unwrap());
                        history.served_time_ms = Some(i64::MAX - inverted);
                    }
                    entries.push(history);
                }
                Err(e) => {
                    tracing::warn!("Failed to deserialize ServedHistory: {e}");

View on GitHub (pinned to 24c60942c5)

Solutions

1. Retry with exponential backoff and jitter — MH scans commonly fail transiently
2. Verify the tenant/table exists and the service has read permissions
3. Check scan size: reduce SCAN_LIMIT or narrow the time range if the scan is too large
4. Inspect MH-side metrics/logs for throttling or node failures during the failure window

Example fix

// before
let entries = client.get_recent(...).await?;

// after
let entries = retry_with_backoff(|| client.get_recent(...), 3).await
    .map_err(|e| anyhow::anyhow!("get_recent failed for user {user_id}: {e}"))?;

null

let entries = backoff::future::retry(backoff::exponential(Duration::from_millis(50)), || async { client.get_recent(...).await.map_err(backoff::Error::transient) }).await?;

When it happens

Trigger: Calling get_recent(timeline_type, user_id, client_platform, ...) when the MH backend returns an error for the scan: network timeout, oversized scan, missing table/tenant permissions, or backend overload.

Common situations: MH server-side throttling; a range scan exceeding SCAN_LIMIT or shard limits; network blips between service and MH; tenant not provisioned for reads.

Related errors

• MH put failed: {e}
• Failed to create native MH client for served history: {e}
• MH delete failed: {e}
• timed out after {}s
• regex match being interrupted. Tmeout: %s.

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/27b3abc3dbdccf73. Report an issue: GitHub.