{"record":{"id":"27b3abc3dbdccf73","repo":"xai-org/x-algorithm","slug":"mh-scan-failed-e","errorCode":null,"errorMessage":"MH scan failed: {e}","messagePattern":"MH scan failed: (.+?)","errorType":"error_code","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"home-mixer/clients/served_history_client.rs","lineNumber":128,"sourceCode":"        client_platform: i32,\n    ) -> Result<Vec<ServedHistory>> {\n        let pkey = timeline_pkey(timeline_type, user_id, client_platform);\n        let range = LkeySelector::Range {\n            from: None,\n            to: None,\n        };\n\n        let items = self\n            .client\n            .scan_with_consistency(\n                self.tenant.clone(),\n                pkey,\n                range,\n                SCAN_LIMIT,\n                ConsistencyLevel::SoftDcReadMyWrites,\n            )\n            .await\n            .map_err(|e| anyhow::anyhow!(\"MH scan failed: {e}\"))?;\n\n        let mut entries = Vec::with_capacity(items.len());\n        for item in &items {\n            let value = item.value();\n            let bytes = value.as_bytes();\n            match xai_x_thrift::deserialize_compact::<ServedHistory>(bytes) {\n                Ok(mut history) => {\n                    let lkey_parts = item.lkey().into_byte_vecs();\n                    if let Some(part) = lkey_parts.first()\n                        && part.len() >= 8\n                    {\n                        let inverted = i64::from_be_bytes(part[..8].try_into().unwrap());\n                        history.served_time_ms = Some(i64::MAX - inverted);\n                    }\n                    entries.push(history);\n                }\n                Err(e) => {\n                    tracing::warn!(\"Failed to deserialize ServedHistory: {e}\");","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/home-mixer/clients/served_history_client.rs#L110-L146","documentation":"Raised in get_recent when the Manhattan scan operation fails at the transport/backend level. The request against the served-history table (pkey + range with SCAN_LIMIT, SoftDcReadMyWrites consistency) errored before returning rows, so recent served history cannot be read.","triggerScenarios":"Calling get_recent(timeline_type, user_id, client_platform, ...) when the MH backend returns an error for the scan: network timeout, oversized scan, missing table/tenant permissions, or backend overload.","commonSituations":"MH server-side throttling; a range scan exceeding SCAN_LIMIT or shard limits; network blips between service and MH; tenant not provisioned for reads.","solutions":["Retry with exponential backoff and jitter — MH scans commonly fail transiently","Verify the tenant/table exists and the service has read permissions","Check scan size: reduce SCAN_LIMIT or narrow the time range if the scan is too large","Inspect MH-side metrics/logs for throttling or node failures during the failure window"],"exampleFix":"// before\nlet entries = client.get_recent(...).await?;\n\n// after\nlet entries = retry_with_backoff(|| client.get_recent(...), 3).await\n    .map_err(|e| anyhow::anyhow!(\"get_recent failed for user {user_id}: {e}\"))?;","handlingStrategy":"retry","validationCode":"null","typeGuard":null,"tryCatchPattern":"let entries = backoff::future::retry(backoff::exponential(Duration::from_millis(50)), || async { client.get_recent(...).await.map_err(backoff::Error::transient) }).await?;","preventionTips":["Set conservative SCAN_LIMIT matching MH shard limits","Circuit-break scans when error rate is high","Degrade to empty history on failure — reads are best-effort"],"tags":["manhattan","scan","timeout","rust","retryable"],"backgroundTag":"kv-store-scan-failed","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}