{"record":{"id":"7ab8546c15d46358","repo":"xai-org/x-algorithm","slug":"mh-delete-failed-e","errorCode":null,"errorMessage":"MH delete failed: {e}","messagePattern":"MH delete failed: (.+?)","errorType":"error_code","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"home-mixer/clients/served_history_client.rs","lineNumber":186,"sourceCode":"            .put(self.tenant.clone(), pkey, lkey, value)\n            .await\n            .map_err(|e| anyhow::anyhow!(\"MH put failed: {e}\"))\n    }\n\n    async fn delete(\n        &self,\n        user_id: u64,\n        timeline_type: TimelineType,\n        client_platform: i32,\n        served_times_ms: &[i64],\n    ) -> Result<()> {\n        let pkey = timeline_pkey(timeline_type, user_id, client_platform);\n        for &ts in served_times_ms {\n            let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];\n            self.client\n                .delete(self.tenant.clone(), pkey.clone(), lkey)\n                .await\n                .map_err(|e| anyhow::anyhow!(\"MH delete failed: {e}\"))?;\n        }\n        Ok(())\n    }\n}\n\npub struct MockServedHistoryClient;\n\n#[async_trait]\nimpl ServedHistoryClient for MockServedHistoryClient {\n    async fn get_recent(\n        &self,\n        _user_id: u64,\n        _timeline_type: TimelineType,\n        _client_platform: i32,\n    ) -> Result<Vec<ServedHistory>> {\n        Ok(vec![])\n    }\n","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/home-mixer/clients/served_history_client.rs#L168-L204","documentation":"Raised in delete when a Manhattan delete RPC fails for one of the provided served_times_ms timestamps (lkey = i64::MAX - ts). The loop aborts on the first failing delete, so later timestamps in the batch are not attempted.","triggerScenarios":"Calling delete(user_id, timeline_type, client_platform, served_times_ms) where any single delete hits a backend error: throttling, network failure, or missing delete permissions; earlier deletes in the loop may have succeeded.","commonSituations":"Batch cleanup jobs deleting many entries hitting rate limits; partial-delete inconsistency when one timestamp fails mid-loop; permission gaps for deletes on the tenant.","solutions":["Retry the whole batch or just the remaining timestamps with backoff","Collect per-timestamp errors instead of aborting on first failure, then retry only failures","Check MH delete quotas/permissions for the tenant","Ensure callers treat delete as eventually-consistent cleanup, tolerating partial completion"],"exampleFix":"// before\nfor &ts in served_times_ms {\n    let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];\n    self.client.delete(self.tenant.clone(), pkey.clone(), lkey).await\n        .map_err(|e| anyhow::anyhow!(\"MH delete failed: {e}\"))?;\n}\n\n// after\nlet mut failures = Vec::new();\nfor &ts in served_times_ms {\n    let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];\n    if let Err(e) = self.client.delete(self.tenant.clone(), pkey.clone(), lkey).await {\n        failures.push((ts, e));\n    }\n}\nif !failures.is_empty() {\n    anyhow::bail!(\"MH delete failed for {} of {} timestamps\", failures.len(), served_times_ms.len());\n}\nOk(())","handlingStrategy":"retry","validationCode":"null","typeGuard":null,"tryCatchPattern":"Collect per-timestamp errors; retry only failures; surface aggregate if some remain.","preventionTips":["Chunk large delete batches to stay under quotas","Schedule cleanups off-peak","Make delete idempotent-safe (missing keys are OK)"],"tags":["manhattan","delete","partial-failure","rust"],"backgroundTag":"kv-store-delete-failed","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}