{"record":{"id":"9adc624210abd3e6","repo":"xai-org/x-algorithm","slug":"mh-put-failed-e","errorCode":null,"errorMessage":"MH put failed: {e}","messagePattern":"MH put failed: (.+?)","errorType":"error_code","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"home-mixer/clients/served_history_client.rs","lineNumber":170,"sourceCode":"    }\n\n    async fn put(\n        &self,\n        user_id: u64,\n        timeline_type: TimelineType,\n        client_platform: i32,\n        served_time_ms: i64,\n        history: &ServedHistory,\n    ) -> Result<()> {\n        let pkey = timeline_pkey(timeline_type, user_id, client_platform);\n        let lkey = vec![(i64::MAX - served_time_ms).to_be_bytes().to_vec()];\n        let value = xai_x_thrift::serialize_compact(history)\n            .map_err(|e| anyhow::anyhow!(\"failed to serialize ServedHistory: {e}\"))?;\n\n        self.client\n            .put(self.tenant.clone(), pkey, lkey, value)\n            .await\n            .map_err(|e| anyhow::anyhow!(\"MH put failed: {e}\"))\n    }\n\n    async fn delete(\n        &self,\n        user_id: u64,\n        timeline_type: TimelineType,\n        client_platform: i32,\n        served_times_ms: &[i64],\n    ) -> Result<()> {\n        let pkey = timeline_pkey(timeline_type, user_id, client_platform);\n        for &ts in served_times_ms {\n            let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];\n            self.client\n                .delete(self.tenant.clone(), pkey.clone(), lkey)\n                .await\n                .map_err(|e| anyhow::anyhow!(\"MH delete failed: {e}\"))?;\n        }\n        Ok(())","sourceCodeStart":152,"sourceCodeEnd":188,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/home-mixer/clients/served_history_client.rs#L152-L188","documentation":"Raised in put when the Manhattan put RPC fails after successful serialization. The write of (pkey, lkey=(i64::MAX - served_time_ms), value) to the served-history table was rejected or errored at the backend.","triggerScenarios":"Calling put() during MH backend errors: write throttling, network failures, permission denial on the tenant, or unavailable MH shards for the pkey.","commonSituations":"Heavy write volume triggering MH rate limits; tenant not provisioned for writes; transient network issues in writes; deployment to a region without MH write access.","solutions":["Retry the put with backoff (writes are idempotent here since lkey is deterministic per served_time)","Check MH tenant write quotas and request limit increases if throttling persists","Verify network connectivity and S2S auth for write path","If persistent, drop the write and log — served history is best-effort cache data; failing the request path may be worse than skipping the write"],"exampleFix":"// before\nself.client.put(self.tenant.clone(), pkey, lkey, value).await\n    .map_err(|e| anyhow::anyhow!(\"MH put failed: {e}\"))\n\n// after\nmatch self.client.put(self.tenant.clone(), pkey, lkey, value).await {\n    Ok(()) => Ok(()),\n    Err(e) if is_retryable(&e) => {\n        tokio::time::sleep(Duration::from_millis(200)).await;\n        self.client.put(self.tenant.clone(), pkey, lkey, value).await\n            .map_err(|e| anyhow::anyhow!(\"MH put failed after retry: {e}\"))\n    }\n    Err(e) => Err(anyhow::anyhow!(\"MH put failed: {e}\")),\n}","handlingStrategy":"retry","validationCode":"null","typeGuard":null,"tryCatchPattern":"if let Err(e) = client.put(...).await { if is_retryable(&e) { retry_with_jitter(...).await? } else { log_and_degrade(); } }","preventionTips":["Treat served-history writes as best-effort: log and continue on persistent failure","Idempotent lkeys make blind retries safe","Watch MH write-error metrics"],"tags":["manhattan","write","retryable","rust"],"backgroundTag":"kv-store-write-failed","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}