What causes "MH put failed: {e}" in xai-org/x-algorithm?

Retry the put with backoff (writes are idempotent here since lkey is deterministic per served_time)

xai-org/x-algorithm · error · anyhow::Error

MH put failed: {e}

Error message

MH put failed: {e}

What it means

Raised in put when the Manhattan put RPC fails after successful serialization. The write of (pkey, lkey=(i64::MAX - served_time_ms), value) to the served-history table was rejected or errored at the backend.

Source

Thrown at home-mixer/clients/served_history_client.rs:170

    }

    async fn put(
        &self,
        user_id: u64,
        timeline_type: TimelineType,
        client_platform: i32,
        served_time_ms: i64,
        history: &ServedHistory,
    ) -> Result<()> {
        let pkey = timeline_pkey(timeline_type, user_id, client_platform);
        let lkey = vec![(i64::MAX - served_time_ms).to_be_bytes().to_vec()];
        let value = xai_x_thrift::serialize_compact(history)
            .map_err(|e| anyhow::anyhow!("failed to serialize ServedHistory: {e}"))?;

        self.client
            .put(self.tenant.clone(), pkey, lkey, value)
            .await
            .map_err(|e| anyhow::anyhow!("MH put failed: {e}"))
    }

    async fn delete(
        &self,
        user_id: u64,
        timeline_type: TimelineType,
        client_platform: i32,
        served_times_ms: &[i64],
    ) -> Result<()> {
        let pkey = timeline_pkey(timeline_type, user_id, client_platform);
        for &ts in served_times_ms {
            let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];
            self.client
                .delete(self.tenant.clone(), pkey.clone(), lkey)
                .await
                .map_err(|e| anyhow::anyhow!("MH delete failed: {e}"))?;
        }
        Ok(())

View on GitHub (pinned to 24c60942c5)

Solutions

Retry the put with backoff (writes are idempotent here since lkey is deterministic per served_time)
Check MH tenant write quotas and request limit increases if throttling persists
Verify network connectivity and S2S auth for write path
If persistent, drop the write and log — served history is best-effort cache data; failing the request path may be worse than skipping the write

Example fix

// before
self.client.put(self.tenant.clone(), pkey, lkey, value).await
    .map_err(|e| anyhow::anyhow!("MH put failed: {e}"))

// after
match self.client.put(self.tenant.clone(), pkey, lkey, value).await {
    Ok(()) => Ok(()),
    Err(e) if is_retryable(&e) => {
        tokio::time::sleep(Duration::from_millis(200)).await;
        self.client.put(self.tenant.clone(), pkey, lkey, value).await
            .map_err(|e| anyhow::anyhow!("MH put failed after retry: {e}"))
    }
    Err(e) => Err(anyhow::anyhow!("MH put failed: {e}")),
}

Defensive patterns

Strategy: retry

Validate before calling

null

Try / catch

if let Err(e) = client.put(...).await { if is_retryable(&e) { retry_with_jitter(...).await? } else { log_and_degrade(); } }

Prevention

Treat served-history writes as best-effort: log and continue on persistent failure
Idempotent lkeys make blind retries safe
Watch MH write-error metrics

When it happens

Trigger: Calling put() during MH backend errors: write throttling, network failures, permission denial on the tenant, or unavailable MH shards for the pkey.

Common situations: Heavy write volume triggering MH rate limits; tenant not provisioned for writes; transient network issues in writes; deployment to a region without MH write access.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/9adc624210abd3e6. Report an issue: GitHub.