xai-org/x-algorithm · error · anyhow::Error

MH delete failed: {e}

Error message

MH delete failed: {e}

What it means

Raised in delete when a Manhattan delete RPC fails for one of the provided served_times_ms timestamps (lkey = i64::MAX - ts). The loop aborts on the first failing delete, so later timestamps in the batch are not attempted.

Source

Thrown at home-mixer/clients/served_history_client.rs:186

            .put(self.tenant.clone(), pkey, lkey, value)
            .await
            .map_err(|e| anyhow::anyhow!("MH put failed: {e}"))
    }

    async fn delete(
        &self,
        user_id: u64,
        timeline_type: TimelineType,
        client_platform: i32,
        served_times_ms: &[i64],
    ) -> Result<()> {
        let pkey = timeline_pkey(timeline_type, user_id, client_platform);
        for &ts in served_times_ms {
            let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];
            self.client
                .delete(self.tenant.clone(), pkey.clone(), lkey)
                .await
                .map_err(|e| anyhow::anyhow!("MH delete failed: {e}"))?;
        }
        Ok(())
    }
}

pub struct MockServedHistoryClient;

#[async_trait]
impl ServedHistoryClient for MockServedHistoryClient {
    async fn get_recent(
        &self,
        _user_id: u64,
        _timeline_type: TimelineType,
        _client_platform: i32,
    ) -> Result<Vec<ServedHistory>> {
        Ok(vec![])
    }

View on GitHub (pinned to 24c60942c5)

Solutions

1. Retry the whole batch or just the remaining timestamps with backoff
2. Collect per-timestamp errors instead of aborting on first failure, then retry only failures
3. Check MH delete quotas/permissions for the tenant
4. Ensure callers treat delete as eventually-consistent cleanup, tolerating partial completion

Example fix

// before
for &ts in served_times_ms {
    let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];
    self.client.delete(self.tenant.clone(), pkey.clone(), lkey).await
        .map_err(|e| anyhow::anyhow!("MH delete failed: {e}"))?;
}

// after
let mut failures = Vec::new();
for &ts in served_times_ms {
    let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];
    if let Err(e) = self.client.delete(self.tenant.clone(), pkey.clone(), lkey).await {
        failures.push((ts, e));
    }
}
if !failures.is_empty() {
    anyhow::bail!("MH delete failed for {} of {} timestamps", failures.len(), served_times_ms.len());
}
Ok(())

null

Collect per-timestamp errors; retry only failures; surface aggregate if some remain.

When it happens

Trigger: Calling delete(user_id, timeline_type, client_platform, served_times_ms) where any single delete hits a backend error: throttling, network failure, or missing delete permissions; earlier deletes in the loop may have succeeded.

Common situations: Batch cleanup jobs deleting many entries hitting rate limits; partial-delete inconsistency when one timestamp fails mid-loop; permission gaps for deletes on the tenant.

Related errors

• Failed to create native MH client for served history: {e}
• MH scan failed: {e}
• MH put failed: {e}
• Strato error code {}: {}
• Failed to create consumer for thread {}: {:#}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/7ab8546c15d46358. Report an issue: GitHub.