xai-org/x-algorithm · error · anyhow::Error

MH delete failed

Error message

MH delete failed: {e}

What it means

Raised in delete when a Manhattan delete RPC fails for one of the provided served_times_ms timestamps (lkey = i64::MAX - ts). The loop aborts on the first failing delete, so later timestamps in the batch are not attempted.

Solutions

1. Retry the whole batch or just the remaining timestamps with backoff
2. Collect per-timestamp errors instead of aborting on first failure, then retry only failures
3. Check MH delete quotas/permissions for the tenant
4. Ensure callers treat delete as eventually-consistent cleanup, tolerating partial completion

Example fix

// before
for &ts in served_times_ms {
    let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];
    self.client.delete(self.tenant.clone(), pkey.clone(), lkey).await
        .map_err(|e| anyhow::anyhow!("MH delete failed: {e}"))?;
}

// after
let mut failures = Vec::new();
for &ts in served_times_ms {
    let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];
    if let Err(e) = self.client.delete(self.tenant.clone(), pkey.clone(), lkey).await {
        failures.push((ts, e));
    }
}
if !failures.is_empty() {
    anyhow::bail!("MH delete failed for {} of {} timestamps", failures.len(), served_times_ms.len());
}
Ok(())

null

Collect per-timestamp errors; retry only failures; surface aggregate if some remain.

When it happens

Trigger: Calling delete(user_id, timeline_type, client_platform, served_times_ms) where any single delete hits a backend error: throttling, network failure, or missing delete permissions; earlier deletes in the loop may have succeeded.

Common situations: Batch cleanup jobs deleting many entries hitting rate limits; partial-delete inconsistency when one timestamp fails mid-loop; permission gaps for deletes on the tenant.

Related errors

• Failed to create native MH client for served history
• MH put failed
• MH scan failed
• Strato error code
• BACKEND_UNAVAILABLE

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/7ab8546c15d46358. Report an issue: GitHub.

Appendix: source

Thrown at home-mixer/clients/served_history_client.rs:186

            .put(self.tenant.clone(), pkey, lkey, value)
            .await
            .map_err(|e| anyhow::anyhow!("MH put failed: {e}"))
    }

    async fn delete(
        &self,
        user_id: u64,
        timeline_type: TimelineType,
        client_platform: i32,
        served_times_ms: &[i64],
    ) -> Result<()> {
        let pkey = timeline_pkey(timeline_type, user_id, client_platform);
        for &ts in served_times_ms {
            let lkey = vec![(i64::MAX - ts).to_be_bytes().to_vec()];
            self.client
                .delete(self.tenant.clone(), pkey.clone(), lkey)
                .await
                .map_err(|e| anyhow::anyhow!("MH delete failed: {e}"))?;
        }
        Ok(())
    }
}

pub struct MockServedHistoryClient;

#[async_trait]
impl ServedHistoryClient for MockServedHistoryClient {
    async fn get_recent(
        &self,
        _user_id: u64,
        _timeline_type: TimelineType,
        _client_platform: i32,
    ) -> Result<Vec<ServedHistory>> {
        Ok(vec![])
    }

View on GitHub (pinned to 24c60942c5)