JuliusBrussee/caveman · error · ReplayRunError

provider_http_status

provider_http_status

Error message

cachebench: provider returned HTTP %d for %q

What it means

The provider answered with a status outside 200-299. The runner records status, ids, and body digest in evidence, then fails the request with FailureCode provider_http_status and the concrete status code.

Source

Thrown at cacheengine/cachebench/replay.go:710

		evidence.FailureCode = "transport_error"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider transport failed for %q: %w", record.RequestID, sendErr),
		}
	}
	if !validProviderRequestID(response.ProviderRequestID) {
		evidence.FailureCode = "provider_response_invalid"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider returned invalid request identity for %q", record.RequestID),
		}
	}
	evidence.ProviderRequestID = response.ProviderRequestID
	if response.StatusCode < 200 || response.StatusCode >= 300 {
		evidence.FailureCode = "provider_http_status"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider returned HTTP %d for %q", response.StatusCode, record.RequestID),
		}
	}
	usage, ok := cacheengine.ExtractProviderUsage(record.Provider, response.Body)
	if !ok {
		evidence.FailureCode = "provider_usage_invalid"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider usage unavailable for %q", record.RequestID),
		}
	}
	evidence.ProviderUsageSHA256 = bodyDigest(usage.RawUsage)
	evidence.ProviderTotalInputTokens = usage.TotalInputTokens
	evidence.ProviderOutputTokens = usage.OutputTokens
	if usage.TotalInputTokens > record.DeclaredInputTokens {
		evidence.FailureCode = "provider_input_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider input tokens %d exceed declared ceiling %d for %q", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Map the status: 401/403 fix credentials; 404 fix record.Endpoint/Model; 429 lower MaxConcurrency or add backoff; 5xx retry later; 400 inspect the optimized body against the provider schema.
2. Check the emitted evidence's ProviderEvidenceSHA256 against the response body you captured for details.
3. Re-verify the trace's Endpoint and Model fields against the provider's current API.
4. For 429s, scale replay rate via TimeScale or concurrency limits.

Example fix

// before
runner.MaxConcurrency = 32 // provider returns 429

// after
runner.MaxConcurrency = 4
// and/or add retry policy around transient 429/5xx at the transport layer

// pre-flight one representative request per provider to catch auth/endpoint issues
if status, err := transport.Probe(ctx, sampleRecord); err != nil || status < 200 || status >= 300 {
    return fmt.Errorf("pre-flight failed with status %d: %w", status, err)
}

func isProviderHTTPStatus(err error) bool {
    var rre *cachebench.ReplayRunError
    return errors.As(err, &rre) && rre.FailureCode == "provider_http_status"
}

if err := runner.Run(ctx, records, emit); err != nil {
    if isProviderHTTPStatus(err) {
        // 401/403: fix credentials; 404: fix Endpoint/Model in trace; 429: cut concurrency; 5xx: retry later
    }
    return err
}

When it happens

Trigger: 4xx/5xx from the provider: 401/403 bad credentials, 404 wrong endpoint in the trace record, 429 rate limiting under concurrent replay, 5xx provider incidents, 400 from a malformed optimized body.

Common situations: Expired or wrong API keys; endpoint drift between capture time and replay (renamed models/routes); replay concurrency tripping rate limits; provider outage during the run.

Related errors

• provider_response_invalid
• provider_usage_invalid
• provider_input_budget_exceeded
• Claude usage request failed with HTTP %d
• registration failed: HTTP ${response.status}

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/45eb8caf707ecd12. Report an issue: GitHub.