JuliusBrussee/caveman · error · ReplayRunError

provider_response_invalid

provider_response_invalid

Error message

cachebench: provider returned invalid request identity for %q

What it means

The transport succeeded but response.ProviderRequestID failed the validProviderRequestID check (empty or malformed). Every replayed request must carry a verifiable provider-side request identity for audit evidence, so the request fails with FailureCode provider_response_invalid even if the HTTP call itself was fine.

Source

Thrown at cacheengine/cachebench/replay.go:702

		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: clock regressed while replaying %q", record.RequestID),
		}
	}
	evidence.CompletedAt = completed.Format(time.RFC3339Nano)
	evidence.LatencyMilliseconds = completed.Sub(started).Milliseconds()
	if sendErr != nil {
		evidence.FailureCode = "transport_error"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider transport failed for %q: %w", record.RequestID, sendErr),
		}
	}
	if !validProviderRequestID(response.ProviderRequestID) {
		evidence.FailureCode = "provider_response_invalid"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider returned invalid request identity for %q", record.RequestID),
		}
	}
	evidence.ProviderRequestID = response.ProviderRequestID
	if response.StatusCode < 200 || response.StatusCode >= 300 {
		evidence.FailureCode = "provider_http_status"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider returned HTTP %d for %q", response.StatusCode, record.RequestID),
		}
	}
	usage, ok := cacheengine.ExtractProviderUsage(record.Provider, response.Body)
	if !ok {
		evidence.FailureCode = "provider_usage_invalid"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider usage unavailable for %q", record.RequestID),
		}
	}

View on GitHub (pinned to 27d5a3981a)

Solutions

Fix the Transport to populate ReplayResponse.ProviderRequestID from the provider's request-id header (e.g. request-id / x-request-id).
If replaying against an emulator, make it return a well-formed id.
Verify the header name still matches the provider's current API documentation after upgrades.
If you control the contract, validate the id format (non-empty, expected charset) inside the transport before returning.

Example fix

// before
return cachebench.ReplayResponse{StatusCode: 200, Body: body}, nil // no ProviderRequestID

// after
return cachebench.ReplayResponse{StatusCode: 200, Body: body, ProviderRequestID: resp.Header.Get("request-id")}, nil

Defensive patterns

Strategy: validation

Validate before calling

func providerIDPresent(id string) bool {
    return strings.TrimSpace(id) != ""
}

Type guard

func isProviderResponseInvalid(err error) bool {
    var rre *cachebench.ReplayRunError
    return errors.As(err, &rre) && rre.FailureCode == "provider_response_invalid"
}

Try / catch

if err := runner.Run(ctx, records, emit); err != nil {
    if isProviderResponseInvalid(err) {
        // transport returned no ProviderRequestID; fix header extraction in the transport
    }
    return err
}

Prevention

Unit-test custom transports: assert ProviderRequestID is populated from the provider's id header.
Emulators used for replay must echo a well-formed request id.
Re-check header names after provider API upgrades.

When it happens

Trigger: A Transport implementation that returns a ReplayResponse without setting ProviderRequestID (e.g. a stub/mock transport), or a provider/endpoint that does not echo a request id header; an id containing only whitespace or invalid characters.

Common situations: Custom test transports forgetting to populate the field; pointing replay at a provider emulator that lacks the request-id header; header name changes after a provider API update so the transport extracts nothing.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/0c31d18097c91ae1. Report an issue: GitHub.