JuliusBrussee/caveman · error · ReplayRunError

provider_usage_invalid

provider_usage_invalid

Error message

cachebench: provider usage unavailable for %q

What it means

cacheengine.ExtractProviderUsage could not find a usable usage block in the provider response body. Replay evidence requires provider-reported token usage, so a 2xx response without parseable usage fails with FailureCode provider_usage_invalid.

Source

Thrown at cacheengine/cachebench/replay.go:718

		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider returned invalid request identity for %q", record.RequestID),
		}
	}
	evidence.ProviderRequestID = response.ProviderRequestID
	if response.StatusCode < 200 || response.StatusCode >= 300 {
		evidence.FailureCode = "provider_http_status"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider returned HTTP %d for %q", response.StatusCode, record.RequestID),
		}
	}
	usage, ok := cacheengine.ExtractProviderUsage(record.Provider, response.Body)
	if !ok {
		evidence.FailureCode = "provider_usage_invalid"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider usage unavailable for %q", record.RequestID),
		}
	}
	evidence.ProviderUsageSHA256 = bodyDigest(usage.RawUsage)
	evidence.ProviderTotalInputTokens = usage.TotalInputTokens
	evidence.ProviderOutputTokens = usage.OutputTokens
	if usage.TotalInputTokens > record.DeclaredInputTokens {
		evidence.FailureCode = "provider_input_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider input tokens %d exceed declared ceiling %d for %q", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),
		}
	}
	if usage.OutputTokens > record.MaxOutputTokens {
		evidence.FailureCode = "provider_output_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider output tokens %d exceed request ceiling %d for %q", usage.OutputTokens, record.MaxOutputTokens, record.RequestID),
		}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Dump response.Body for the failing RequestID and check the usage field name/shape against ExtractProviderUsage's expectations.
2. If a new provider dialect, add or extend the usage extractor in cacheengine for it.
3. Ensure the transport aggregates usage from the final stream chunk before returning.
4. If replaying against an emulator, make its responses include a valid usage block.

Example fix

// before
emulator.Respond("{\"output\": \"hi\"}") // no usage

// after
emulator.Respond("{\"output\": \"hi\", \"usage\": {\"input_tokens\": 10, \"output_tokens\": 2}}")

if _, ok := cacheengine.ExtractProviderUsage(record.Provider, sampleResponseBody); !ok {
    return fmt.Errorf("provider %s responses lack parseable usage; check extractor/dialect", record.Provider)
}

func isProviderUsageInvalid(err error) bool {
    var rre *cachebench.ReplayRunError
    return errors.As(err, &rre) && rre.FailureCode == "provider_usage_invalid"
}

if err := runner.Run(ctx, records, emit); err != nil {
    if isProviderUsageInvalid(err) {
        // inspect the raw response body from the emitted result; extend the usage extractor for this dialect
    }
    return err
}

When it happens

Trigger: A provider response whose usage field is absent, null, or in an unexpected shape for the record's Provider; streaming responses where usage arrives in a final chunk that the transport did not aggregate; a provider dialect ExtractProviderUsage does not know.

Common situations: Replaying traces against a new provider or endpoint format the usage extractor hasn't learned; emulator responses with realistic text but no usage object; partial/truncated response bodies from transport timeouts that still yielded 2xx.

Related errors

• provider_response_invalid
• provider_http_status
• provider_input_budget_exceeded
• cave_eve_terminal_${result.status}
• cave_harness_upstream_version_mismatch

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/3f587c36d748718f. Report an issue: GitHub.