JuliusBrussee/caveman · error · ReplayRunError

cachebench: provider output tokens %d exceed request ceiling

Error message

cachebench: provider output tokens %d exceed request ceiling %d for %q

What it means

Budget guard after a replayed provider call: the provider-reported output tokens exceed the request's declared MaxOutputTokens ceiling, so declared budgets understated actual spend. The request's max-output declaration is at fault.

Source

Thrown at cacheengine/cachebench/replay.go:735

			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider usage unavailable for %q", record.RequestID),
		}
	}
	evidence.ProviderUsageSHA256 = bodyDigest(usage.RawUsage)
	evidence.ProviderTotalInputTokens = usage.TotalInputTokens
	evidence.ProviderOutputTokens = usage.OutputTokens
	if usage.TotalInputTokens > record.DeclaredInputTokens {
		evidence.FailureCode = "provider_input_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider input tokens %d exceed declared ceiling %d for %q", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),
		}
	}
	if usage.OutputTokens > record.MaxOutputTokens {
		evidence.FailureCode = "provider_output_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider output tokens %d exceed request ceiling %d for %q", usage.OutputTokens, record.MaxOutputTokens, record.RequestID),
		}
	}
	verification, err := runner.Verifier.Verify(ctx, ReplayVerificationInput{Trace: record, Optimized: optimized, Response: response})
	if err != nil {
		evidence.FailureCode = "quality_verifier_error"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: quality verifier failed for %q: %w", record.RequestID, err),
		}
	}
	if !validTaskVerification(verification) {
		evidence.FailureCode = "quality_provenance_missing"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: quality provenance missing for %q", record.RequestID),
		}
	}
	evidence.QualityPassed = verification.Passed

View on GitHub (pinned to 766dce6b13)

Solutions

1. Raise MaxOutputTokens for the request in the trace
2. Cap provider-side max tokens in the request body

When it happens

Trigger: Thrown at cacheengine/cachebench/replay.go:735 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/2d43a6247f81b037. Report an issue: GitHub.