JuliusBrussee/caveman · error · ReplayRunError

provider_output_budget_exceeded

provider_output_budget_exceeded

Error message

cachebench: provider output tokens %d exceed request ceiling %d for %q

What it means

Provider-reported output tokens exceeded record.MaxOutputTokens, the request's own output ceiling from the trace. The harness enforces the ceiling to catch responses that blow past what the original request allowed; the request fails with FailureCode provider_output_budget_exceeded.

Source

Thrown at cacheengine/cachebench/replay.go:735

			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider usage unavailable for %q", record.RequestID),
		}
	}
	evidence.ProviderUsageSHA256 = bodyDigest(usage.RawUsage)
	evidence.ProviderTotalInputTokens = usage.TotalInputTokens
	evidence.ProviderOutputTokens = usage.OutputTokens
	if usage.TotalInputTokens > record.DeclaredInputTokens {
		evidence.FailureCode = "provider_input_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider input tokens %d exceed declared ceiling %d for %q", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),
		}
	}
	if usage.OutputTokens > record.MaxOutputTokens {
		evidence.FailureCode = "provider_output_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider output tokens %d exceed request ceiling %d for %q", usage.OutputTokens, record.MaxOutputTokens, record.RequestID),
		}
	}
	verification, err := runner.Verifier.Verify(ctx, ReplayVerificationInput{Trace: record, Optimized: optimized, Response: response})
	if err != nil {
		evidence.FailureCode = "quality_verifier_error"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: quality verifier failed for %q: %w", record.RequestID, err),
		}
	}
	if !validTaskVerification(verification) {
		evidence.FailureCode = "quality_provenance_missing"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: quality provenance missing for %q", record.RequestID),
		}
	}
	evidence.QualityPassed = verification.Passed

View on GitHub (pinned to 27d5a3981a)

Solutions

Raise record.MaxOutputTokens (or re-capture) if the ceiling was set from the original request but legitimate replay variance overshoots it.
Pin sampling parameters (temperature, seed, max_tokens) in the replayed body to reduce output variance.
Retry the replay once — a single over-long sample can be transient under non-deterministic settings.
If overshoot is systematic after a model change, re-baseline the trace against the new model.

Example fix

// before
record := cachebench.TraceRecord{MaxOutputTokens: 128, Body: body}

// after
record := cachebench.TraceRecord{MaxOutputTokens: 1024, Body: body} // or set max_tokens:128 in body and re-capture

Defensive patterns

Strategy: retry

Validate before calling

if record.MaxOutputTokens < 2*expectedTypicalOutput(record) {
    // ceiling too tight for sampling variance; consider raising before replay
    log.Printf("record %s: MaxOutputTokens %d close to typical output", record.RequestID, record.MaxOutputTokens)
}

Type guard

func isOutputBudgetExceeded(err error) bool {
    var rre *cachebench.ReplayRunError
    return errors.As(err, &rre) && rre.FailureCode == "provider_output_budget_exceeded"
}

Try / catch

if err := runner.Run(ctx, records, emit); err != nil {
    if isOutputBudgetExceeded(err) {
        // transient under non-deterministic sampling: re-run; systematic: re-baseline MaxOutputTokens or pin sampling
    }
    return err
}

Prevention

Pin temperature/seed/max_tokens in replayed bodies to bound output length variance.
Re-baseline traces after model or provider changes that shift verbosity.
Set MaxOutputTokens from observed response distributions, not request parameters alone.

When it happens

Trigger: A response more verbose than MaxOutputTokens in the trace record — typical when replaying with temperature/sampling differences, a different model version, or a record whose MaxOutputTokens was captured from the request parameter but the provider overshot it.

Common situations: Non-deterministic sampling on replay producing longer outputs; model upgrades yielding chattier responses; trace records where MaxOutputTokens reflects a per-field max_tokens the provider treats differently.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/deb7c7d2fb248fbd. Report an issue: GitHub.