{"record":{"id":"deb7c7d2fb248fbd","repo":"JuliusBrussee/caveman","slug":"provider-output-budget-exceeded","errorCode":"provider_output_budget_exceeded","errorMessage":"cachebench: provider output tokens %d exceed request ceiling %d for %q","messagePattern":"cachebench: provider output tokens (.+?) exceed request ceiling (.+?) for %q","errorType":"validation","errorClass":"ReplayRunError","httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/replay.go","lineNumber":735,"sourceCode":"\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider usage unavailable for %q\", record.RequestID),\n\t\t}\n\t}\n\tevidence.ProviderUsageSHA256 = bodyDigest(usage.RawUsage)\n\tevidence.ProviderTotalInputTokens = usage.TotalInputTokens\n\tevidence.ProviderOutputTokens = usage.OutputTokens\n\tif usage.TotalInputTokens > record.DeclaredInputTokens {\n\t\tevidence.FailureCode = \"provider_input_budget_exceeded\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider input tokens %d exceed declared ceiling %d for %q\", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),\n\t\t}\n\t}\n\tif usage.OutputTokens > record.MaxOutputTokens {\n\t\tevidence.FailureCode = \"provider_output_budget_exceeded\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider output tokens %d exceed request ceiling %d for %q\", usage.OutputTokens, record.MaxOutputTokens, record.RequestID),\n\t\t}\n\t}\n\tverification, err := runner.Verifier.Verify(ctx, ReplayVerificationInput{Trace: record, Optimized: optimized, Response: response})\n\tif err != nil {\n\t\tevidence.FailureCode = \"quality_verifier_error\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: quality verifier failed for %q: %w\", record.RequestID, err),\n\t\t}\n\t}\n\tif !validTaskVerification(verification) {\n\t\tevidence.FailureCode = \"quality_provenance_missing\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: quality provenance missing for %q\", record.RequestID),\n\t\t}\n\t}\n\tevidence.QualityPassed = verification.Passed","sourceCodeStart":717,"sourceCodeEnd":753,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/replay.go#L717-L753","documentation":"Provider-reported output tokens exceeded record.MaxOutputTokens, the request's own output ceiling from the trace. The harness enforces the ceiling to catch responses that blow past what the original request allowed; the request fails with FailureCode provider_output_budget_exceeded.","triggerScenarios":"A response more verbose than MaxOutputTokens in the trace record — typical when replaying with temperature/sampling differences, a different model version, or a record whose MaxOutputTokens was captured from the request parameter but the provider overshot it.","commonSituations":"Non-deterministic sampling on replay producing longer outputs; model upgrades yielding chattier responses; trace records where MaxOutputTokens reflects a per-field max_tokens the provider treats differently.","solutions":["Raise record.MaxOutputTokens (or re-capture) if the ceiling was set from the original request but legitimate replay variance overshoots it.","Pin sampling parameters (temperature, seed, max_tokens) in the replayed body to reduce output variance.","Retry the replay once — a single over-long sample can be transient under non-deterministic settings.","If overshoot is systematic after a model change, re-baseline the trace against the new model."],"exampleFix":"// before\nrecord := cachebench.TraceRecord{MaxOutputTokens: 128, Body: body}\n\n// after\nrecord := cachebench.TraceRecord{MaxOutputTokens: 1024, Body: body} // or set max_tokens:128 in body and re-capture","handlingStrategy":"retry","validationCode":"if record.MaxOutputTokens < 2*expectedTypicalOutput(record) {\n    // ceiling too tight for sampling variance; consider raising before replay\n    log.Printf(\"record %s: MaxOutputTokens %d close to typical output\", record.RequestID, record.MaxOutputTokens)\n}","typeGuard":"func isOutputBudgetExceeded(err error) bool {\n    var rre *cachebench.ReplayRunError\n    return errors.As(err, &rre) && rre.FailureCode == \"provider_output_budget_exceeded\"\n}","tryCatchPattern":"if err := runner.Run(ctx, records, emit); err != nil {\n    if isOutputBudgetExceeded(err) {\n        // transient under non-deterministic sampling: re-run; systematic: re-baseline MaxOutputTokens or pin sampling\n    }\n    return err\n}","preventionTips":["Pin temperature/seed/max_tokens in replayed bodies to bound output length variance.","Re-baseline traces after model or provider changes that shift verbosity.","Set MaxOutputTokens from observed response distributions, not request parameters alone."],"tags":["cachebench","budget","tokens","replay"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}