{"record":{"id":"3f587c36d748718f","repo":"JuliusBrussee/caveman","slug":"provider-usage-invalid","errorCode":"provider_usage_invalid","errorMessage":"cachebench: provider usage unavailable for %q","messagePattern":"cachebench: provider usage unavailable for %q","errorType":"validation","errorClass":"ReplayRunError","httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/replay.go","lineNumber":718,"sourceCode":"\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider returned invalid request identity for %q\", record.RequestID),\n\t\t}\n\t}\n\tevidence.ProviderRequestID = response.ProviderRequestID\n\tif response.StatusCode < 200 || response.StatusCode >= 300 {\n\t\tevidence.FailureCode = \"provider_http_status\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider returned HTTP %d for %q\", response.StatusCode, record.RequestID),\n\t\t}\n\t}\n\tusage, ok := cacheengine.ExtractProviderUsage(record.Provider, response.Body)\n\tif !ok {\n\t\tevidence.FailureCode = \"provider_usage_invalid\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider usage unavailable for %q\", record.RequestID),\n\t\t}\n\t}\n\tevidence.ProviderUsageSHA256 = bodyDigest(usage.RawUsage)\n\tevidence.ProviderTotalInputTokens = usage.TotalInputTokens\n\tevidence.ProviderOutputTokens = usage.OutputTokens\n\tif usage.TotalInputTokens > record.DeclaredInputTokens {\n\t\tevidence.FailureCode = \"provider_input_budget_exceeded\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider input tokens %d exceed declared ceiling %d for %q\", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),\n\t\t}\n\t}\n\tif usage.OutputTokens > record.MaxOutputTokens {\n\t\tevidence.FailureCode = \"provider_output_budget_exceeded\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider output tokens %d exceed request ceiling %d for %q\", usage.OutputTokens, record.MaxOutputTokens, record.RequestID),\n\t\t}","sourceCodeStart":700,"sourceCodeEnd":736,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/replay.go#L700-L736","documentation":"cacheengine.ExtractProviderUsage could not find a usable usage block in the provider response body. Replay evidence requires provider-reported token usage, so a 2xx response without parseable usage fails with FailureCode provider_usage_invalid.","triggerScenarios":"A provider response whose usage field is absent, null, or in an unexpected shape for the record's Provider; streaming responses where usage arrives in a final chunk that the transport did not aggregate; a provider dialect ExtractProviderUsage does not know.","commonSituations":"Replaying traces against a new provider or endpoint format the usage extractor hasn't learned; emulator responses with realistic text but no usage object; partial/truncated response bodies from transport timeouts that still yielded 2xx.","solutions":["Dump response.Body for the failing RequestID and check the usage field name/shape against ExtractProviderUsage's expectations.","If a new provider dialect, add or extend the usage extractor in cacheengine for it.","Ensure the transport aggregates usage from the final stream chunk before returning.","If replaying against an emulator, make its responses include a valid usage block."],"exampleFix":"// before\nemulator.Respond(\"{\\\"output\\\": \\\"hi\\\"}\") // no usage\n\n// after\nemulator.Respond(\"{\\\"output\\\": \\\"hi\\\", \\\"usage\\\": {\\\"input_tokens\\\": 10, \\\"output_tokens\\\": 2}}\")","handlingStrategy":"validation","validationCode":"if _, ok := cacheengine.ExtractProviderUsage(record.Provider, sampleResponseBody); !ok {\n    return fmt.Errorf(\"provider %s responses lack parseable usage; check extractor/dialect\", record.Provider)\n}","typeGuard":"func isProviderUsageInvalid(err error) bool {\n    var rre *cachebench.ReplayRunError\n    return errors.As(err, &rre) && rre.FailureCode == \"provider_usage_invalid\"\n}","tryCatchPattern":"if err := runner.Run(ctx, records, emit); err != nil {\n    if isProviderUsageInvalid(err) {\n        // inspect the raw response body from the emitted result; extend the usage extractor for this dialect\n    }\n    return err\n}","preventionTips":["Probe one response per provider and assert ExtractProviderUsage succeeds before replay.","When adding a provider, add a usage-extraction fixture test alongside it.","Ensure transports aggregate final-chunk usage for streaming endpoints."],"tags":["cachebench","provider-api","usage","parsing"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}