{"record":{"id":"07ee506d0c60e276","repo":"JuliusBrussee/caveman","slug":"quality-verifier-error","errorCode":"quality_verifier_error","errorMessage":"cachebench: quality verifier failed for %q: %w","messagePattern":"cachebench: quality verifier failed for %q: %w","errorType":"exception","errorClass":"ReplayRunError","httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/replay.go","lineNumber":743,"sourceCode":"\t\tevidence.FailureCode = \"provider_input_budget_exceeded\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider input tokens %d exceed declared ceiling %d for %q\", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),\n\t\t}\n\t}\n\tif usage.OutputTokens > record.MaxOutputTokens {\n\t\tevidence.FailureCode = \"provider_output_budget_exceeded\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider output tokens %d exceed request ceiling %d for %q\", usage.OutputTokens, record.MaxOutputTokens, record.RequestID),\n\t\t}\n\t}\n\tverification, err := runner.Verifier.Verify(ctx, ReplayVerificationInput{Trace: record, Optimized: optimized, Response: response})\n\tif err != nil {\n\t\tevidence.FailureCode = \"quality_verifier_error\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: quality verifier failed for %q: %w\", record.RequestID, err),\n\t\t}\n\t}\n\tif !validTaskVerification(verification) {\n\t\tevidence.FailureCode = \"quality_provenance_missing\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: quality provenance missing for %q\", record.RequestID),\n\t\t}\n\t}\n\tevidence.QualityPassed = verification.Passed\n\tevidence.QualityVerifier = verification.Verifier\n\tevidence.QualityEvidenceSHA256 = bodyDigest(verification.Evidence)\n\tevidence.Success = true\n\tobservation := NewObservationRecord(record.RequestID, record.Provider, record.Epoch, usage.TotalInputTokens, record.Body, response.Body, optimized, verification, usage.RawUsage)\n\treturn ReplayResult{\n\t\tEvidence: evidence, Observation: &observation,\n\t\tProviderResponse:     append([]byte(nil), response.Body...),\n\t\tVerificationEvidence: append([]byte(nil), verification.Evidence...),","sourceCodeStart":725,"sourceCodeEnd":761,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/replay.go#L725-L761","documentation":"The quality Verifier returned an error (not a failed verdict — an execution error) while checking the response for the request. The runner fails the request with FailureCode quality_verifier_error wrapping the verifier's error; quality evidence cannot be recorded.","triggerScenarios":"Verifier.Verify failing internally: its backing service unreachable, response content in a format it cannot score, missing credentials for a grading API, or its own context cancellation.","commonSituations":"LLM-as-judge verifiers whose API key expired or endpoint is blocked; a verifier expecting a response schema the provider changed; verifier subprocess crashing under concurrent load.","solutions":["Unwrap the %w error to find the verifier's root cause (auth, network, parse).","Check verifier configuration and credentials before the run; probe it with one record first.","If the verifier cannot score the new response format, update it to match the provider's current schema.","If verifier capacity is the issue under concurrency, rate-limit verify calls or use a sturdier verifier."],"exampleFix":"// before\nverifier := myjudge.New(myjudge.Config{}) // no key set -> 401 on every Verify\n\n// after\nverifier := myjudge.New(myjudge.Config{APIKey: os.Getenv(\"JUDGE_API_KEY\")})\n// smoke-test: _ = verifier.Verify(ctx, sampleInput)","handlingStrategy":"try-catch","validationCode":"if _, err := verifier.Verify(ctx, cachebench.ReplayVerificationInput{Trace: sampleRecord}); err != nil {\n    return fmt.Errorf(\"verifier not ready: %w\", err)\n}","typeGuard":"func isVerifierError(err error) bool {\n    var rre *cachebench.ReplayRunError\n    return errors.As(err, &rre) && rre.FailureCode == \"quality_verifier_error\"\n}","tryCatchPattern":"if err := runner.Run(ctx, records, emit); err != nil {\n    if isVerifierError(err) {\n        cause := errors.Unwrap(err)\n        // fix verifier creds/endpoint/format support, probe once, then re-run\n        _ = cause\n    }\n    return err\n}","preventionTips":["Probe the verifier with one sample input before every full replay.","Keep verifier credentials and endpoints in monitored config; alert before they expire.","Update verifiers when provider response schemas change; add fixture tests for each format."],"tags":["cachebench","verifier","quality","replay"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}