{"record":{"id":"7e6c85f676ba56a1","repo":"JuliusBrussee/caveman","slug":"cachebench-quality-verifier-failed-for-q-w","errorCode":null,"errorMessage":"cachebench: quality verifier failed for %q: %w","messagePattern":"cachebench: quality verifier failed for %q: %w","errorType":"exception","errorClass":"ReplayRunError","httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/replay.go","lineNumber":743,"sourceCode":"\t\tevidence.FailureCode = \"provider_input_budget_exceeded\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider input tokens %d exceed declared ceiling %d for %q\", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),\n\t\t}\n\t}\n\tif usage.OutputTokens > record.MaxOutputTokens {\n\t\tevidence.FailureCode = \"provider_output_budget_exceeded\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: provider output tokens %d exceed request ceiling %d for %q\", usage.OutputTokens, record.MaxOutputTokens, record.RequestID),\n\t\t}\n\t}\n\tverification, err := runner.Verifier.Verify(ctx, ReplayVerificationInput{Trace: record, Optimized: optimized, Response: response})\n\tif err != nil {\n\t\tevidence.FailureCode = \"quality_verifier_error\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: quality verifier failed for %q: %w\", record.RequestID, err),\n\t\t}\n\t}\n\tif !validTaskVerification(verification) {\n\t\tevidence.FailureCode = \"quality_provenance_missing\"\n\t\treturn ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{\n\t\t\tRequestID: record.RequestID, FailureCode: evidence.FailureCode,\n\t\t\tErr: fmt.Errorf(\"cachebench: quality provenance missing for %q\", record.RequestID),\n\t\t}\n\t}\n\tevidence.QualityPassed = verification.Passed\n\tevidence.QualityVerifier = verification.Verifier\n\tevidence.QualityEvidenceSHA256 = bodyDigest(verification.Evidence)\n\tevidence.Success = true\n\tobservation := NewObservationRecord(record.RequestID, record.Provider, record.Epoch, usage.TotalInputTokens, record.Body, response.Body, optimized, verification, usage.RawUsage)\n\treturn ReplayResult{\n\t\tEvidence: evidence, Observation: &observation,\n\t\tProviderResponse:     append([]byte(nil), response.Body...),\n\t\tVerificationEvidence: append([]byte(nil), verification.Evidence...),","sourceCodeStart":725,"sourceCodeEnd":761,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/766dce6b1394ebb56a3090748d5a0240a5aefb36/cacheengine/cachebench/replay.go#L725-L761","documentation":"The external quality verifier returned an error while checking one replay result; failure code quality_verifier_error. The wrapped error carries the verifier-side cause (often transient grader failures).","triggerScenarios":"Thrown at cacheengine/cachebench/replay.go:743 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Inspect the wrapped verifier error and retry the run if transient","Stabilise the verifier service/inputs before re-running"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"766dce6b1394ebb56a3090748d5a0240a5aefb36","analyzedAt":"2026-08-18T03:14:35.516Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}