JuliusBrussee/caveman · error · ReplayRunError

quality_verifier_error

quality_verifier_error

Error message

cachebench: quality verifier failed for %q: %w

What it means

The quality Verifier returned an error (not a failed verdict — an execution error) while checking the response for the request. The runner fails the request with FailureCode quality_verifier_error wrapping the verifier's error; quality evidence cannot be recorded.

Source

Thrown at cacheengine/cachebench/replay.go:743

		evidence.FailureCode = "provider_input_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider input tokens %d exceed declared ceiling %d for %q", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),
		}
	}
	if usage.OutputTokens > record.MaxOutputTokens {
		evidence.FailureCode = "provider_output_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider output tokens %d exceed request ceiling %d for %q", usage.OutputTokens, record.MaxOutputTokens, record.RequestID),
		}
	}
	verification, err := runner.Verifier.Verify(ctx, ReplayVerificationInput{Trace: record, Optimized: optimized, Response: response})
	if err != nil {
		evidence.FailureCode = "quality_verifier_error"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: quality verifier failed for %q: %w", record.RequestID, err),
		}
	}
	if !validTaskVerification(verification) {
		evidence.FailureCode = "quality_provenance_missing"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: quality provenance missing for %q", record.RequestID),
		}
	}
	evidence.QualityPassed = verification.Passed
	evidence.QualityVerifier = verification.Verifier
	evidence.QualityEvidenceSHA256 = bodyDigest(verification.Evidence)
	evidence.Success = true
	observation := NewObservationRecord(record.RequestID, record.Provider, record.Epoch, usage.TotalInputTokens, record.Body, response.Body, optimized, verification, usage.RawUsage)
	return ReplayResult{
		Evidence: evidence, Observation: &observation,
		ProviderResponse:     append([]byte(nil), response.Body...),
		VerificationEvidence: append([]byte(nil), verification.Evidence...),

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Unwrap the %w error to find the verifier's root cause (auth, network, parse).
2. Check verifier configuration and credentials before the run; probe it with one record first.
3. If the verifier cannot score the new response format, update it to match the provider's current schema.
4. If verifier capacity is the issue under concurrency, rate-limit verify calls or use a sturdier verifier.

Example fix

// before
verifier := myjudge.New(myjudge.Config{}) // no key set -> 401 on every Verify

// after
verifier := myjudge.New(myjudge.Config{APIKey: os.Getenv("JUDGE_API_KEY")})
// smoke-test: _ = verifier.Verify(ctx, sampleInput)

if _, err := verifier.Verify(ctx, cachebench.ReplayVerificationInput{Trace: sampleRecord}); err != nil {
    return fmt.Errorf("verifier not ready: %w", err)
}

func isVerifierError(err error) bool {
    var rre *cachebench.ReplayRunError
    return errors.As(err, &rre) && rre.FailureCode == "quality_verifier_error"
}

if err := runner.Run(ctx, records, emit); err != nil {
    if isVerifierError(err) {
        cause := errors.Unwrap(err)
        // fix verifier creds/endpoint/format support, probe once, then re-run
        _ = cause
    }
    return err
}

When it happens

Trigger: Verifier.Verify failing internally: its backing service unreachable, response content in a format it cannot score, missing credentials for a grading API, or its own context cancellation.

Common situations: LLM-as-judge verifiers whose API key expired or endpoint is blocked; a verifier expecting a response schema the provider changed; verifier subprocess crashing under concurrent load.

Related errors

• cachebench: session %q request %d: %w
• cachebench: observation request %q body digest mismatch
• cachebench: replay request %q has empty provider
• cachebench: provider %q population %d cannot meet minimum el
• cachebench: replay preparation interrupted: %w

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/07ee506d0c60e276. Report an issue: GitHub.