JuliusBrussee/caveman · error · ReplayRunError

cachebench: quality verifier failed for %q: %w

Error message

cachebench: quality verifier failed for %q: %w

What it means

The external quality verifier returned an error while checking one replay result; failure code quality_verifier_error. The wrapped error carries the verifier-side cause (often transient grader failures).

Source

Thrown at cacheengine/cachebench/replay.go:743

		evidence.FailureCode = "provider_input_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider input tokens %d exceed declared ceiling %d for %q", usage.TotalInputTokens, record.DeclaredInputTokens, record.RequestID),
		}
	}
	if usage.OutputTokens > record.MaxOutputTokens {
		evidence.FailureCode = "provider_output_budget_exceeded"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider output tokens %d exceed request ceiling %d for %q", usage.OutputTokens, record.MaxOutputTokens, record.RequestID),
		}
	}
	verification, err := runner.Verifier.Verify(ctx, ReplayVerificationInput{Trace: record, Optimized: optimized, Response: response})
	if err != nil {
		evidence.FailureCode = "quality_verifier_error"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: quality verifier failed for %q: %w", record.RequestID, err),
		}
	}
	if !validTaskVerification(verification) {
		evidence.FailureCode = "quality_provenance_missing"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: quality provenance missing for %q", record.RequestID),
		}
	}
	evidence.QualityPassed = verification.Passed
	evidence.QualityVerifier = verification.Verifier
	evidence.QualityEvidenceSHA256 = bodyDigest(verification.Evidence)
	evidence.Success = true
	observation := NewObservationRecord(record.RequestID, record.Provider, record.Epoch, usage.TotalInputTokens, record.Body, response.Body, optimized, verification, usage.RawUsage)
	return ReplayResult{
		Evidence: evidence, Observation: &observation,
		ProviderResponse:     append([]byte(nil), response.Body...),
		VerificationEvidence: append([]byte(nil), verification.Evidence...),

View on GitHub (pinned to 766dce6b13)

Solutions

1. Inspect the wrapped verifier error and retry the run if transient
2. Stabilise the verifier service/inputs before re-running

When it happens

Trigger: Thrown at cacheengine/cachebench/replay.go:743 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/7e6c85f676ba56a1. Report an issue: GitHub.