JuliusBrussee/caveman · error

cachebench: verifier returned incomplete or mismatched evide

Error message

cachebench: verifier returned incomplete or mismatched evidence

What it means

The final gate in ParseVerificationCommandOutput: no trailing JSON after the object, schema equal to VerificationSchema, request_id equal to the expected one, verifier a bounded non-empty text (<=256), and evidence present, valid JSON, and not null. Any mismatch means the evidence cannot be attributed to this request and is rejected.

Source

Thrown at cacheengine/cachebench/replay.go:62

	Verifier  string          `json:"verifier"`
	Evidence  json.RawMessage `json:"evidence"`
}

// ParseVerificationCommandOutput validates and binds external grader evidence.
func ParseVerificationCommandOutput(raw []byte, requestID string) (TaskVerification, error) {
	raw = bytes.TrimSpace(raw)
	if !validUniqueJSONObject(raw) {
		return TaskVerification{}, errors.New("cachebench: verifier returned duplicate or invalid JSON")
	}
	decoder := json.NewDecoder(bytes.NewReader(raw))
	decoder.DisallowUnknownFields()
	var result VerificationCommandOutput
	if decoder.Decode(&result) != nil {
		return TaskVerification{}, errors.New("cachebench: verifier returned invalid JSON")
	}
	var trailing any
	if decoder.Decode(&trailing) != io.EOF || result.Schema != VerificationSchema || result.RequestID != requestID || !validBoundedText(result.Verifier, 256, false) || len(result.Evidence) == 0 || !json.Valid(result.Evidence) || bytes.Equal(bytes.TrimSpace(result.Evidence), []byte("null")) {
		return TaskVerification{}, errors.New("cachebench: verifier returned incomplete or mismatched evidence")
	}
	return TaskVerification{Passed: result.Passed, Verifier: result.Verifier, Evidence: append([]byte(nil), raw...)}, nil
}

// ReplayLimits bounds paid population, schedule, and concurrency.
type ReplayLimits struct {
	MaxRequests             int
	MaxDeclaredBilledTokens int64
	MaxGap                  time.Duration
	MaxScheduleDrift        time.Duration
	MaxConcurrency          int
	RequireGroundedTiming   bool
	RequireProviderTokens   bool
}

// ReplayPreflight is zero-network validation and declared-budget summary.
type ReplayPreflight struct {
	Requests                          int      `json:"requests"`

View on GitHub (pinned to 27d5a3981a)

Solutions

  1. Pass the exact request_id being verified and have the grader echo it unchanged
  2. Ensure evidence is always a non-null JSON value (empty object {} is acceptable, null is not)
  3. Keep schema pinned to VerificationSchema and update both sides together on version bumps
  4. Run graders one-at-a-time per request or prefix outputs per goroutine to avoid interleaving

Example fix

// before
v, err := ParseVerificationCommandOutput(raw, otherRequestID) // mismatch
// grader emitted "evidence": null

// after
v, err := ParseVerificationCommandOutput(raw, req.ID)
// grader emits "evidence": {} minimum
Defensive patterns

Strategy: validation

Validate before calling

if v.RequestID != requestID || v.Schema != expectedSchema || len(bytes.TrimSpace(v.Evidence)) == 0 || bytes.Equal(bytes.TrimSpace(v.Evidence), []byte("null")) {
    return errors.New("evidence not attributable to this request")
}
verification, err := cachebench.ParseVerificationCommandOutput(raw, requestID)

Type guard

func evidenceAttributable(raw []byte, requestID, schema string) bool {
    var v struct {
        Schema string `json:"schema"`; RequestID string `json:"request_id"`
        Verifier string `json:"verifier"`; Evidence json.RawMessage `json:"evidence"`
    }
    if json.Unmarshal(raw, &v) != nil { return false }
    return v.Schema == schema && v.RequestID == requestID && v.Verifier != "" && len(v.Verifier) <= 256 &&
        len(v.Evidence) > 0 && !bytes.Equal(bytes.TrimSpace(v.Evidence), []byte("null"))
}

Try / catch

if _, err := cachebench.ParseVerificationCommandOutput(raw, req.ID); err != nil {
    if err.Error() == "cachebench: verifier returned incomplete or mismatched evidence" {
        // verify request_id plumbing and evidence non-null before rerunning the grader
    }
}

Prevention

When it happens

Trigger: request_id in the grader output differing from the request being verified; missing/empty evidence or literal null evidence; schema field not matching the expected VerificationSchema constant; verifier name blank or over 256 chars; concatenated trailing JSON.

Common situations: Grader runs concurrently and outputs interleaved by request, so IDs cross; passing the wrong request_id argument when parsing; grader emitting "evidence": null on skip paths; schema version mismatch after an upgrade.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/67a5d62304dd2101. Report an issue: GitHub.