JuliusBrussee/caveman · error

cachebench: verifier returned invalid JSON

Error message

cachebench: verifier returned invalid JSON

What it means

After the uniqueness check passes, the output must decode into VerificationCommandOutput with DisallowUnknownFields. A decode failure (malformed JSON, wrong types, or unknown fields) yields 'verifier returned invalid JSON' — the payload was a unique object but not a schema-conforming one.

Source

Thrown at cacheengine/cachebench/replay.go:58

type VerificationCommandOutput struct {
	Schema    string          `json:"schema"`
	RequestID string          `json:"request_id"`
	Passed    bool            `json:"passed"`
	Verifier  string          `json:"verifier"`
	Evidence  json.RawMessage `json:"evidence"`
}

// ParseVerificationCommandOutput validates and binds external grader evidence.
func ParseVerificationCommandOutput(raw []byte, requestID string) (TaskVerification, error) {
	raw = bytes.TrimSpace(raw)
	if !validUniqueJSONObject(raw) {
		return TaskVerification{}, errors.New("cachebench: verifier returned duplicate or invalid JSON")
	}
	decoder := json.NewDecoder(bytes.NewReader(raw))
	decoder.DisallowUnknownFields()
	var result VerificationCommandOutput
	if decoder.Decode(&result) != nil {
		return TaskVerification{}, errors.New("cachebench: verifier returned invalid JSON")
	}
	var trailing any
	if decoder.Decode(&trailing) != io.EOF || result.Schema != VerificationSchema || result.RequestID != requestID || !validBoundedText(result.Verifier, 256, false) || len(result.Evidence) == 0 || !json.Valid(result.Evidence) || bytes.Equal(bytes.TrimSpace(result.Evidence), []byte("null")) {
		return TaskVerification{}, errors.New("cachebench: verifier returned incomplete or mismatched evidence")
	}
	return TaskVerification{Passed: result.Passed, Verifier: result.Verifier, Evidence: append([]byte(nil), raw...)}, nil
}

// ReplayLimits bounds paid population, schedule, and concurrency.
type ReplayLimits struct {
	MaxRequests             int
	MaxDeclaredBilledTokens int64
	MaxGap                  time.Duration
	MaxScheduleDrift        time.Duration
	MaxConcurrency          int
	RequireGroundedTiming   bool
	RequireProviderTokens   bool
}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Align the verifier's output struct with VerificationCommandOutput: schema, request_id, passed, verifier, evidence only
2. Nest any extra information inside the evidence object rather than adding top-level fields
3. Pin/test the grader output against ParseVerificationCommandOutput in CI so drift is caught before replay

Example fix

// before
{"schema":"...","request_id":"r1","passed":true,"verifier":"v","evidence":{},"note":"extra"} // unknown field

// after
{"schema":"...","request_id":"r1","passed":true,"verifier":"v","evidence":{"note":"extra"}}

var probe struct {
    Schema string `json:"schema"`
    RequestID string `json:"request_id"`
    Passed bool `json:"passed"`
    Verifier string `json:"verifier"`
    Evidence json.RawMessage `json:"evidence"`
}
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if dec.Decode(&probe) != nil {
    return errors.New("grader output does not match verification schema")
}

func matchesVerificationSchema(raw []byte) bool {
    dec := json.NewDecoder(bytes.NewReader(raw))
    dec.DisallowUnknownFields()
    var out struct {
        Schema string `json:"schema"`; RequestID string `json:"request_id"`
        Passed bool `json:"passed"`; Verifier string `json:"verifier"`
        Evidence json.RawMessage `json:"evidence"`
    }
    return dec.Decode(&out) == nil
}

if _, err := cachebench.ParseVerificationCommandOutput(raw, id); err != nil {
    if err.Error() == "cachebench: verifier returned invalid JSON" {
        // diff grader fields against the four allowed keys; move extras into evidence
    }
    return err
}

When it happens

Trigger: Grader output missing required structure, wrong field types (passed as string), or containing extra fields not in the schema (DisallowUnknownFields rejects them). Also genuinely broken JSON that still passed the earlier top-level object check is unlikely, so unknown fields and type mismatches dominate.

Common situations: Grader schema drift after upgrading cachebench (new/renamed fields); verifier adding a convenience field like "notes" at top level; passing "evidence" as a string instead of raw JSON.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• cachebench: verifier returned duplicate or invalid JSON
• cachebench: verifier returned incomplete or mismatched evide
• ${path} is not a JSON object
• corpus provider conversion supports string or null content
• cachebench: no replay evidence

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/489df0c181e0e538. Report an issue: GitHub.