JuliusBrussee/caveman · error

cachebench: verifier returned duplicate or invalid JSON

Error message

cachebench: verifier returned duplicate or invalid JSON

What it means

ParseVerificationCommandOutput first requires the external grader's raw stdout to be a single valid, duplicate-free JSON object (checked by validUniqueJSONObject). Duplicate keys or non-object JSON are rejected before decoding because they make evidence ambiguous — two values for 'passed' cannot be trusted.

Source

Thrown at cacheengine/cachebench/replay.go:52

	OriginalRequest  json.RawMessage `json:"original_request"`
	OptimizedRequest json.RawMessage `json:"optimized_request"`
	ProviderResponse json.RawMessage `json:"provider_response"`
}

// VerificationCommandOutput is strict external task-grader response.
type VerificationCommandOutput struct {
	Schema    string          `json:"schema"`
	RequestID string          `json:"request_id"`
	Passed    bool            `json:"passed"`
	Verifier  string          `json:"verifier"`
	Evidence  json.RawMessage `json:"evidence"`
}

// ParseVerificationCommandOutput validates and binds external grader evidence.
func ParseVerificationCommandOutput(raw []byte, requestID string) (TaskVerification, error) {
	raw = bytes.TrimSpace(raw)
	if !validUniqueJSONObject(raw) {
		return TaskVerification{}, errors.New("cachebench: verifier returned duplicate or invalid JSON")
	}
	decoder := json.NewDecoder(bytes.NewReader(raw))
	decoder.DisallowUnknownFields()
	var result VerificationCommandOutput
	if decoder.Decode(&result) != nil {
		return TaskVerification{}, errors.New("cachebench: verifier returned invalid JSON")
	}
	var trailing any
	if decoder.Decode(&trailing) != io.EOF || result.Schema != VerificationSchema || result.RequestID != requestID || !validBoundedText(result.Verifier, 256, false) || len(result.Evidence) == 0 || !json.Valid(result.Evidence) || bytes.Equal(bytes.TrimSpace(result.Evidence), []byte("null")) {
		return TaskVerification{}, errors.New("cachebench: verifier returned incomplete or mismatched evidence")
	}
	return TaskVerification{Passed: result.Passed, Verifier: result.Verifier, Evidence: append([]byte(nil), raw...)}, nil
}

// ReplayLimits bounds paid population, schedule, and concurrency.
type ReplayLimits struct {
	MaxRequests             int
	MaxDeclaredBilledTokens int64

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Make the verifier print exactly one JSON object and route any logging to stderr
2. Regenerate the grader output with an encoder that cannot emit duplicate keys (encoding/json Marshal, jq -c)
3. Capture and inspect the raw bytes passed to ParseVerificationCommandOutput when debugging

Example fix

// before
fmt.Println(`{"passed":true,"request_id":"r1"}`)
fmt.Println(`{"passed":false,"request_id":"r1"}`) // duplicate output

// after
enc, _ := json.Marshal(result) // single object
os.Stdout.Write(append(enc, '\n'))
log.Println("extra info") // stderr only

if !validUniqueJSONObject(bytes.TrimSpace(raw)) {
    return fmt.Errorf("verifier stdout is not a single unique-key JSON object")
}
// only then:
v, err := cachebench.ParseVerificationCommandOutput(raw, requestID)

func isSingleJSONObject(b []byte) bool {
    t := bytes.TrimSpace(b)
    if len(t) == 0 || t[0] != '{' { return false }
    var m map[string]json.RawMessage
    return json.Unmarshal(t, &m) == nil
}

if _, err := cachebench.ParseVerificationCommandOutput(raw, id); err != nil {
    if err.Error() == "cachebench: verifier returned duplicate or invalid JSON" {
        // rerun grader with stdout logging redirected to stderr
    }
}

When it happens

Trigger: An external verification command emits JSON with repeated keys (e.g. {"passed":true,...,"passed":false}) or emits an array/scalar/concatenated objects. The parse fails on the uniqueness/shape check before field binding.

Common situations: Grader script printing two JSON objects (one partial, one final) to stdout; log lines interleaving with the JSON payload; hand-written grader using map iteration that can serialize duplicate keys via streaming writers.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• cachebench: verifier returned invalid JSON
• cachebench: verifier returned incomplete or mismatched evide
• corpus provider conversion supports string or null content
• cachebench: no replay evidence
• cachebench: replay token summary overflow

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/8bd9ca598c52793a. Report an issue: GitHub.