{"record":{"id":"8bd9ca598c52793a","repo":"JuliusBrussee/caveman","slug":"cachebench-verifier-returned-duplicate-or-invalid","errorCode":null,"errorMessage":"cachebench: verifier returned duplicate or invalid JSON","messagePattern":"cachebench: verifier returned duplicate or invalid JSON","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/replay.go","lineNumber":52,"sourceCode":"\tOriginalRequest  json.RawMessage `json:\"original_request\"`\n\tOptimizedRequest json.RawMessage `json:\"optimized_request\"`\n\tProviderResponse json.RawMessage `json:\"provider_response\"`\n}\n\n// VerificationCommandOutput is strict external task-grader response.\ntype VerificationCommandOutput struct {\n\tSchema    string          `json:\"schema\"`\n\tRequestID string          `json:\"request_id\"`\n\tPassed    bool            `json:\"passed\"`\n\tVerifier  string          `json:\"verifier\"`\n\tEvidence  json.RawMessage `json:\"evidence\"`\n}\n\n// ParseVerificationCommandOutput validates and binds external grader evidence.\nfunc ParseVerificationCommandOutput(raw []byte, requestID string) (TaskVerification, error) {\n\traw = bytes.TrimSpace(raw)\n\tif !validUniqueJSONObject(raw) {\n\t\treturn TaskVerification{}, errors.New(\"cachebench: verifier returned duplicate or invalid JSON\")\n\t}\n\tdecoder := json.NewDecoder(bytes.NewReader(raw))\n\tdecoder.DisallowUnknownFields()\n\tvar result VerificationCommandOutput\n\tif decoder.Decode(&result) != nil {\n\t\treturn TaskVerification{}, errors.New(\"cachebench: verifier returned invalid JSON\")\n\t}\n\tvar trailing any\n\tif decoder.Decode(&trailing) != io.EOF || result.Schema != VerificationSchema || result.RequestID != requestID || !validBoundedText(result.Verifier, 256, false) || len(result.Evidence) == 0 || !json.Valid(result.Evidence) || bytes.Equal(bytes.TrimSpace(result.Evidence), []byte(\"null\")) {\n\t\treturn TaskVerification{}, errors.New(\"cachebench: verifier returned incomplete or mismatched evidence\")\n\t}\n\treturn TaskVerification{Passed: result.Passed, Verifier: result.Verifier, Evidence: append([]byte(nil), raw...)}, nil\n}\n\n// ReplayLimits bounds paid population, schedule, and concurrency.\ntype ReplayLimits struct {\n\tMaxRequests             int\n\tMaxDeclaredBilledTokens int64","sourceCodeStart":34,"sourceCodeEnd":70,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/replay.go#L34-L70","documentation":"ParseVerificationCommandOutput first requires the external grader's raw stdout to be a single valid, duplicate-free JSON object (checked by validUniqueJSONObject). Duplicate keys or non-object JSON are rejected before decoding because they make evidence ambiguous — two values for 'passed' cannot be trusted.","triggerScenarios":"An external verification command emits JSON with repeated keys (e.g. {\"passed\":true,...,\"passed\":false}) or emits an array/scalar/concatenated objects. The parse fails on the uniqueness/shape check before field binding.","commonSituations":"Grader script printing two JSON objects (one partial, one final) to stdout; log lines interleaving with the JSON payload; hand-written grader using map iteration that can serialize duplicate keys via streaming writers.","solutions":["Make the verifier print exactly one JSON object and route any logging to stderr","Regenerate the grader output with an encoder that cannot emit duplicate keys (encoding/json Marshal, jq -c)","Capture and inspect the raw bytes passed to ParseVerificationCommandOutput when debugging"],"exampleFix":"// before\nfmt.Println(`{\"passed\":true,\"request_id\":\"r1\"}`)\nfmt.Println(`{\"passed\":false,\"request_id\":\"r1\"}`) // duplicate output\n\n// after\nenc, _ := json.Marshal(result) // single object\nos.Stdout.Write(append(enc, '\\n'))\nlog.Println(\"extra info\") // stderr only","handlingStrategy":"validation","validationCode":"if !validUniqueJSONObject(bytes.TrimSpace(raw)) {\n    return fmt.Errorf(\"verifier stdout is not a single unique-key JSON object\")\n}\n// only then:\nv, err := cachebench.ParseVerificationCommandOutput(raw, requestID)","typeGuard":"func isSingleJSONObject(b []byte) bool {\n    t := bytes.TrimSpace(b)\n    if len(t) == 0 || t[0] != '{' { return false }\n    var m map[string]json.RawMessage\n    return json.Unmarshal(t, &m) == nil\n}","tryCatchPattern":"if _, err := cachebench.ParseVerificationCommandOutput(raw, id); err != nil {\n    if err.Error() == \"cachebench: verifier returned duplicate or invalid JSON\" {\n        // rerun grader with stdout logging redirected to stderr\n    }\n}","preventionTips":["Direct all verifier logging to stderr; stdout carries exactly one JSON object","Build grader output with encoding/json Marshal (it cannot emit duplicate keys)","Lint grader scripts against double-printing results"],"tags":["go","replay","verification","json"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}