{"record":{"id":"67a5d62304dd2101","repo":"JuliusBrussee/caveman","slug":"cachebench-verifier-returned-incomplete-or-mismat","errorCode":null,"errorMessage":"cachebench: verifier returned incomplete or mismatched evidence","messagePattern":"cachebench: verifier returned incomplete or mismatched evidence","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/replay.go","lineNumber":62,"sourceCode":"\tVerifier  string          `json:\"verifier\"`\n\tEvidence  json.RawMessage `json:\"evidence\"`\n}\n\n// ParseVerificationCommandOutput validates and binds external grader evidence.\nfunc ParseVerificationCommandOutput(raw []byte, requestID string) (TaskVerification, error) {\n\traw = bytes.TrimSpace(raw)\n\tif !validUniqueJSONObject(raw) {\n\t\treturn TaskVerification{}, errors.New(\"cachebench: verifier returned duplicate or invalid JSON\")\n\t}\n\tdecoder := json.NewDecoder(bytes.NewReader(raw))\n\tdecoder.DisallowUnknownFields()\n\tvar result VerificationCommandOutput\n\tif decoder.Decode(&result) != nil {\n\t\treturn TaskVerification{}, errors.New(\"cachebench: verifier returned invalid JSON\")\n\t}\n\tvar trailing any\n\tif decoder.Decode(&trailing) != io.EOF || result.Schema != VerificationSchema || result.RequestID != requestID || !validBoundedText(result.Verifier, 256, false) || len(result.Evidence) == 0 || !json.Valid(result.Evidence) || bytes.Equal(bytes.TrimSpace(result.Evidence), []byte(\"null\")) {\n\t\treturn TaskVerification{}, errors.New(\"cachebench: verifier returned incomplete or mismatched evidence\")\n\t}\n\treturn TaskVerification{Passed: result.Passed, Verifier: result.Verifier, Evidence: append([]byte(nil), raw...)}, nil\n}\n\n// ReplayLimits bounds paid population, schedule, and concurrency.\ntype ReplayLimits struct {\n\tMaxRequests             int\n\tMaxDeclaredBilledTokens int64\n\tMaxGap                  time.Duration\n\tMaxScheduleDrift        time.Duration\n\tMaxConcurrency          int\n\tRequireGroundedTiming   bool\n\tRequireProviderTokens   bool\n}\n\n// ReplayPreflight is zero-network validation and declared-budget summary.\ntype ReplayPreflight struct {\n\tRequests                          int      `json:\"requests\"`","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/replay.go#L44-L80","documentation":"The final gate in ParseVerificationCommandOutput: no trailing JSON after the object, schema equal to VerificationSchema, request_id equal to the expected one, verifier a bounded non-empty text (<=256), and evidence present, valid JSON, and not null. Any mismatch means the evidence cannot be attributed to this request and is rejected.","triggerScenarios":"request_id in the grader output differing from the request being verified; missing/empty evidence or literal null evidence; schema field not matching the expected VerificationSchema constant; verifier name blank or over 256 chars; concatenated trailing JSON.","commonSituations":"Grader runs concurrently and outputs interleaved by request, so IDs cross; passing the wrong request_id argument when parsing; grader emitting \"evidence\": null on skip paths; schema version mismatch after an upgrade.","solutions":["Pass the exact request_id being verified and have the grader echo it unchanged","Ensure evidence is always a non-null JSON value (empty object {} is acceptable, null is not)","Keep schema pinned to VerificationSchema and update both sides together on version bumps","Run graders one-at-a-time per request or prefix outputs per goroutine to avoid interleaving"],"exampleFix":"// before\nv, err := ParseVerificationCommandOutput(raw, otherRequestID) // mismatch\n// grader emitted \"evidence\": null\n\n// after\nv, err := ParseVerificationCommandOutput(raw, req.ID)\n// grader emits \"evidence\": {} minimum","handlingStrategy":"validation","validationCode":"if v.RequestID != requestID || v.Schema != expectedSchema || len(bytes.TrimSpace(v.Evidence)) == 0 || bytes.Equal(bytes.TrimSpace(v.Evidence), []byte(\"null\")) {\n    return errors.New(\"evidence not attributable to this request\")\n}\nverification, err := cachebench.ParseVerificationCommandOutput(raw, requestID)","typeGuard":"func evidenceAttributable(raw []byte, requestID, schema string) bool {\n    var v struct {\n        Schema string `json:\"schema\"`; RequestID string `json:\"request_id\"`\n        Verifier string `json:\"verifier\"`; Evidence json.RawMessage `json:\"evidence\"`\n    }\n    if json.Unmarshal(raw, &v) != nil { return false }\n    return v.Schema == schema && v.RequestID == requestID && v.Verifier != \"\" && len(v.Verifier) <= 256 &&\n        len(v.Evidence) > 0 && !bytes.Equal(bytes.TrimSpace(v.Evidence), []byte(\"null\"))\n}","tryCatchPattern":"if _, err := cachebench.ParseVerificationCommandOutput(raw, req.ID); err != nil {\n    if err.Error() == \"cachebench: verifier returned incomplete or mismatched evidence\" {\n        // verify request_id plumbing and evidence non-null before rerunning the grader\n    }\n}","preventionTips":["Echo the request_id argument verbatim in grader output; assert equality on both sides","Never emit null evidence — use {} for empty","Upgrade grader and cachebench schema versions together","Serialize concurrent grader output per request to prevent cross-attribution"],"tags":["go","replay","verification","correlation"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}