{"record":{"id":"489df0c181e0e538","repo":"JuliusBrussee/caveman","slug":"cachebench-verifier-returned-invalid-json","errorCode":null,"errorMessage":"cachebench: verifier returned invalid JSON","messagePattern":"cachebench: verifier returned invalid JSON","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/replay.go","lineNumber":58,"sourceCode":"type VerificationCommandOutput struct {\n\tSchema    string          `json:\"schema\"`\n\tRequestID string          `json:\"request_id\"`\n\tPassed    bool            `json:\"passed\"`\n\tVerifier  string          `json:\"verifier\"`\n\tEvidence  json.RawMessage `json:\"evidence\"`\n}\n\n// ParseVerificationCommandOutput validates and binds external grader evidence.\nfunc ParseVerificationCommandOutput(raw []byte, requestID string) (TaskVerification, error) {\n\traw = bytes.TrimSpace(raw)\n\tif !validUniqueJSONObject(raw) {\n\t\treturn TaskVerification{}, errors.New(\"cachebench: verifier returned duplicate or invalid JSON\")\n\t}\n\tdecoder := json.NewDecoder(bytes.NewReader(raw))\n\tdecoder.DisallowUnknownFields()\n\tvar result VerificationCommandOutput\n\tif decoder.Decode(&result) != nil {\n\t\treturn TaskVerification{}, errors.New(\"cachebench: verifier returned invalid JSON\")\n\t}\n\tvar trailing any\n\tif decoder.Decode(&trailing) != io.EOF || result.Schema != VerificationSchema || result.RequestID != requestID || !validBoundedText(result.Verifier, 256, false) || len(result.Evidence) == 0 || !json.Valid(result.Evidence) || bytes.Equal(bytes.TrimSpace(result.Evidence), []byte(\"null\")) {\n\t\treturn TaskVerification{}, errors.New(\"cachebench: verifier returned incomplete or mismatched evidence\")\n\t}\n\treturn TaskVerification{Passed: result.Passed, Verifier: result.Verifier, Evidence: append([]byte(nil), raw...)}, nil\n}\n\n// ReplayLimits bounds paid population, schedule, and concurrency.\ntype ReplayLimits struct {\n\tMaxRequests             int\n\tMaxDeclaredBilledTokens int64\n\tMaxGap                  time.Duration\n\tMaxScheduleDrift        time.Duration\n\tMaxConcurrency          int\n\tRequireGroundedTiming   bool\n\tRequireProviderTokens   bool\n}","sourceCodeStart":40,"sourceCodeEnd":76,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/replay.go#L40-L76","documentation":"After the uniqueness check passes, the output must decode into VerificationCommandOutput with DisallowUnknownFields. A decode failure (malformed JSON, wrong types, or unknown fields) yields 'verifier returned invalid JSON' — the payload was a unique object but not a schema-conforming one.","triggerScenarios":"Grader output missing required structure, wrong field types (passed as string), or containing extra fields not in the schema (DisallowUnknownFields rejects them). Also genuinely broken JSON that still passed the earlier top-level object check is unlikely, so unknown fields and type mismatches dominate.","commonSituations":"Grader schema drift after upgrading cachebench (new/renamed fields); verifier adding a convenience field like \"notes\" at top level; passing \"evidence\" as a string instead of raw JSON.","solutions":["Align the verifier's output struct with VerificationCommandOutput: schema, request_id, passed, verifier, evidence only","Nest any extra information inside the evidence object rather than adding top-level fields","Pin/test the grader output against ParseVerificationCommandOutput in CI so drift is caught before replay"],"exampleFix":"// before\n{\"schema\":\"...\",\"request_id\":\"r1\",\"passed\":true,\"verifier\":\"v\",\"evidence\":{},\"note\":\"extra\"} // unknown field\n\n// after\n{\"schema\":\"...\",\"request_id\":\"r1\",\"passed\":true,\"verifier\":\"v\",\"evidence\":{\"note\":\"extra\"}}","handlingStrategy":"try-catch","validationCode":"var probe struct {\n    Schema string `json:\"schema\"`\n    RequestID string `json:\"request_id\"`\n    Passed bool `json:\"passed\"`\n    Verifier string `json:\"verifier\"`\n    Evidence json.RawMessage `json:\"evidence\"`\n}\ndec := json.NewDecoder(bytes.NewReader(raw))\ndec.DisallowUnknownFields()\nif dec.Decode(&probe) != nil {\n    return errors.New(\"grader output does not match verification schema\")\n}","typeGuard":"func matchesVerificationSchema(raw []byte) bool {\n    dec := json.NewDecoder(bytes.NewReader(raw))\n    dec.DisallowUnknownFields()\n    var out struct {\n        Schema string `json:\"schema\"`; RequestID string `json:\"request_id\"`\n        Passed bool `json:\"passed\"`; Verifier string `json:\"verifier\"`\n        Evidence json.RawMessage `json:\"evidence\"`\n    }\n    return dec.Decode(&out) == nil\n}","tryCatchPattern":"if _, err := cachebench.ParseVerificationCommandOutput(raw, id); err != nil {\n    if err.Error() == \"cachebench: verifier returned invalid JSON\" {\n        // diff grader fields against the four allowed keys; move extras into evidence\n    }\n    return err\n}","preventionTips":["Keep the grader's top-level fields exactly schema/request_id/passed/verifier/evidence","Put extra metadata inside the evidence object","Contract-test the grader against ParseVerificationCommandOutput in CI"],"tags":["go","replay","verification","json","schema"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}