{"record":{"id":"6a1a19edcb96de48","repo":"JuliusBrussee/caveman","slug":"cache-replay-verifier-process-failed","errorCode":null,"errorMessage":"cache-replay: verifier process failed","messagePattern":"cache-replay: verifier process failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cacheengine/cmd/cache-replay/main.go","lineNumber":122,"sourceCode":"\tcommand := exec.CommandContext(verifyCtx, verifier.path, verifier.args...)\n\tinputReader, inputWriter := io.Pipe()\n\tencodeDone := make(chan error, 1)\n\tgo func() {\n\t\terr := json.NewEncoder(inputWriter).Encode(verificationInput)\n\t\t_ = inputWriter.CloseWithError(err)\n\t\tencodeDone <- err\n\t}()\n\tcommand.Stdin = inputReader\n\tcommand.Env = append([]string(nil), verifier.environment...)\n\tstdout := &boundedBuffer{max: verifier.maxOutputBytes}\n\tstderr := &boundedBuffer{max: 64 << 10}\n\tcommand.Stdout = stdout\n\tcommand.Stderr = stderr\n\trunErr := command.Run()\n\t_ = inputReader.Close()\n\tencodeErr := <-encodeDone\n\tif runErr != nil {\n\t\treturn cachebench.TaskVerification{}, errors.New(\"cache-replay: verifier process failed\")\n\t}\n\tif encodeErr != nil {\n\t\treturn cachebench.TaskVerification{}, errors.New(\"cache-replay: could not encode verifier input\")\n\t}\n\treturn cachebench.ParseVerificationCommandOutput(stdout.Bytes(), input.Trace.RequestID)\n}\n\ntype boundedBuffer struct {\n\tbuffer bytes.Buffer\n\tmax    int64\n}\n\nfunc (buffer *boundedBuffer) Write(value []byte) (int, error) {\n\tif int64(buffer.buffer.Len())+int64(len(value)) > buffer.max {\n\t\tremaining := buffer.max - int64(buffer.buffer.Len())\n\t\tif remaining > 0 {\n\t\t\twritten, _ := buffer.buffer.Write(value[:remaining])\n\t\t\treturn written, errors.New(\"output limit exceeded\")","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cmd/cache-replay/main.go#L104-L140","documentation":"Returned by the cache-replay task verifier wrapper when command.Run() returns a non-nil error for the verifier subprocess. The child process either exited non-zero, was killed (e.g. by the -verifier-timeout deadline), or failed to start. The verifier's captured stderr is buffered in a bounded 64KiB buffer; the wrapper deliberately discards runErr detail and returns this fixed message, so the underlying exit status must be diagnosed from the child itself.","triggerScenarios":"Running cache-replay with -execute and a -verifier-command that exits non-zero, is not executable, is killed by the 5-minute default verifier-timeout, or whose interpreter/shebang is missing. Also fires when the verifier binary cannot execute due to permission bits or a missing runtime.","commonSituations":"Verifier script not chmod +x; verifier needs an interpreter path in its shebang; verifier crashes on unexpected input JSON; CI environment missing the verifier's runtime (python/node); timeout too tight for slow tasks.","solutions":["Run the verifier command manually with a sample input record to see its real exit error/stderr","Ensure the verifier path passed via -verifier-command is executable (chmod +x) and has a valid shebang","If the verifier is slow, raise -verifier-timeout above the default 5m","Check the child's exit reason (signal vs non-zero code) via a wrapper that logs os/exec details"],"exampleFix":"# before\ncache-replay -execute -verifier-command /opt/verify.sh ...\n\n# after\nchmod +x /opt/verify.sh\n# test it standalone first:\necho '{\"trace\": ...}' | /opt/verify.sh; echo exit=$?","handlingStrategy":"retry","validationCode":"info, err := os.Stat(verifierPath)\nif err != nil || !info.Mode().IsRegular() || info.Mode()&0111 == 0 {\n\treturn fmt.Errorf(\"verifier %s missing or not executable\", verifierPath)\n}","typeGuard":null,"tryCatchPattern":"verification, err := runVerifier(input)\nif err != nil {\n\tif err.Error() == \"cache-replay: verifier process failed\" {\n\t\t// inspect the child manually; wrapper intentionally hides exit detail\n\t\treturn verification, fmt.Errorf(\"verifier crashed or timed out; run %s on a sample input to diagnose\", verifierPath)\n\t}\n\treturn verification, err\n}","preventionTips":["Smoke-test the verifier standalone in the same container/image CI uses","Set -verifier-timeout with headroom over the slowst task, and make the verifier stream progress to stderr (bounded to 64KiB)"],"tags":["go","subprocess","cli","verification"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}