JuliusBrussee/caveman · error

cache-replay: verifier process failed

Error message

cache-replay: verifier process failed

What it means

Returned by the cache-replay task verifier wrapper when command.Run() returns a non-nil error for the verifier subprocess. The child process either exited non-zero, was killed (e.g. by the -verifier-timeout deadline), or failed to start. The verifier's captured stderr is buffered in a bounded 64KiB buffer; the wrapper deliberately discards runErr detail and returns this fixed message, so the underlying exit status must be diagnosed from the child itself.

Source

Thrown at cacheengine/cmd/cache-replay/main.go:122

	command := exec.CommandContext(verifyCtx, verifier.path, verifier.args...)
	inputReader, inputWriter := io.Pipe()
	encodeDone := make(chan error, 1)
	go func() {
		err := json.NewEncoder(inputWriter).Encode(verificationInput)
		_ = inputWriter.CloseWithError(err)
		encodeDone <- err
	}()
	command.Stdin = inputReader
	command.Env = append([]string(nil), verifier.environment...)
	stdout := &boundedBuffer{max: verifier.maxOutputBytes}
	stderr := &boundedBuffer{max: 64 << 10}
	command.Stdout = stdout
	command.Stderr = stderr
	runErr := command.Run()
	_ = inputReader.Close()
	encodeErr := <-encodeDone
	if runErr != nil {
		return cachebench.TaskVerification{}, errors.New("cache-replay: verifier process failed")
	}
	if encodeErr != nil {
		return cachebench.TaskVerification{}, errors.New("cache-replay: could not encode verifier input")
	}
	return cachebench.ParseVerificationCommandOutput(stdout.Bytes(), input.Trace.RequestID)
}

type boundedBuffer struct {
	buffer bytes.Buffer
	max    int64
}

func (buffer *boundedBuffer) Write(value []byte) (int, error) {
	if int64(buffer.buffer.Len())+int64(len(value)) > buffer.max {
		remaining := buffer.max - int64(buffer.buffer.Len())
		if remaining > 0 {
			written, _ := buffer.buffer.Write(value[:remaining])
			return written, errors.New("output limit exceeded")

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Run the verifier command manually with a sample input record to see its real exit error/stderr
2. Ensure the verifier path passed via -verifier-command is executable (chmod +x) and has a valid shebang
3. If the verifier is slow, raise -verifier-timeout above the default 5m
4. Check the child's exit reason (signal vs non-zero code) via a wrapper that logs os/exec details

Example fix

# before
cache-replay -execute -verifier-command /opt/verify.sh ...

# after
chmod +x /opt/verify.sh
# test it standalone first:
echo '{"trace": ...}' | /opt/verify.sh; echo exit=$?

info, err := os.Stat(verifierPath)
if err != nil || !info.Mode().IsRegular() || info.Mode()&0111 == 0 {
	return fmt.Errorf("verifier %s missing or not executable", verifierPath)
}

verification, err := runVerifier(input)
if err != nil {
	if err.Error() == "cache-replay: verifier process failed" {
		// inspect the child manually; wrapper intentionally hides exit detail
		return verification, fmt.Errorf("verifier crashed or timed out; run %s on a sample input to diagnose", verifierPath)
	}
	return verification, err
}

When it happens

Trigger: Running cache-replay with -execute and a -verifier-command that exits non-zero, is not executable, is killed by the 5-minute default verifier-timeout, or whose interpreter/shebang is missing. Also fires when the verifier binary cannot execute due to permission bits or a missing runtime.

Common situations: Verifier script not chmod +x; verifier needs an interpreter path in its shebang; verifier crashes on unexpected input JSON; CI environment missing the verifier's runtime (python/node); timeout too tight for slow tasks.

Related errors

• cache-replay: could not encode verifier input
• empty version output
• Claude call failed: {e.stderr}
• cachebench: verifier returned duplicate or invalid JSON
• cachebench: verifier returned invalid JSON

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/6a1a19edcb96de48. Report an issue: GitHub.