What causes "cave_input_too_large: stdin exceeds %d bytes" in JuliusBrussee/caveman?

Split the input into chunks smaller than the engine's stdin limit and process each separately.

cave_input_too_large

cave_input_too_large

Error message

cave_input_too_large: stdin exceeds %d bytes

What it means

readBoundedInput reads stdin with a LimitReader of maxBytes+1 and rejects input larger than maxStdinBytes with the coded error "cave_input_too_large: stdin exceeds %d bytes". The engine reads inputs into memory, so oversized payloads are refused up front (error code cave_input_too_large) instead of risking OOM.

Solutions

Split the input into chunks smaller than the engine's stdin limit and process each separately.
Compress or trim the payload upstream before invoking the engine.
Check the documented maxStdinBytes value and measure your input size (`wc -c`) before calling.
If you control deployment, raise the configured maxStdinBytes limit deliberately.

Example fix

// before
cat huge.json | caveman-engine compress
// after
split -b 1m huge.json chunk_
for f in chunk_*; do caveman-engine compress < "$f"; done

Defensive patterns

Strategy: validation

Validate before calling

info, _ := os.Stdin.Stat()
if info.Mode()&os.ModeCharDevice == 0 {
    n, _ := io.Copy(io.Discard, io.TeeReader(os.Stdin, io.MultiWriter())) // pre-measure or use wc -c upstream
}

Try / catch

out, err := run(...)
if err != nil && strings.Contains(err.Error(), "cave_input_too_large") {
    return fmt.Errorf("split input below %d bytes and retry", maxStdinBytes)
}

Prevention

Measure input size (`wc -c`) before invoking the engine
Chunk large datasets before streaming to stdin
Document maxStdinBytes in your pipeline config

When it happens

Trigger: runCompress, runDetect, or runTOON calls readBoundedInput(os.Stdin, maxStdinBytes) and the piped/redirected stdin length exceeds maxStdinBytes — detected because LimitReader yielded maxBytes+1 bytes.

Common situations: Piping a very large log or dataset into the compress/detect/TOON verbs; an upstream agent forwarding an unbounded document; misjudging the byte limit when batching many files into one stream.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/cd8c4ecab8594864. Report an issue: GitHub.

Appendix: source

Thrown at engine/cmd/caveman-engine/main.go:239

		out, err = toonDecodeBytes(input)
	default:
		fatal("usage: caveman-engine toon encode|decode")
	}
	if err != nil {
		fatal("toon %s: %v", args[0], err)
	}
	if _, err := os.Stdout.Write(out); err != nil {
		fatal("write stdout: %v", err)
	}
}

func readBoundedInput(r io.Reader, maxBytes int64) ([]byte, error) {
	input, err := io.ReadAll(io.LimitReader(r, maxBytes+1))
	if err != nil {
		return nil, err
	}
	if int64(len(input)) > maxBytes {
		return nil, fmt.Errorf("cave_input_too_large: stdin exceeds %d bytes", maxBytes)
	}
	return input, nil
}

func runPixel(args []string) {
	if len(args) < 1 {
		pixelUsage()
		os.Exit(2)
	}
	switch args[0] {
	case "render":
		runPixelRender(args[1:])
	case "simulate":
		runPixelSimulate(args[1:])
	case "help", "--help", "-h":
		pixelUsage()
	default:
		fatal("usage: caveman-engine pixel render|simulate")

View on GitHub (pinned to 3ee70a1026)