What causes "%w: %d bytes (limit %d)" in JuliusBrussee/caveman?

Enforce an 8 MiB body limit upstream (request size middleware) so oversized inputs are rejected with 413 before reaching redaction.

JuliusBrussee/caveman · error · ErrBodyTooLarge

%w: %d bytes (limit %d)

Error message

%w: %d bytes (limit %d)

What it means

redact.Payload refuses to process a body larger than MaxPayloadBytes (8 MiB, payload.go:45) before doing any regex work. The error wraps ErrBodyTooLarge and reports the actual size and the limit. The cap exists so a hostile or oversized upload cannot turn the redaction pass (many regexes over the whole body) into a CPU/memory amplification vector.

Source

Thrown at shared/platform/redact/payload.go:476

//
// rules are the organization's enabled redaction_rules rows. They are additive
// only — the built-in floor runs first and cannot be disabled, shadowed, or
// reordered by anything a tenant configures.
//
// The returned slice ALIASES body when no rule fired; body is never mutated
// (TestPayloadDoesNotMutateInput), so this is safe to read, but a caller that
// intends to mutate the result in place must copy it first.
//
// Payload does not judge how BROAD a rule is. A pattern like ".+" compiles,
// matches, and collapses the whole body into one placeholder — a self-inflicted
// loss of corpus value, not a leak, and the only pattern shapes rejected here
// are the ones that are unusable rather than merely greedy. Rejecting breadth
// belongs at rule-write time, where an operator is present to see the error;
// there is no such surface yet (see the H1 report's concern on the missing
// redaction_rules validator).
func Payload(body []byte, rules []Rule) ([]byte, RedactionReport, error) {
	if len(body) > MaxPayloadBytes {
		return nil, RedactionReport{}, fmt.Errorf("%w: %d bytes (limit %d)", ErrBodyTooLarge, len(body), MaxPayloadBytes)
	}
	orgRules, fingerprint, err := compileOrgRules(rules)
	if err != nil {
		return nil, RedactionReport{}, err
	}

	sum := sha256.Sum256([]byte(builtinRuleSetFingerprint + "\x00" + fingerprint))
	report := RedactionReport{
		RuleSetHash: hex.EncodeToString(sum[:16]),
		BytesIn:     len(body),
	}

	out := body
	// The prescreen is derived ONCE, then re-derived only after a rule whose
	// replacement can introduce a needle (see replIntroducesNeedle). Of the
	// built-ins only three can, and only when they actually fire.
	lowered := asciiLower(out)
	// Built-ins first: whatever an org rule does afterwards, it acts on a body

View on GitHub (pinned to 27d5a3981a)

Solutions

Enforce an 8 MiB body limit upstream (request size middleware) so oversized inputs are rejected with 413 before reaching redaction.
If legitimate payloads exceed 8 MiB, redact in chunks that respect record boundaries (e.g. per log line) rather than raising the global cap — the cap is a DoS guard.
For huge text files, stream-filter with the same rule set instead of using Payload, accepting per-chunk rather than whole-body guarantees.

Example fix

// before
red, rep, err := redact.Payload(body, rules) // body is 50 MiB

// after
const max = redact.MaxPayloadBytes
if len(body) > max {
    http.Error(w, fmt.Sprintf("body exceeds %d bytes", max), http.StatusRequestEntityTooLarge)
    return
}
red, rep, err := redact.Payload(body, rules)

Defensive patterns

Strategy: validation

Validate before calling

if len(body) > redact.MaxPayloadBytes {
    return fmt.Errorf("rejecting body of %d bytes (limit %d)", len(body), redact.MaxPayloadBytes)
}
out, rep, err := redact.Payload(body, rules)

Try / catch

out, rep, err := redact.Payload(body, rules)
if errors.Is(err, redact.ErrBodyTooLarge) {
    // reject upstream (413) or chunk the body along record boundaries; do not silently pass through unredacted
}

Prevention

Enforce the same 8 MiB cap at ingestion (HTTP middleware) so rejection happens at the boundary.
Never 'recover' by redacting nothing — a too-large body must fail closed, not pass through.
Chunk large text along line/record boundaries instead of raising the global cap.

When it happens

Trigger: Calling redact.Payload(body, rules) with len(body) > 8<<20 — e.g. a captured HTTP request/response body, a log blob, or an artifact fed into the redaction pipeline wholesale.

Common situations: Redacting large API responses or binary-ish payloads that slipped past upstream size limits; a proxy/capture tool that buffers full bodies before redaction; increasing upload limits elsewhere in the app without updating this constant's expectations.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/f12bb1f739aa63a1. Report an issue: GitHub.