JuliusBrussee/caveman · error · ErrDecompressedTooLarge

objectstore: decompressed payload exceeds limit

Error message

objectstore: decompressed payload exceeds limit

What it means

ErrDecompressedTooLarge is a sentinel error returned by DecompressLimit (and the default decompress path for CompressionNone) when a payload would exceed the caller's maxBytes ceiling. It is raised before allocation blowup for compressed data and after a length check for uncompressed data, protecting against decompression-bomb attacks and accidental oversized reads.

Solutions

  1. Call DecompressLimit with the trusted plaintext size from your metadata rather than relying on the default
  2. If the data is genuinely needed, raise maxBytes deliberately for that call path
  3. Verify the object's expected size before fetch; reject oversized uploads at ingest time
  4. Check that CompressionNone payloads were stored correctly (an oversized raw payload may indicate a mis-tagged object)

Example fix

// before
data, err := objectstore.DecompressLimit(blob, objectstore.DefaultMaxDecompressedBytes)
// after
maxBytes := int64(obj.Metadata.PlaintextSize) // trusted, validated upstream
data, err := objectstore.DecompressLimit(blob, maxBytes)
Defensive patterns

Strategy: try-catch

Validate before calling

if int64(len(expectedPlaintext)) > maxBytes {
	return errors.New("payload exceeds decompression ceiling")
}
// prefer: maxBytes from trusted metadata, not the 128 MiB default

Try / catch

data, err := objectstore.DecompressLimit(blob, maxBytes)
if errors.Is(err, objectstore.ErrDecompressedTooLarge) {
	return fmt.Errorf("object exceeds %d bytes after decompression", maxBytes)
}
if err != nil {
	return err
}

Prevention

When it happens

Trigger: Decompressing a zstd payload whose expanded size exceeds maxBytes; passing an uncompressed (CompressionNone/"") payload longer than maxBytes; using DefaultMaxDecompressedBytes (128 MiB) implicitly and reading an object larger than that.

Common situations: User-supplied compressed uploads (decompression bombs), reading an old object stored before size limits were introduced, decompressing with the default limit instead of a trusted metadata-driven limit.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/2ab92b4d4c3d4200. Report an issue: GitHub.

Appendix: source

Thrown at shared/platform/objectstore/compress.go:26

	"io"

	"github.com/klauspost/compress/zstd"
)

// Compression names the supported codecs; they mirror the artifacts.compression
// CHECK constraint (none|gzip|zstd). An unknown codec fails closed.
const (
	CompressionNone = "none"
	CompressionGzip = "gzip"
	CompressionZstd = "zstd"
)

var (
	zstdEnc, _ = zstd.NewWriter(nil)

	// ErrDecompressedTooLarge is returned before a compressed payload can
	// expand beyond the caller's plaintext ceiling.
	ErrDecompressedTooLarge = errors.New("objectstore: decompressed payload exceeds limit")
)

// DefaultMaxDecompressedBytes protects callers that do not have trusted size
// metadata. Callers with an expected plaintext size should use
// DecompressLimit with the smaller, metadata-validated limit.
const DefaultMaxDecompressedBytes int64 = 128 << 20

// Compress encodes data with the named codec. "none" returns data unchanged.
func Compress(codec string, data []byte) ([]byte, error) {
	switch codec {
	case CompressionNone, "":
		return data, nil
	case CompressionGzip:
		var buf bytes.Buffer
		zw := gzip.NewWriter(&buf)
		if _, err := zw.Write(data); err != nil {
			return nil, fmt.Errorf("gzip: %w", err)
		}

View on GitHub (pinned to 3ee70a1026)