JuliusBrussee/caveman · error

gzip reader

Error message

gzip reader: %w

What it means

DecompressLimit could not construct a gzip.Reader from the supplied data, meaning the payload is not a valid gzip stream (bad magic number, truncated header, or corrupt format). The error is wrapped with "gzip reader: " and decoding never starts. This is the earliest gzip-specific failure in DecompressLimit and indicates malformed input rather than a size problem.

Solutions

1. Verify the stored codec label matches the actual bytes (gzip streams start with 0x1f 0x8b); re-label or re-compress mismatched objects.
2. Check the object's integrity: re-download or compare checksums to rule out truncation/corruption during transfer.
3. Confirm the writer path used Compress(CompressionGzip, ...) and completed without error before the data was stored.
4. If input may be uncompressed, sniff the magic bytes and call Decompress with CompressionNone when appropriate.
5. Re-upload the object from source if the stored blob is irrecoverably corrupt.

Example fix

// before
out, err := objectstore.DecompressLimit("gzip", blob, limit) // gzip reader: gzip: invalid header
// after
if len(blob) >= 2 && blob[0] == 0x1f && blob[1] == 0x8b {
    out, err = objectstore.DecompressLimit("gzip", blob, limit)
} else {
    // codec label mismatch or corrupt object: fetch metadata or re-fetch object
    out, err = objectstore.DecompressLimit(storedCodecFromMetadata, blob, limit)
}

func isGzip(b []byte) bool {
    return len(b) >= 2 && b[0] == 0x1f && b[1] == 0x8b
}

if !isGzip(data) {
    return fmt.Errorf("payload is not gzip (len=%d, magic=%x)", len(data), data[:min(2, len(data))])
}
out, err := objectstore.DecompressLimit("gzip", data, limit)

func isGzip(b []byte) bool {
    return len(b) >= 2 && b[0] == 0x1f && b[1] == 0x8b
}

if err != nil {
    if strings.HasPrefix(err.Error(), "gzip reader:") {
        // payload is not a valid gzip stream: check stored codec metadata, re-fetch object
        return fmt.Errorf("corrupt or mislabeled gzip object: %w", err)
    }
    if strings.HasPrefix(err.Error(), "gzip read:") {
        // truncated mid-stream
        return fmt.Errorf("truncated gzip payload: %w", err)
    }
    return err
}

When it happens

Trigger: Calling objectstore.Decompress/DecompressLimit with CompressionGzip on data that was never gzip-compressed (plain bytes, zstd frames, JSON), data truncated during storage/transfer, or a corrupted/stale blob whose stored codec label says gzip but whose bytes are something else.

Common situations: Codec metadata drifting from actual stored format after a migration, partial writes or failed uploads persisted, downloading an HTML/JSON error body instead of the compressed object, or hand-edited blobs.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• gzip close
• gzip
• invalid decompression limit
• openai compress: invalid splice range
• unknown compression codec

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/d3c68f783bc5afab. Report an issue: GitHub.

Appendix: source

Thrown at shared/platform/objectstore/compress.go:79

// DecompressLimit reverses Compress while refusing to produce more than
// maxBytes of plaintext. The limit is applied while decoding gzip and through
// zstd's decoder memory bound, so highly-compressible hostile payloads cannot
// first allocate their full expanded size and only then be rejected.
func DecompressLimit(codec string, data []byte, maxBytes int64) ([]byte, error) {
	if maxBytes < 0 || maxBytes > DefaultMaxDecompressedBytes {
		return nil, fmt.Errorf("invalid decompression limit %d", maxBytes)
	}
	switch codec {
	case CompressionNone, "":
		if int64(len(data)) > maxBytes {
			return nil, ErrDecompressedTooLarge
		}
		return data, nil
	case CompressionGzip:
		zr, err := gzip.NewReader(bytes.NewReader(data))
		if err != nil {
			return nil, fmt.Errorf("gzip reader: %w", err)
		}
		defer zr.Close()
		out, err := io.ReadAll(io.LimitReader(zr, maxBytes+1))
		if err != nil {
			return nil, fmt.Errorf("gzip read: %w", err)
		}
		if int64(len(out)) > maxBytes {
			return nil, ErrDecompressedTooLarge
		}
		return out, nil
	case CompressionZstd:
		// Zstd's smallest legal window is 1 KiB. Keep that parser minimum for
		// tiny valid payloads, then independently enforce the exact byte limit.
		memoryLimit := uint64(maxBytes)
		if memoryLimit < zstd.MinWindowSize {
			memoryLimit = zstd.MinWindowSize
		}
		zr, err := zstd.NewReader(nil,

View on GitHub (pinned to 3ee70a1026)