valyala/fasthttp · error

too much data unzstd: %d

Error message

too much data unzstd: %d

What it means

writeUnzstd decompresses a zstd payload into a writer limited by maxBodySize. If the decompressed stream exceeds that limit, copyZeroAllocWithLimit stops and the function reports 'too much data unzstd' instead of returning silently truncated output.

Source

Thrown at zstd.go:170

	switch dst := w.(type) {
	case *byteSliceWriter:
		dst.b = slices.Grow(dst.b, estimatedDecompressedSize)
	case *bytebufferpool.ByteBuffer:
		dst.B = slices.Grow(dst.B, estimatedDecompressedSize)
	case *bytes.Buffer:
		dst.Grow(estimatedDecompressedSize)
	}

	r := &byteSliceReader{b: p}
	zr, err := acquireZstdReader(r)
	if err != nil {
		return 0, err
	}
	n, err := copyZeroAllocWithLimit(w, zr, maxBodySize)
	releaseZstdReader(zr)
	nn := int(n)
	if int64(nn) != n {
		return 0, fmt.Errorf("too much data unzstd: %d", n)
	}
	return nn, err
}

func estimateUnzstdSize(p []byte) int {
	// Somewhat reasonable and conservative expectation of compression factor of 2
	sizeHint := 2 * len(p)

	// We look for the first non-skippable header
	var header zstd.Header
	for {
		if err := header.Decode(p); err != nil {
			break
		}
		if !header.Skippable {
			break
		}
		skippedBytes := header.HeaderSize + int(header.SkippableSize)

View on GitHub (pinned to c96f600972)

Solutions

1. Raise the configured limit (e.g. server.MaxRequestBodySize or maxBodySize argument) above the largest expected uncompressed payload
2. Stream the decompressed body to disk/network instead of buffering through a limited writer
3. Compress data more aggressively server-side or paginate responses

Example fix

// before
s.MaxRequestBodySize = 4 * 1024 * 1024 // 4 MiB
// after
s.MaxRequestBodySize = 64 * 1024 * 1024 // 64 MiB, fits largest unzstd payload

// before decompressing, check the configured limit
cap := int64(64 * 1024 * 1024)
if int64(len(compressed)) > 0 && int64(len(compressed)) > maxBodySize {
    return errors.New("payload exceeds maxBodySize; raise limit before unzstd")
}
_ = cap

n, err := fasthttp.WriteUnzstd(dst, src)
if err != nil {
    if strings.HasPrefix(err.Error(), "too much data unzstd") {
        // retry with a larger buffer/stream to file
        return streamUnzstdToFile(src)
    }
    return err
}

When it happens

Trigger: Calling Unzstd/UnzstdBytes-style helpers (or fasthttp body decompression paths) on zstd content whose uncompressed size exceeds maxBodySize, typically fasthttp's default 4 GiB cap or a smaller user-set limit (Server.MaxRequestBodySize / client MaxResponseBodySize).

Common situations: API responses or uploaded files much larger than the configured body limit; limits lowered for security but legitimate large payloads still arriving; misjudged estimateUnzstdSize expectations.

Related errors

• fasthttp: no args value for the given key
• fasthttp: pipelined requests' queue has been overflowed. inc
• pipeline connection has been stopped
• fasthttp: no cookies found
• fasthttp: invalid cookie value

AI-assisted analysis of valyala/fasthttp@c96f600972 (2026-08-31). Data as JSON: /api/errors/f60a380e5521cccc. Report an issue: GitHub.