{"record":{"id":"f60a380e5521cccc","repo":"valyala/fasthttp","slug":"too-much-data-unzstd-d","errorCode":null,"errorMessage":"too much data unzstd: %d","messagePattern":"too much data unzstd: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"zstd.go","lineNumber":170,"sourceCode":"\tswitch dst := w.(type) {\n\tcase *byteSliceWriter:\n\t\tdst.b = slices.Grow(dst.b, estimatedDecompressedSize)\n\tcase *bytebufferpool.ByteBuffer:\n\t\tdst.B = slices.Grow(dst.B, estimatedDecompressedSize)\n\tcase *bytes.Buffer:\n\t\tdst.Grow(estimatedDecompressedSize)\n\t}\n\n\tr := &byteSliceReader{b: p}\n\tzr, err := acquireZstdReader(r)\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\tn, err := copyZeroAllocWithLimit(w, zr, maxBodySize)\n\treleaseZstdReader(zr)\n\tnn := int(n)\n\tif int64(nn) != n {\n\t\treturn 0, fmt.Errorf(\"too much data unzstd: %d\", n)\n\t}\n\treturn nn, err\n}\n\nfunc estimateUnzstdSize(p []byte) int {\n\t// Somewhat reasonable and conservative expectation of compression factor of 2\n\tsizeHint := 2 * len(p)\n\n\t// We look for the first non-skippable header\n\tvar header zstd.Header\n\tfor {\n\t\tif err := header.Decode(p); err != nil {\n\t\t\tbreak\n\t\t}\n\t\tif !header.Skippable {\n\t\t\tbreak\n\t\t}\n\t\tskippedBytes := header.HeaderSize + int(header.SkippableSize)","sourceCodeStart":152,"sourceCodeEnd":188,"githubUrl":"https://github.com/valyala/fasthttp/blob/c96f600972c6f4a7a30d664257b340ebe9d60124/zstd.go#L152-L188","documentation":"writeUnzstd decompresses a zstd payload into a writer limited by maxBodySize. If the decompressed stream exceeds that limit, copyZeroAllocWithLimit stops and the function reports 'too much data unzstd' instead of returning silently truncated output.","triggerScenarios":"Calling Unzstd/UnzstdBytes-style helpers (or fasthttp body decompression paths) on zstd content whose uncompressed size exceeds maxBodySize, typically fasthttp's default 4 GiB cap or a smaller user-set limit (Server.MaxRequestBodySize / client MaxResponseBodySize).","commonSituations":"API responses or uploaded files much larger than the configured body limit; limits lowered for security but legitimate large payloads still arriving; misjudged estimateUnzstdSize expectations.","solutions":["Raise the configured limit (e.g. server.MaxRequestBodySize or maxBodySize argument) above the largest expected uncompressed payload","Stream the decompressed body to disk/network instead of buffering through a limited writer","Compress data more aggressively server-side or paginate responses"],"exampleFix":"// before\ns.MaxRequestBodySize = 4 * 1024 * 1024 // 4 MiB\n// after\ns.MaxRequestBodySize = 64 * 1024 * 1024 // 64 MiB, fits largest unzstd payload","handlingStrategy":"validation","validationCode":"// before decompressing, check the configured limit\ncap := int64(64 * 1024 * 1024)\nif int64(len(compressed)) > 0 && int64(len(compressed)) > maxBodySize {\n    return errors.New(\"payload exceeds maxBodySize; raise limit before unzstd\")\n}\n_ = cap","typeGuard":null,"tryCatchPattern":"n, err := fasthttp.WriteUnzstd(dst, src)\nif err != nil {\n    if strings.HasPrefix(err.Error(), \"too much data unzstd\") {\n        // retry with a larger buffer/stream to file\n        return streamUnzstdToFile(src)\n    }\n    return err\n}","preventionTips":["Set MaxRequestBodySize/MaxResponseBodySize above the largest legitimate uncompressed payload","Prefer streaming decompression to disk for user-supplied archives","Log compressed size and limit together so mismatches are obvious"],"tags":["zstd","fasthttp","body-size-limit","decompression"],"backgroundTag":"response-body-too-large","analyzedSha":"c96f600972c6f4a7a30d664257b340ebe9d60124","analyzedAt":"2026-08-31T22:48:28.265Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}