{"record":{"id":"87a32eafb6aa7e23","repo":"crowdsecurity/crowdsec","slug":"failed-to-create-gzip-reader-for-object-s-s-w","errorCode":null,"errorMessage":"failed to create gzip reader for object %s/%s: %w","messagePattern":"failed to create gzip reader for object (.+?)/(.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/acquisition/modules/s3/run.go","lineNumber":369,"sourceCode":"\t})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get object %s/%s: %w\", bucket, key, err)\n\t}\n\tdefer output.Body.Close()\n\n\tif strings.HasSuffix(key, \".gz\") {\n\t\t// This *might* be a gzipped file, but sometimes the SDK will decompress the data for us (it's not clear when it happens, only had the issue with cloudtrail logs)\n\t\theader := make([]byte, 2)\n\n\t\t_, err := output.Body.Read(header)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to read header of object %s/%s: %w\", bucket, key, err)\n\t\t}\n\n\t\tif header[0] == 0x1f && header[1] == 0x8b {\n\t\t\tgz, err := gzip.NewReader(io.MultiReader(bytes.NewReader(header), output.Body))\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to create gzip reader for object %s/%s: %w\", bucket, key, err)\n\t\t\t}\n\t\t\tscanner = bufio.NewScanner(gz)\n\t\t} else {\n\t\t\tscanner = bufio.NewScanner(io.MultiReader(bytes.NewReader(header), output.Body))\n\t\t}\n\t} else {\n\t\tscanner = bufio.NewScanner(output.Body)\n\t}\n\n\tif s.Config.MaxBufferSize > 0 {\n\t\ts.logger.Infof(\"Setting max buffer size to %d\", s.Config.MaxBufferSize)\n\n\t\tbuf := make([]byte, 0, bufio.MaxScanTokenSize)\n\t\tscanner.Buffer(buf, s.Config.MaxBufferSize)\n\t}\n\n\tfor scanner.Scan() {\n\t\tselect {","sourceCodeStart":351,"sourceCodeEnd":387,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/acquisition/modules/s3/run.go#L351-L387","documentation":"When a .gz object's header matches the gzip magic bytes (0x1f 0x8b), readFile creates a gzip.Reader over the body. If gzip.NewReader fails, the data claims to be gzipped but is not a valid gzip stream.","triggerScenarios":"An object with a .gz suffix whose first two bytes are 0x1f 0x8b but whose remaining data is corrupt, truncated, or an unsupported gzip variant; sometimes SDK auto-decompression confusion noted in the code comment.","commonSituations":"Partially uploaded objects (failed multipart upload); corrupted uploads; producers that gzip only partially; objects re-encoded by intermediaries.","solutions":["Verify the object downloads and decompresses with `aws s3 cp - | gunzip`.","Re-upload the corrupted object.","If the data is not actually gzipped despite the name, remove the .gz suffix or check the producer."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"f, _ := os.Open(localCopy)\nmagic := make([]byte, 2)\nio.ReadFull(f, magic)\nif magic[0] == 0x1f && magic[1] == 0x8b {\n    if _, err := gzip.NewReader(f); err != nil {\n        return fmt.Errorf(\"object is not a valid gzip stream: %v\", err)\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := readFile(bucket, key); err != nil && strings.Contains(err.Error(), \"gzip\") {\n    log.Errorf(\"corrupt gzip object %s/%s — re-upload required\", bucket, key)\n}","preventionTips":["Verify gzip integrity at upload time (gunzip -t in the producer pipeline).","Check multipart upload completion before publishing notifications.","Only name files .gz when they are actually gzipped."],"tags":["aws","s3","gzip","data-corruption"],"backgroundTag":"file-read-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}