What causes "failed to create gzip reader for object %s/%s: %w" in crowdsecurity/crowdsec?

Verify the object downloads and decompresses with `aws s3 cp - | gunzip`.

failed to create gzip reader for object %s/%s: %w

Error message

failed to create gzip reader for object %s/%s: %w

What it means

When a .gz object's header matches the gzip magic bytes (0x1f 0x8b), readFile creates a gzip.Reader over the body. If gzip.NewReader fails, the data claims to be gzipped but is not a valid gzip stream.

Source

Thrown at pkg/acquisition/modules/s3/run.go:369

	})
	if err != nil {
		return fmt.Errorf("failed to get object %s/%s: %w", bucket, key, err)
	}
	defer output.Body.Close()

	if strings.HasSuffix(key, ".gz") {
		// This *might* be a gzipped file, but sometimes the SDK will decompress the data for us (it's not clear when it happens, only had the issue with cloudtrail logs)
		header := make([]byte, 2)

		_, err := output.Body.Read(header)
		if err != nil {
			return fmt.Errorf("failed to read header of object %s/%s: %w", bucket, key, err)
		}

		if header[0] == 0x1f && header[1] == 0x8b {
			gz, err := gzip.NewReader(io.MultiReader(bytes.NewReader(header), output.Body))
			if err != nil {
				return fmt.Errorf("failed to create gzip reader for object %s/%s: %w", bucket, key, err)
			}
			scanner = bufio.NewScanner(gz)
		} else {
			scanner = bufio.NewScanner(io.MultiReader(bytes.NewReader(header), output.Body))
		}
	} else {
		scanner = bufio.NewScanner(output.Body)
	}

	if s.Config.MaxBufferSize > 0 {
		s.logger.Infof("Setting max buffer size to %d", s.Config.MaxBufferSize)

		buf := make([]byte, 0, bufio.MaxScanTokenSize)
		scanner.Buffer(buf, s.Config.MaxBufferSize)
	}

	for scanner.Scan() {
		select {

View on GitHub (pinned to 909b515798)

Solutions

Verify the object downloads and decompresses with `aws s3 cp - | gunzip`.
Re-upload the corrupted object.
If the data is not actually gzipped despite the name, remove the .gz suffix or check the producer.

Defensive patterns

Strategy: validation

Validate before calling

f, _ := os.Open(localCopy)
magic := make([]byte, 2)
io.ReadFull(f, magic)
if magic[0] == 0x1f && magic[1] == 0x8b {
    if _, err := gzip.NewReader(f); err != nil {
        return fmt.Errorf("object is not a valid gzip stream: %v", err)
    }
}

Try / catch

if err := readFile(bucket, key); err != nil && strings.Contains(err.Error(), "gzip") {
    log.Errorf("corrupt gzip object %s/%s — re-upload required", bucket, key)
}

Prevention

Verify gzip integrity at upload time (gunzip -t in the producer pipeline).
Check multipart upload completion before publishing notifications.
Only name files .gz when they are actually gzipped.

When it happens

Trigger: An object with a .gz suffix whose first two bytes are 0x1f 0x8b but whose remaining data is corrupt, truncated, or an unsupported gzip variant; sometimes SDK auto-decompression confusion noted in the code comment.

Common situations: Partially uploaded objects (failed multipart upload); corrupted uploads; producers that gzip only partially; objects re-encoded by intermediaries.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/87a32eafb6aa7e23. Report an issue: GitHub.