{"record":{"id":"47306a4b49928502","repo":"crowdsecurity/crowdsec","slug":"failed-to-read-header-of-object-s-s-w","errorCode":null,"errorMessage":"failed to read header of object %s/%s: %w","messagePattern":"failed to read header of object (.+?)/(.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/acquisition/modules/s3/run.go","lineNumber":363,"sourceCode":"\t\t\"key\":    key,\n\t})\n\n\toutput, err := s.s3Client.GetObject(s.ctx, &s3.GetObjectInput{\n\t\tBucket: aws.String(bucket),\n\t\tKey:    aws.String(key),\n\t})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get object %s/%s: %w\", bucket, key, err)\n\t}\n\tdefer output.Body.Close()\n\n\tif strings.HasSuffix(key, \".gz\") {\n\t\t// This *might* be a gzipped file, but sometimes the SDK will decompress the data for us (it's not clear when it happens, only had the issue with cloudtrail logs)\n\t\theader := make([]byte, 2)\n\n\t\t_, err := output.Body.Read(header)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to read header of object %s/%s: %w\", bucket, key, err)\n\t\t}\n\n\t\tif header[0] == 0x1f && header[1] == 0x8b {\n\t\t\tgz, err := gzip.NewReader(io.MultiReader(bytes.NewReader(header), output.Body))\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to create gzip reader for object %s/%s: %w\", bucket, key, err)\n\t\t\t}\n\t\t\tscanner = bufio.NewScanner(gz)\n\t\t} else {\n\t\t\tscanner = bufio.NewScanner(io.MultiReader(bytes.NewReader(header), output.Body))\n\t\t}\n\t} else {\n\t\tscanner = bufio.NewScanner(output.Body)\n\t}\n\n\tif s.Config.MaxBufferSize > 0 {\n\t\ts.logger.Infof(\"Setting max buffer size to %d\", s.Config.MaxBufferSize)\n","sourceCodeStart":345,"sourceCodeEnd":381,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/acquisition/modules/s3/run.go#L345-L381","documentation":"For keys ending in .gz, readFile reads a 2-byte header from the object body to detect gzip magic bytes. If the read fails (body errored, truncated, or already consumed), this wrapped error is returned.","triggerScenarios":"The GetObject response body returns an error on Read: interrupted connection, streaming body closed early, or an empty/zero-byte object named *.gz.","commonSituations":"Network interruptions mid-stream; zero-byte .gz placeholder objects uploaded by producers; proxies/MinIO endpoints cutting the response short.","solutions":["Retry acquisition; transient network errors usually resolve.","Check the object is a valid, non-empty file.","Verify stable connectivity to the S3 endpoint; check proxy/MinIO logs for truncation."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// ensure the object is non-empty before reading\nhead, err := client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: &b, Key: &k})\nif err == nil && (head.ContentLength == nil || *head.ContentLength == 0) {\n    return fmt.Errorf(\"object %s/%s is empty\", b, k)\n}","typeGuard":null,"tryCatchPattern":"if err := readFile(bucket, key); err != nil && strings.Contains(err.Error(), \"failed to read header\") {\n    // transient IO problem: requeue with backoff\n}","preventionTips":["Avoid uploading zero-byte .gz placeholders.","Use stable, low-latency connectivity to the S3 endpoint.","Monitor proxy/load-balancer truncation errors."],"tags":["aws","s3","io","gzip"],"backgroundTag":"file-read-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}