crowdsecurity/crowdsec · error

failed to get object %s/%s: %w

Error message

failed to get object %s/%s: %w

What it means

readFile calls s3Client.GetObject for the given bucket/key and wraps any SDK failure in this error. It means the object could not be fetched from S3 (network, permissions, missing object, bad credentials).

Source

Thrown at pkg/acquisition/modules/s3/run.go:353

	}
}

func (s *Source) readFile(bucket string, key string) error {
	// TODO: Handle SSE-C
	var scanner *bufio.Scanner

	logger := s.logger.WithFields(log.Fields{
		"method": "readFile",
		"bucket": bucket,
		"key":    key,
	})

	output, err := s.s3Client.GetObject(s.ctx, &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	})
	if err != nil {
		return fmt.Errorf("failed to get object %s/%s: %w", bucket, key, err)
	}
	defer output.Body.Close()

	if strings.HasSuffix(key, ".gz") {
		// This *might* be a gzipped file, but sometimes the SDK will decompress the data for us (it's not clear when it happens, only had the issue with cloudtrail logs)
		header := make([]byte, 2)

		_, err := output.Body.Read(header)
		if err != nil {
			return fmt.Errorf("failed to read header of object %s/%s: %w", bucket, key, err)
		}

		if header[0] == 0x1f && header[1] == 0x8b {
			gz, err := gzip.NewReader(io.MultiReader(bytes.NewReader(header), output.Body))
			if err != nil {
				return fmt.Errorf("failed to create gzip reader for object %s/%s: %w", bucket, key, err)
			}
			scanner = bufio.NewScanner(gz)

View on GitHub (pinned to 909b515798)

Solutions

1. Verify the bucket/key still exists (aws s3 api get-object or console).
2. Grant the credentials' IAM principal s3:GetObject on bucket/prefix.
3. Check the configured region/endpoint matches the bucket.
4. Confirm credentials are valid and not expired (env vars, instance role, profile).

Example fix

// before: policy without read access
// after: add to the IAM policy
{"Effect": "Allow", "Action": ["s3:GetObject"], "Resource": ["arn:aws:s3:::my-bucket/*"]}

// pre-check with a head request
_, err := client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(b), Key: aws.String(k)})
if err != nil { /* object missing or not readable */ }

if err := readFile(bucket, key); err != nil {
    if errors.Is(err, context.DeadlineExceeded) || isRetryable(err) {
        // requeue with backoff
    } else {
        log.Errorf("permanent failure reading %s/%s: %v", bucket, key, err)
    }
}

When it happens

Trigger: GetObject returns an error: object deleted between notification and read, missing s3:GetObject permission, wrong region, expired credentials, or connectivity problems.

Common situations: Notifications arriving for objects already lifecycle-deleted; IAM policies lacking s3:GetObject on the prefix; cross-account buckets; misconfigured region causing 301 errors.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• while describing group %s: %w
• while reading %s/%s: %w
• while reading logs from %s/%s: %w
• cannot deregister stream consumer: %w
• cannot register stream consumer: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/da6f4abb57f84388. Report an issue: GitHub.