crowdsecurity/crowdsec · error

failed to read object %s/%s: %s

Error message

failed to read object %s/%s: %s

What it means

After scanning the object line by line, readFile checks scanner.Err(). A non-nil error means the bufio.Scanner aborted — most commonly a line exceeding the scanner buffer (ErrTooLong) — and is reported as this error.

Source

Thrown at pkg/acquisition/modules/s3/run.go:427

			case metrics.AcquisitionMetricsLevelAggregated, metrics.AcquisitionMetricsLevelNone: // Even if metrics are disabled, we want to source in the event
				l.Src = bucket
			}

			evt := pipeline.MakeEvent(s.Config.UseTimeMachine, pipeline.LOG, true)
			evt.Line = l

			// don't block in shutdown
			select {
			case s.out <-evt:
			case <-s.t.Dying():
				s.logger.Infof("tomb is dying, dropping event for %s/%s", bucket, key)
				return nil
			}
		}
	}

	if err := scanner.Err(); err != nil {
		return fmt.Errorf("failed to read object %s/%s: %s", bucket, key, err)
	}

	if s.metricsLevel != metrics.AcquisitionMetricsLevelNone {
		metrics.S3DataSourceObjectsRead.WithLabelValues(bucket).Inc()
	}

	return nil
}

func (s *Source) OneShotAcquisition(ctx context.Context, out chan pipeline.Event, t *tomb.Tomb) error {
	s.logger.Infof("starting acquisition of %s/%s/%s", s.Config.BucketName, s.Config.Prefix, s.Config.Key)
	s.out = out
	s.ctx, s.cancel = context.WithCancel(ctx)
	s.Config.UseTimeMachine = true
	s.t = t

	if s.Config.Key != "" {
		err := s.readFile(s.Config.BucketName, s.Config.Key)

View on GitHub (pinned to 909b515798)

Solutions

1. Increase the buffer via the max_buffer_size DSN parameter (e.g. max_buffer_size=10485760).
2. If the producer emits multi-MB single lines, re-export the data with one record per line.
3. For IO errors, retry acquisition and check endpoint connectivity.

Example fix

// before
crowdsec-s3://my-bucket/path/
// after
crowdsec-s3://my-bucket/path/?max_buffer_size=10485760

// raise the scanner buffer for big lines before reading
buf := make([]byte, 0, 64*1024)
scanner.Buffer(buf, 10*1024*1024) // 10MB max line

if err := readFile(bucket, key); err != nil {
    var tooLong bool
    if errors.As(err, new(error)) && strings.Contains(err.Error(), "token too long") {
        tooLong = true
    }
    if tooLong { /* retry with larger max_buffer_size */ }
}

When it happens

Trigger: An object contains a line (or JSON record) longer than the scanner's max buffer, or an IO error occurs while streaming the body.

Common situations: Very large single-line cloudtrail/JSON exports; buffer size not raised via max_buffer_size DSN parameter; network mid-stream IO errors.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• failed to read header of object %s/%s: %w
• bucket_name and sqs_name are mutually exclusive
• sqs_name is required when using sqs polling method
• bucket_name is required
• failed to read body: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/55ccb32298c93714. Report an issue: GitHub.