{"record":{"id":"55ccb32298c93714","repo":"crowdsecurity/crowdsec","slug":"failed-to-read-object-s-s-s","errorCode":null,"errorMessage":"failed to read object %s/%s: %s","messagePattern":"failed to read object (.+?)/(.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/acquisition/modules/s3/run.go","lineNumber":427,"sourceCode":"\t\t\tcase metrics.AcquisitionMetricsLevelAggregated, metrics.AcquisitionMetricsLevelNone: // Even if metrics are disabled, we want to source in the event\n\t\t\t\tl.Src = bucket\n\t\t\t}\n\n\t\t\tevt := pipeline.MakeEvent(s.Config.UseTimeMachine, pipeline.LOG, true)\n\t\t\tevt.Line = l\n\n\t\t\t// don't block in shutdown\n\t\t\tselect {\n\t\t\tcase s.out <-evt:\n\t\t\tcase <-s.t.Dying():\n\t\t\t\ts.logger.Infof(\"tomb is dying, dropping event for %s/%s\", bucket, key)\n\t\t\t\treturn nil\n\t\t\t}\n\t\t}\n\t}\n\n\tif err := scanner.Err(); err != nil {\n\t\treturn fmt.Errorf(\"failed to read object %s/%s: %s\", bucket, key, err)\n\t}\n\n\tif s.metricsLevel != metrics.AcquisitionMetricsLevelNone {\n\t\tmetrics.S3DataSourceObjectsRead.WithLabelValues(bucket).Inc()\n\t}\n\n\treturn nil\n}\n\nfunc (s *Source) OneShotAcquisition(ctx context.Context, out chan pipeline.Event, t *tomb.Tomb) error {\n\ts.logger.Infof(\"starting acquisition of %s/%s/%s\", s.Config.BucketName, s.Config.Prefix, s.Config.Key)\n\ts.out = out\n\ts.ctx, s.cancel = context.WithCancel(ctx)\n\ts.Config.UseTimeMachine = true\n\ts.t = t\n\n\tif s.Config.Key != \"\" {\n\t\terr := s.readFile(s.Config.BucketName, s.Config.Key)","sourceCodeStart":409,"sourceCodeEnd":445,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/acquisition/modules/s3/run.go#L409-L445","documentation":"After scanning the object line by line, readFile checks scanner.Err(). A non-nil error means the bufio.Scanner aborted — most commonly a line exceeding the scanner buffer (ErrTooLong) — and is reported as this error.","triggerScenarios":"An object contains a line (or JSON record) longer than the scanner's max buffer, or an IO error occurs while streaming the body.","commonSituations":"Very large single-line cloudtrail/JSON exports; buffer size not raised via max_buffer_size DSN parameter; network mid-stream IO errors.","solutions":["Increase the buffer via the max_buffer_size DSN parameter (e.g. max_buffer_size=10485760).","If the producer emits multi-MB single lines, re-export the data with one record per line.","For IO errors, retry acquisition and check endpoint connectivity."],"exampleFix":"// before\ncrowdsec-s3://my-bucket/path/\n// after\ncrowdsec-s3://my-bucket/path/?max_buffer_size=10485760","handlingStrategy":"validation","validationCode":"// raise the scanner buffer for big lines before reading\nbuf := make([]byte, 0, 64*1024)\nscanner.Buffer(buf, 10*1024*1024) // 10MB max line","typeGuard":null,"tryCatchPattern":"if err := readFile(bucket, key); err != nil {\n    var tooLong bool\n    if errors.As(err, new(error)) && strings.Contains(err.Error(), \"token too long\") {\n        tooLong = true\n    }\n    if tooLong { /* retry with larger max_buffer_size */ }\n}","preventionTips":["Set max_buffer_size generously when reading cloudtrail/JSON exports.","Producers should emit one record per line.","Alert on ErrTooLong-style failures so buffer limits surface early."],"tags":["s3","io","scanner","buffer-size"],"backgroundTag":"file-read-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}