crowdsecurity/crowdsec · error

while reading logs from %s/%s: %w

Error message

while reading logs from %s/%s: %w

What it means

CatLogStream reads a log stream once to the end (one-shot acquisition) via GetLogEvents pagination. A failing page fetch is wrapped as `while reading logs from <group>/<stream>: <err>` and aborts the whole one-shot read, so events already paginated may have been consumed but the acquisition fails.

Source

Thrown at pkg/acquisition/modules/cloudwatch/run.go:413

				cfg.logger.Tracef("next_token: %s", *startFrom)
			}

			p := cloudwatchlogs.NewGetLogEventsPaginator(
				s.cwClient,
				&cloudwatchlogs.GetLogEventsInput{
					Limit:         aws.Int32(10),
					LogGroupName:  aws.String(cfg.GroupName),
					LogStreamName: aws.String(cfg.StreamName),
					StartTime:     aws.Int64(startTime),
					EndTime:       aws.Int64(endTime),
					StartFromHead: &head,
					NextToken:     startFrom,
				},
				)
			for p.HasMorePages() {
				page, err := p.NextPage(ctx)
				if err != nil {
					return fmt.Errorf("while reading logs from %s/%s: %w", cfg.GroupName, cfg.StreamName, err)
				}

				for _, e := range page.Events {
					evt, err := cwLogToEvent(e, cfg)
					if err != nil {
						cfg.logger.Warningf("discard event: %s", err)
					}

					cfg.logger.Debugf("pushing message: %s", evt.Line.Raw)

					outChan <- evt
				}

				if startFrom != nil && page.NextForwardToken != nil && *page.NextForwardToken == *startFrom {
					cfg.logger.Debugf("reached end of available events")
					hasMoreEvents = false
					break
				}

View on GitHub (pinned to 909b515798)

Solutions

1. Confirm group/stream names and region with `aws logs describe-log-streams --log-group-name <g>`
2. Check IAM permissions include logs:GetLogEvents
3. Re-run the one-shot acquisition; treat as transient if the wrapped error is a network/throttle error
4. Validate credentials with `aws sts get-caller-identity`

Example fix

// before
cloudwatch://my-group?log_stream=typo-stream (oneshot)
// after
cloudwatch://my-group?log_stream=real-stream (oneshot)

_, err := client.GetLogEvents(ctx, &cwlogs.GetLogEventsInput{
    LogGroupName: aws.String(group), LogStreamName: aws.String(stream), Limit: aws.Int32(1),
})
if err != nil { return fmt.Errorf("preflight read failed: %w", err) }

err := OneShotAcquisition(ctx)
if err != nil && strings.Contains(err.Error(), "while reading logs from") {
    // inspect wrapped AWS error; retry once on transient codes
}

When it happens

Trigger: OneShotAcquisition calls CatLogStream and p.NextPage(ctx) fails — wrong region, group/stream not found, throttling, network error, or missing logs:GetLogEvents permission.

Common situations: cscli/crowdsec one-shot collection run with a mistyped stream name, credentials expired mid-run, or AWS API outage during batch log harvesting.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• while describing group %s: %w
• while reading %s/%s: %w
• group_name is mandatory for CloudwatchSource
• aws_region is not specified, specify it or aws_config_dir
• query is mandatory (at least start_date and end_date or back

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/7a8f4342295464ab. Report an issue: GitHub.