{"record":{"id":"7a8f4342295464ab","repo":"crowdsecurity/crowdsec","slug":"while-reading-logs-from-s-s-w","errorCode":null,"errorMessage":"while reading logs from %s/%s: %w","messagePattern":"while reading logs from (.+?)/(.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/acquisition/modules/cloudwatch/run.go","lineNumber":413,"sourceCode":"\t\t\t\tcfg.logger.Tracef(\"next_token: %s\", *startFrom)\n\t\t\t}\n\n\t\t\tp := cloudwatchlogs.NewGetLogEventsPaginator(\n\t\t\t\ts.cwClient,\n\t\t\t\t&cloudwatchlogs.GetLogEventsInput{\n\t\t\t\t\tLimit:         aws.Int32(10),\n\t\t\t\t\tLogGroupName:  aws.String(cfg.GroupName),\n\t\t\t\t\tLogStreamName: aws.String(cfg.StreamName),\n\t\t\t\t\tStartTime:     aws.Int64(startTime),\n\t\t\t\t\tEndTime:       aws.Int64(endTime),\n\t\t\t\t\tStartFromHead: &head,\n\t\t\t\t\tNextToken:     startFrom,\n\t\t\t\t},\n\t\t\t\t)\n\t\t\tfor p.HasMorePages() {\n\t\t\t\tpage, err := p.NextPage(ctx)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"while reading logs from %s/%s: %w\", cfg.GroupName, cfg.StreamName, err)\n\t\t\t\t}\n\n\t\t\t\tfor _, e := range page.Events {\n\t\t\t\t\tevt, err := cwLogToEvent(e, cfg)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tcfg.logger.Warningf(\"discard event: %s\", err)\n\t\t\t\t\t}\n\n\t\t\t\t\tcfg.logger.Debugf(\"pushing message: %s\", evt.Line.Raw)\n\n\t\t\t\t\toutChan <- evt\n\t\t\t\t}\n\n\t\t\t\tif startFrom != nil && page.NextForwardToken != nil && *page.NextForwardToken == *startFrom {\n\t\t\t\t\tcfg.logger.Debugf(\"reached end of available events\")\n\t\t\t\t\thasMoreEvents = false\n\t\t\t\t\tbreak\n\t\t\t\t}","sourceCodeStart":395,"sourceCodeEnd":431,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/acquisition/modules/cloudwatch/run.go#L395-L431","documentation":"CatLogStream reads a log stream once to the end (one-shot acquisition) via GetLogEvents pagination. A failing page fetch is wrapped as `while reading logs from <group>/<stream>: <err>` and aborts the whole one-shot read, so events already paginated may have been consumed but the acquisition fails.","triggerScenarios":"OneShotAcquisition calls CatLogStream and p.NextPage(ctx) fails — wrong region, group/stream not found, throttling, network error, or missing logs:GetLogEvents permission.","commonSituations":"cscli/crowdsec one-shot collection run with a mistyped stream name, credentials expired mid-run, or AWS API outage during batch log harvesting.","solutions":["Confirm group/stream names and region with `aws logs describe-log-streams --log-group-name <g>`","Check IAM permissions include logs:GetLogEvents","Re-run the one-shot acquisition; treat as transient if the wrapped error is a network/throttle error","Validate credentials with `aws sts get-caller-identity`"],"exampleFix":"// before\ncloudwatch://my-group?log_stream=typo-stream (oneshot)\n// after\ncloudwatch://my-group?log_stream=real-stream (oneshot)","handlingStrategy":"retry","validationCode":"_, err := client.GetLogEvents(ctx, &cwlogs.GetLogEventsInput{\n    LogGroupName: aws.String(group), LogStreamName: aws.String(stream), Limit: aws.Int32(1),\n})\nif err != nil { return fmt.Errorf(\"preflight read failed: %w\", err) }","typeGuard":null,"tryCatchPattern":"err := OneShotAcquisition(ctx)\nif err != nil && strings.Contains(err.Error(), \"while reading logs from\") {\n    // inspect wrapped AWS error; retry once on transient codes\n}","preventionTips":["Preflight-check group/stream existence before batch harvest","Ensure logs:GetLogEvents permission for one-shot runs","Retry transient network errors before failing the run","Keep credentials fresh for long batch jobs"],"tags":["aws","cloudwatch","oneshot","network"],"backgroundTag":"api-request-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}