{"record":{"id":"b2bcbc47d532a332","repo":"crowdsecurity/crowdsec","slug":"cannot-get-shard-iterator-w","errorCode":null,"errorMessage":"cannot get shard iterator: %w","messagePattern":"cannot get shard iterator: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/acquisition/modules/kinesis/run.go","lineNumber":342,"sourceCode":"\n\t\t\tcontinue\n\t\t}\n\t}\n}\n\nfunc (s *Source) ReadFromShard(ctx context.Context, out chan pipeline.Event, shardID string) error {\n\tlogger := s.logger.WithField(\"shard\", shardID)\n\tlogger.Debugf(\"Starting to read shard\")\n\n\tsharIt, err := s.kClient.GetShardIterator(ctx,\n\t\t&kinesis.GetShardIteratorInput{\n\t\t\tShardId:           aws.String(shardID),\n\t\t\tStreamName:        &s.Config.StreamName,\n\t\t\tShardIteratorType: kinTypes.ShardIteratorTypeLatest,\n\t\t})\n\tif err != nil {\n\t\tlogger.Errorf(\"Cannot get shard iterator: %s\", err)\n\t\treturn fmt.Errorf(\"cannot get shard iterator: %w\", err)\n\t}\n\n\tit := sharIt.ShardIterator\n\t// AWS recommends to wait for a second between calls to GetRecords for a given shard\n\tticker := time.NewTicker(time.Second)\n\n\tfor {\n\t\tselect {\n\t\tcase <-ticker.C:\n\t\t\trecords, err := s.kClient.GetRecords(ctx, &kinesis.GetRecordsInput{ShardIterator: it})\n\n\t\t\tvar throughputErr *kinTypes.ProvisionedThroughputExceededException\n\t\t\tif errors.As(err, &throughputErr) {\n\t\t\t\tlogger.Warn(\"Provisioned throughput exceeded\")\n\t\t\t\t// TODO: implement exponential backoff\n\t\t\t\tcontinue\n\t\t\t}\n","sourceCodeStart":324,"sourceCodeEnd":360,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/acquisition/modules/kinesis/run.go#L324-L360","documentation":"GetShardIterator failed in classic (polling) ReadFromShard, so the shard could not be read at all. AWS rejects iterator requests when the stream/shard doesn't exist, permissions are missing, the stream is not active, or the iterator type/parameters are invalid. The error is both logged and returned, killing the per-shard reader goroutine.","triggerScenarios":"ReadFromShard calling GetShardIterator with ShardIteratorType LATEST when: the shard was merged away after a reshard (ResourceNotFound), stream_name in config doesn't match any stream, IAM lacks kinesis:GetShardIterator, or the stream is in CREATING/UPDATING state.","commonSituations":"Renamed/deleted stream between config write and start; rescaling a stream while crowdsec runs, closing old shard IDs; region mismatch making the stream invisible; policy missing kinesis:GetShardIterator/GetRecords.","solutions":["Confirm stream_name and region point at an ACTIVE stream (aws kinesis describe-stream --stream-name <name>).","Refresh the shard list after resharding; the ReadFromStream loop restarts readers, but stale shard IDs need a ListShards refresh.","Add kinesis:GetShardIterator and kinesis:GetRecords to the IAM policy.","Retry GetShardIterator with backoff on transient/throttling errors instead of killing the reader."],"exampleFix":"// before: hard fail on transient error\nreturn fmt.Errorf(\"cannot get shard iterator: %w\", err)\n// after: retry a few times\nvar itErr *kinTypes.ResourceNotFoundException\nfor i := 0; i < 3; i++ {\n\tsharIt, err = s.kClient.GetShardIterator(ctx, in)\n\tif err == nil {\n\t\tbreak\n\t}\n\tif errors.As(err, &itErr) {\n\t\treturn fmt.Errorf(\"shard gone: %w\", err)\n\t}\n\ttime.Sleep(time.Duration(i+1) * time.Second)\n}","handlingStrategy":"retry","validationCode":"desc, err := client.DescribeStream(ctx, &kinesis.DescribeStreamInput{StreamName: aws.String(streamName)})\n// only spawn readers for shard IDs present in desc and while StreamStatus == ACTIVE","typeGuard":null,"tryCatchPattern":"var rnfe *kinTypes.ResourceNotFoundException\nif err := getShardIterator(ctx); err != nil {\n\tif errors.As(err, &rnfe) {\n\t\t// resharding: exit so the outer loop refreshes the shard list\n\t\treturn nil\n\t}\n\t// else: bounded retry with backoff\n}","preventionTips":["Refresh ListShards after reshard events instead of caching shard IDs.","Check stream_name/region before start.","IAM: GetShardIterator + GetRecords + DescribeStream.","Avoid renaming streams that an active datasource points to."],"tags":["aws","kinesis","streaming","iam"],"backgroundTag":"api-request-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}