crowdsecurity/crowdsec · error

cannot get shard iterator: %w

Error message

cannot get shard iterator: %w

What it means

GetShardIterator failed in classic (polling) ReadFromShard, so the shard could not be read at all. AWS rejects iterator requests when the stream/shard doesn't exist, permissions are missing, the stream is not active, or the iterator type/parameters are invalid. The error is both logged and returned, killing the per-shard reader goroutine.

Source

Thrown at pkg/acquisition/modules/kinesis/run.go:342

			continue
		}
	}
}

func (s *Source) ReadFromShard(ctx context.Context, out chan pipeline.Event, shardID string) error {
	logger := s.logger.WithField("shard", shardID)
	logger.Debugf("Starting to read shard")

	sharIt, err := s.kClient.GetShardIterator(ctx,
		&kinesis.GetShardIteratorInput{
			ShardId:           aws.String(shardID),
			StreamName:        &s.Config.StreamName,
			ShardIteratorType: kinTypes.ShardIteratorTypeLatest,
		})
	if err != nil {
		logger.Errorf("Cannot get shard iterator: %s", err)
		return fmt.Errorf("cannot get shard iterator: %w", err)
	}

	it := sharIt.ShardIterator
	// AWS recommends to wait for a second between calls to GetRecords for a given shard
	ticker := time.NewTicker(time.Second)

	for {
		select {
		case <-ticker.C:
			records, err := s.kClient.GetRecords(ctx, &kinesis.GetRecordsInput{ShardIterator: it})

			var throughputErr *kinTypes.ProvisionedThroughputExceededException
			if errors.As(err, &throughputErr) {
				logger.Warn("Provisioned throughput exceeded")
				// TODO: implement exponential backoff
				continue
			}

View on GitHub (pinned to 909b515798)

Solutions

1. Confirm stream_name and region point at an ACTIVE stream (aws kinesis describe-stream --stream-name <name>).
2. Refresh the shard list after resharding; the ReadFromStream loop restarts readers, but stale shard IDs need a ListShards refresh.
3. Add kinesis:GetShardIterator and kinesis:GetRecords to the IAM policy.
4. Retry GetShardIterator with backoff on transient/throttling errors instead of killing the reader.

Example fix

// before: hard fail on transient error
return fmt.Errorf("cannot get shard iterator: %w", err)
// after: retry a few times
var itErr *kinTypes.ResourceNotFoundException
for i := 0; i < 3; i++ {
	sharIt, err = s.kClient.GetShardIterator(ctx, in)
	if err == nil {
		break
	}
	if errors.As(err, &itErr) {
		return fmt.Errorf("shard gone: %w", err)
	}
	time.Sleep(time.Duration(i+1) * time.Second)
}

desc, err := client.DescribeStream(ctx, &kinesis.DescribeStreamInput{StreamName: aws.String(streamName)})
// only spawn readers for shard IDs present in desc and while StreamStatus == ACTIVE

var rnfe *kinTypes.ResourceNotFoundException
if err := getShardIterator(ctx); err != nil {
	if errors.As(err, &rnfe) {
		// resharding: exit so the outer loop refreshes the shard list
		return nil
	}
	// else: bounded retry with backoff
}

When it happens

Trigger: ReadFromShard calling GetShardIterator with ShardIteratorType LATEST when: the shard was merged away after a reshard (ResourceNotFound), stream_name in config doesn't match any stream, IAM lacks kinesis:GetShardIterator, or the stream is in CREATING/UPDATING state.

Common situations: Renamed/deleted stream between config write and start; rescaling a stream while crowdsec runs, closing old shard IDs; region mismatch making the stream invisible; policy missing kinesis:GetShardIterator/GetRecords.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• cannot subscribe to shard: %w
• cannot list shards for enhanced_read: %w
• cannot deregister consumer: %w
• cannot register consumer: %w
• cannot subscribe to shards: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/b2bcbc47d532a332. Report an issue: GitHub.