crowdsecurity/crowdsec · error

cannot list shards for enhanced_read: %w

Error message

cannot list shards for enhanced_read: %w

What it means

This error wraps any failure returned by the AWS Kinesis ListShards API while the source is enumerating the shards of a stream in enhanced fan-out mode. ListShards fails when the request cannot be served — typically because the stream does not exist, the IAM principal lacks kinesis:ListShards/DescribeStreamSummary permission, or the AWS credentials/region are wrong. It is thrown by SubscribeToShards, the entry point for enhanced fan-out shard enumeration, and bubbles up as 'cannot subscribe to shards' through EnhancedRead.

Source

Thrown at pkg/acquisition/modules/kinesis/run.go:250

				return nil
			}

			switch et := event.(type) {
			case *kinTypes.SubscribeToShardEventStreamMemberSubscribeToShardEvent:
				s.ParseAndPushRecords(et.Value.Records, out, logger, shardID)
			default:
				logger.Infof("unhandled SubscribeToShard event: %T", et)
			}
		}
	}
}

func (s *Source) SubscribeToShards(ctx context.Context, arn arn.ARN, streamConsumer *kinesis.RegisterStreamConsumerOutput, out chan pipeline.Event) error {
	shards, err := s.kClient.ListShards(ctx, &kinesis.ListShardsInput{
			StreamName: aws.String(arn.Resource[7:]),
		})
	if err != nil {
		return fmt.Errorf("cannot list shards for enhanced_read: %w", err)
	}

	for _, shard := range shards.Shards {
		shardID := *shard.ShardId

		r, err := s.kClient.SubscribeToShard(ctx, &kinesis.SubscribeToShardInput{
				ShardId:          aws.String(shardID),
				StartingPosition: &kinTypes.StartingPosition{Type: kinTypes.ShardIteratorTypeLatest},
				ConsumerARN:      streamConsumer.Consumer.ConsumerARN,
			})
		if err != nil {
			return fmt.Errorf("cannot subscribe to shard: %w", err)
		}

		s.shardReaderTomb.Go(func() error {
			return s.ReadFromSubscription(r.GetStream().Reader, out, shardID, arn.Resource[7:])
		})
	}

View on GitHub (pinned to 909b515798)

Solutions

1. Verify the stream exists and is ACTIVE in the configured region: aws kinesis describe-stream-summary --stream-name <name>
2. Fix the stream_arn in the crowdsec acquisition config (wrong region/account is the most common cause).
3. Check IAM permissions: the principal needs kinesis:ListShards (and DescribeStreamSummary) on the stream resource.
4. Validate credentials (aws sts get-caller-identity) and that AWS_REGION/profile match the ARN.
5. Retry on transient failures; if it persists, check network reachability to the Kinesis endpoint.

Example fix

// before: ambiguous ARN region
stream_arn: arn:aws:kinesis:us-west-2:111122223333:stream/my-stream
// after: region corrected to match the actual stream
stream_arn: arn:aws:kinesis:us-east-1:111122223333:stream/my-stream

// Pre-flight before starting the datasource
out, err := awsClient.DescribeStreamSummary(ctx, &kinesis.DescribeStreamSummaryInput{
	StreamName: aws.String(streamNameFromARN),
})
if err != nil || *out.StreamDescriptionSummary.StreamStatus != "ACTIVE" {
	return fmt.Errorf("stream %s not usable: %w", streamName, err)
}

var opErr *types.Error
if err := runKinesis(); err != nil {
	if errors.As(err, &opErr) {
		log.Printf("kinesis API error: %s", opErr.ErrorCode()) // ResourceNotFound, AccessDenied...
	}
	return err
}

When it happens

Trigger: Calling SubscribeToShards (via EnhancedRead) when: the StreamARN names a stream that was deleted or is in a different region/account; the IAM role lacks kinesis:ListShards or kinesis:DescribeStreamSummary; credentials are expired, missing, or point to the wrong profile; the stream is in CREATING/DELETING status; or network/DNS blocks the Kinesis endpoint.

Common situations: Typos in stream_arn in the acquis.yaml kinesis source; running crowdsec on an instance whose IAM role was changed; region mismatch between the configured ARN and AWS_REGION; deleting/rescaling the stream while the datasource starts; corporate egress firewall dropping kinesis.<region>.amazonaws.com.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• while describing group %s: %w
• cannot subscribe to shard: %w
• cannot deregister consumer: %w
• cannot register consumer: %w
• cannot get shard iterator: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/ccc74519f0c65de4. Report an issue: GitHub.