risingwavelabs/risingwave · error · ConnectorError

failed to list kinesis shards

Error message

failed to list kinesis shards

What it means

In the Kinesis split enumerator client (`list_splits`, src/connector/src/source/kinesis/enumerator/client.rs:72), a `list_shard` SDK call (including retries via `next_token`) returned an error that was not an expired-token retryable case. The SDK error is wrapped with `anyhow!` and given the context "failed to list kinesis shards", so the original AWS SDK error is preserved as the chain source.

Source

Thrown at src/connector/src/source/kinesis/enumerator/client.rs:72

        loop {
            let mut req = self.client.list_shards();
            if let Some(token) = next_token.take() {
                req = req.next_token(token);
            } else {
                req = req.stream_name(&self.stream_name);
            }

            let list_shard_output = match req.send().await {
                Ok(output) => output,
                Err(e) => {
                    if let Some(e_inner) = e.as_service_error()
                        && e_inner.is_expired_next_token_exception()
                    {
                        tracing::info!("Kinesis ListShard token expired, retrying...");
                        next_token = None;
                        continue;
                    }
                    return Err(anyhow!(e).context("failed to list kinesis shards").into());
                }
            };
            match list_shard_output.shards {
                Some(shard) => shard_collect.extend(shard),
                None => bail!("no shards in stream {}", &self.stream_name),
            }

            match list_shard_output.next_token {
                Some(token) => next_token = Some(token),
                None => break,
            }
        }
        Ok(shard_collect
            .into_iter()
            .map(|x| KinesisSplit {
                shard_id: x.shard_id().to_owned().into(),
                // handle start with position in reader part
                next_offset: KinesisOffset::None,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the chained SDK error (source of the anyhow error) to identify the root cause (throttling, permission, or missing stream).
2. Verify IAM policy grants `kinesis:ListShards` on the stream resource.
3. Verify the stream name in the source WITH options and that the stream exists in the configured region.
4. If throttled (LimitExceededException), retry later or reduce enumeration frequency; expired NextToken cases are already retried internally.

// Before creating the source, check access and stream existence
aws kinesis list-shards --stream-name <stream> --region <region>
// Success proves credentials, region, and kinesis:ListShards are OK

// Wrap enumeration with retry on transient SDK errors; inspect the anyhow chain
match enumerator.list_splits().await {
    Err(e) if e.root_cause().downcast_ref::<SdkError<ListShardsError>>().map_or(false, is_throttling) => {
        tokio::time::sleep(backoff).await; // retry
    }
    Err(e) => return Err(e.context("failed to list kinesis shards")),
    Ok(splits) => splits,
}

When it happens

Trigger: Kinesis `ListShards` API call fails due to stream not existing, IAM permission denial (kinesis:ListShards), throttling (LimitExceededException), network errors, or any non-expired-token SDK error surfaced during split enumeration.

Common situations: Misconfigured AWS credentials or region, missing `kinesis:ListShards` IAM permission, stream name typo, stream recently deleted, or Kinesis throttling under load.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• failed to send records. sent {} out of {}
• Kinesis error: {0}
• failed to generate AWS MSK IAM token
• no shards in stream {}
• s3 error: {inner}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/9429f6e87d2e47a3. Report an issue: GitHub.