apache/beam · warning

Failed to refresh shards.

Error message

Failed to refresh shards.

What it means

The periodic background refresh of Kinesis shard boundaries (via listShards, asynchronously) failed. The refresh task logs the exception, resets the running flag, and schedules a retry at the next refresh interval; writes continue with the previously cached shard bounds.

Source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java:1261

        @SuppressWarnings("FutureReturnValueIgnored") // safe to ignore
        private void refresh(
            KinesisAsyncClient client,
            Supplier<Instant> nextRefreshFn,
            NavigableSet<BigInteger> bounds,
            @Nullable String nextToken) {
          ListShardsRequest.Builder reqBuilder =
              ListShardsRequest.builder().shardFilter(f -> f.type(AT_LATEST));
          if (nextToken != null) {
            reqBuilder.nextToken(nextToken);
          } else {
            reqBuilder.streamName(streamName);
          }
          client
              .listShards(reqBuilder.build())
              .whenComplete(
                  (resp, exc) -> {
                    if (exc != null) {
                      LOG.warn("Failed to refresh shards.", exc);
                      nextRefresh = nextRefreshFn.get(); // retry later
                      running.set(false);
                      return;
                    }
                    resp.shards().forEach(shard -> bounds.add(lowerHashKey(shard)));
                    if (resp.nextToken() != null) {
                      refresh(client, nextRefreshFn, bounds, resp.nextToken());
                      return;
                    }
                    LOG.debug("Done refreshing {} shards.", bounds.size());
                    nextRefresh = nextRefreshFn.get();
                    running.set(false);
                    shardBounds = bounds; // swap key ranges
                  });
        }

        private BigInteger lowerHashKey(Shard shard) {
          return new BigInteger(shard.hashKeyRange().startingHashKey());

View on GitHub (pinned to 12126d8942)

Solutions

1. Grant kinesis:ListShards permission to the producer's IAM role
2. Check AWS credentials validity/expiry in the pipeline's AwsOptions provider
3. Increase withRefreshPeriod and add AWS SDK retry/timeout tuning to reduce throttling
4. No action needed for transient failures — the refresher retries at the next interval automatically

Example fix

// IAM policy before (missing list)
{"Effect":"Allow","Action":["kinesis:PutRecord*"],"Resource":"*"}
// after
{"Effect":"Allow","Action":["kinesis:PutRecord*","kinesis:ListShards"],"Resource":"*"}

// preflight IAM check
try { client.listShards(b -> b.streamName(stream).maxResults(1)); }
catch (Exception e) { throw new IllegalStateException("ListShards not permitted/failing: " + e.getMessage()); }

// the library already retries on next refresh; for your own calls:
CompletableFuture<ListShardsResponse> f = client.listShards(req);
f.exceptionally(exc -> { log.warn("shard refresh failed, keeping stale bounds", exc); return lastGoodResponse; });

When it happens

Trigger: client.listShards(...) async call completed exceptionally — expired/invalid AWS credentials, throttling (LimitExceededException), network failure, or the producer lacks kinesis:ListShards IAM permission.

Common situations: IAM policies that grant PutRecord but not ListShards; credential rotation failures; Kinesis throttling on streams with many concurrent listShards callers; transient network blips in VPC endpoints.

Related errors

• Too many requests to Kinesis. Wait some time and retry.
• Kinesis backend failed. Wait some time and retry.
• Pool {} - shard {} subscriber got error
• Thread was interrupted, finishing the read loop
• Transient exception occurred.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2af85835d2be3624. Report an issue: GitHub.