apache/beam · warning · KinesisClientThrottledException

Too many requests to Kinesis. Wait some time and retry.

Error message

Too many requests to Kinesis. Wait some time and retry.

What it means

SimplifiedKinesisClient.wrapExceptions translates AWS SDK LimitExceededException and ProvisionedThroughputExceededException into KinesisClientThrottledException with this message. Kinesis is rate-limiting the account/stream: too many API calls per second (listShards, getShardIterator, getRecords, getBacklogBytes).

Source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/SimplifiedKinesisClient.java:227

        .dimensions(Dimension.builder().name(STREAM_NAME_DIMENSION).value(streamName).build())
        .build();
  }

  /**
   * Wraps Amazon specific exceptions into more friendly format.
   *
   * @throws TransientKinesisException - in case of recoverable situation, i.e. the request rate is
   *     too high, Kinesis remote service failed, network issue, etc.
   * @throws ExpiredIteratorException - if iterator needs to be refreshed
   * @throws RuntimeException - in all other cases
   */
  private static <T> T wrapExceptions(Callable<T> callable) throws TransientKinesisException {
    try {
      return callable.call();
    } catch (ExpiredIteratorException e) {
      throw e;
    } catch (LimitExceededException | ProvisionedThroughputExceededException e) {
      throw new KinesisClientThrottledException(
          "Too many requests to Kinesis. Wait some time and retry.", e);
    } catch (SdkServiceException e) {
      if (e.isThrottlingException()
          || SdkDefaultRetrySetting.RETRYABLE_STATUS_CODES.contains(e.statusCode())) {
        throw new TransientKinesisException("Kinesis backend failed. Wait some time and retry.", e);
      }
      throw e; // others, such as 4xx, are not retryable
    } catch (SdkClientException e) {
      if (SdkDefaultRetrySetting.RETRYABLE_EXCEPTIONS.contains(e.getClass())) {
        throw new TransientKinesisException("Retryable failure", e);
      }
      throw e;
    } catch (Exception e) {
      throw new RuntimeException("Unknown kinesis failure, when trying to reach kinesis", e);
    }
  }

  @Override

View on GitHub (pinned to 12126d8942)

Solutions

1. Add exponential backoff with jitter and retry the request after a delay (KinesisClientThrottledException is intended to be retried).
2. Reduce polling frequency / batch size and consolidate shard-iterator requests; avoid listing shards from every worker.
3. Request a Kinesis service quota increase or shard the workload across more streams/shards.

catch (KinesisClientThrottledException e) {
  Thread.sleep(backoffMs);
  backoffMs = Math.min(backoffMs * 2, maxBackoffMs);
  // retry
}

When it happens

Trigger: Any of getShardIterator, listShardsFollowingClosedShard, getRecords, or getBacklogBytes invoked faster than the per-stream/per-shard limits allow — e.g. many consumers listing shards concurrently or frequent getRecords polling on a high-shard-count stream.

Common situations: Scaling a Beam pipeline to many workers all hammering the Kinesis API; small stream with aggressive polling frequency; shared AWS account where other applications consume the Kinesis throughput quota.

Understand the failure class

• HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

• Kinesis backend failed. Wait some time and retry.
• Retryable failure
• Error writing to ES after %d attempt(s). No more attempts al
• Rate Limit Exceeded, Could not process this batch.
• Transient exception occurred.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/bd20219ad2f873a5. Report an issue: GitHub.