{"record":{"id":"bd20219ad2f873a5","repo":"apache/beam","slug":"too-many-requests-to-kinesis-wait-some-time-and-retry","errorCode":null,"errorMessage":"Too many requests to Kinesis. Wait some time and retry.","messagePattern":"Too many requests to Kinesis\\. Wait some time and retry\\.","errorType":"exception","errorClass":"KinesisClientThrottledException","httpStatus":null,"severity":"warning","filePath":"sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/SimplifiedKinesisClient.java","lineNumber":227,"sourceCode":"        .dimensions(Dimension.builder().name(STREAM_NAME_DIMENSION).value(streamName).build())\n        .build();\n  }\n\n  /**\n   * Wraps Amazon specific exceptions into more friendly format.\n   *\n   * @throws TransientKinesisException - in case of recoverable situation, i.e. the request rate is\n   *     too high, Kinesis remote service failed, network issue, etc.\n   * @throws ExpiredIteratorException - if iterator needs to be refreshed\n   * @throws RuntimeException - in all other cases\n   */\n  private static <T> T wrapExceptions(Callable<T> callable) throws TransientKinesisException {\n    try {\n      return callable.call();\n    } catch (ExpiredIteratorException e) {\n      throw e;\n    } catch (LimitExceededException | ProvisionedThroughputExceededException e) {\n      throw new KinesisClientThrottledException(\n          \"Too many requests to Kinesis. Wait some time and retry.\", e);\n    } catch (SdkServiceException e) {\n      if (e.isThrottlingException()\n          || SdkDefaultRetrySetting.RETRYABLE_STATUS_CODES.contains(e.statusCode())) {\n        throw new TransientKinesisException(\"Kinesis backend failed. Wait some time and retry.\", e);\n      }\n      throw e; // others, such as 4xx, are not retryable\n    } catch (SdkClientException e) {\n      if (SdkDefaultRetrySetting.RETRYABLE_EXCEPTIONS.contains(e.getClass())) {\n        throw new TransientKinesisException(\"Retryable failure\", e);\n      }\n      throw e;\n    } catch (Exception e) {\n      throw new RuntimeException(\"Unknown kinesis failure, when trying to reach kinesis\", e);\n    }\n  }\n\n  @Override","sourceCodeStart":209,"sourceCodeEnd":245,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/SimplifiedKinesisClient.java#L209-L245","documentation":"SimplifiedKinesisClient.wrapExceptions translates AWS SDK LimitExceededException and ProvisionedThroughputExceededException into KinesisClientThrottledException with this message. Kinesis is rate-limiting the account/stream: too many API calls per second (listShards, getShardIterator, getRecords, getBacklogBytes).","triggerScenarios":"Any of getShardIterator, listShardsFollowingClosedShard, getRecords, or getBacklogBytes invoked faster than the per-stream/per-shard limits allow — e.g. many consumers listing shards concurrently or frequent getRecords polling on a high-shard-count stream.","commonSituations":"Scaling a Beam pipeline to many workers all hammering the Kinesis API; small stream with aggressive polling frequency; shared AWS account where other applications consume the Kinesis throughput quota.","solutions":["Add exponential backoff with jitter and retry the request after a delay (KinesisClientThrottledException is intended to be retried).","Reduce polling frequency / batch size and consolidate shard-iterator requests; avoid listing shards from every worker.","Request a Kinesis service quota increase or shard the workload across more streams/shards."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (KinesisClientThrottledException e) {\n  Thread.sleep(backoffMs);\n  backoffMs = Math.min(backoffMs * 2, maxBackoffMs);\n  // retry\n}","preventionTips":["Use exponential backoff with jitter on all Kinesis API calls","Limit concurrent workers/polling rates against a stream","Monitor CloudWatch ThrottledRequests metrics"],"tags":["kinesis","throttling","rate-limit","retry"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}