apache/beam · warning · TransientKinesisException

Kinesis backend failed. Wait some time and retry.

Error message

Kinesis backend failed. Wait some time and retry.

What it means

wrapExceptions converts SdkServiceException responses that AWS marks as throttling or that carry retryable HTTP status codes into TransientKinesisException with this message. It signals a server-side Kinesis error the SDK considers safe to retry later (e.g. 5xx), unlike 4xx client errors which are rethrown as-is.

Source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/SimplifiedKinesisClient.java:232

   * Wraps Amazon specific exceptions into more friendly format.
   *
   * @throws TransientKinesisException - in case of recoverable situation, i.e. the request rate is
   *     too high, Kinesis remote service failed, network issue, etc.
   * @throws ExpiredIteratorException - if iterator needs to be refreshed
   * @throws RuntimeException - in all other cases
   */
  private static <T> T wrapExceptions(Callable<T> callable) throws TransientKinesisException {
    try {
      return callable.call();
    } catch (ExpiredIteratorException e) {
      throw e;
    } catch (LimitExceededException | ProvisionedThroughputExceededException e) {
      throw new KinesisClientThrottledException(
          "Too many requests to Kinesis. Wait some time and retry.", e);
    } catch (SdkServiceException e) {
      if (e.isThrottlingException()
          || SdkDefaultRetrySetting.RETRYABLE_STATUS_CODES.contains(e.statusCode())) {
        throw new TransientKinesisException("Kinesis backend failed. Wait some time and retry.", e);
      }
      throw e; // others, such as 4xx, are not retryable
    } catch (SdkClientException e) {
      if (SdkDefaultRetrySetting.RETRYABLE_EXCEPTIONS.contains(e.getClass())) {
        throw new TransientKinesisException("Retryable failure", e);
      }
      throw e;
    } catch (Exception e) {
      throw new RuntimeException("Unknown kinesis failure, when trying to reach kinesis", e);
    }
  }

  @Override
  public void close() throws Exception {
    try (AutoCloseable c1 = kinesis;
        AutoCloseable c2 = cloudWatch) {
      // nothing to do
    }

View on GitHub (pinned to 12126d8942)

Solutions

1. Retry the operation with exponential backoff — TransientKinesisException explicitly indicates the failure is temporary.
2. Verify AWS service health (Kinesis status on the AWS Health Dashboard).
3. Check region/endpoint configuration if 5xx errors persist (wrong endpoint can also surface as server errors).

catch (TransientKinesisException e) {
  // transient server-side failure: retry with backoff
}

When it happens

Trigger: getShardIterator, listShardsFollowingClosedShard, getRecords, or getBacklogBytes receives an SdkServiceException where isThrottlingException() is true or the status code is in SdkDefaultRetrySetting.RETRYABLE_STATUS_CODES (typically 500/502/503/504).

Common situations: Kinesis service degradation or internal server errors; transient AWS outages; excessive load on the Kinesis endpoint triggering server-side throttling not classified by the SDK exception types.

Related errors

• Retryable failure
• Transient exception occurred.
• Too many requests to Kinesis. Wait some time and retry.
• Failed to delete pendingDeletes.size() messages after retrie
• Pool {} - shard {} subscriber got error

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4865badf14d84b73. Report an issue: GitHub.