apache/beam · warning · TransientKinesisException

Retryable failure

Error message

Retryable failure

What it means

wrapExceptions catches SdkClientException and, if the exception class is in SdkDefaultRetrySetting.RETRYABLE_EXCEPTIONS (client-side failures the AWS SDK deems retryable, e.g. connection resets, IO interruptions), wraps it in TransientKinesisException with the message 'Retryable failure'. It means the request never succeeded at the HTTP layer, not that Kinesis returned an error.

Source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/SimplifiedKinesisClient.java:237

   * @throws RuntimeException - in all other cases
   */
  private static <T> T wrapExceptions(Callable<T> callable) throws TransientKinesisException {
    try {
      return callable.call();
    } catch (ExpiredIteratorException e) {
      throw e;
    } catch (LimitExceededException | ProvisionedThroughputExceededException e) {
      throw new KinesisClientThrottledException(
          "Too many requests to Kinesis. Wait some time and retry.", e);
    } catch (SdkServiceException e) {
      if (e.isThrottlingException()
          || SdkDefaultRetrySetting.RETRYABLE_STATUS_CODES.contains(e.statusCode())) {
        throw new TransientKinesisException("Kinesis backend failed. Wait some time and retry.", e);
      }
      throw e; // others, such as 4xx, are not retryable
    } catch (SdkClientException e) {
      if (SdkDefaultRetrySetting.RETRYABLE_EXCEPTIONS.contains(e.getClass())) {
        throw new TransientKinesisException("Retryable failure", e);
      }
      throw e;
    } catch (Exception e) {
      throw new RuntimeException("Unknown kinesis failure, when trying to reach kinesis", e);
    }
  }

  @Override
  public void close() throws Exception {
    try (AutoCloseable c1 = kinesis;
        AutoCloseable c2 = cloudWatch) {
      // nothing to do
    }
  }

  /** Memoizing supplier that closes resources appropriately. */
  private static class LazyResource<T extends AutoCloseable> implements Supplier<T>, AutoCloseable {
    private final Supplier<T> initializer;

View on GitHub (pinned to 12126d8942)

Solutions

1. Retry with backoff — the wrapper deliberately marks this as transient.
2. Check network connectivity and DNS from the job environment to the Kinesis endpoint.
3. Configure the AWS SDK client with retry policy and increased timeouts for unstable networks.

catch (TransientKinesisException e) {
  // retryable client-side/network failure: retry with backoff
}

When it happens

Trigger: A Kinesis API call from getShardIterator/listShardsFollowingClosedShard/getRecords/getBacklogBytes fails client-side with a retryable SdkClientException subclass such as IOException-based connection failures or interrupted requests.

Common situations: Network flakiness between the job and AWS; DNS resolution hiccups; worker network interruptions in a cluster environment; proxy/firewall dropping connections mid-request.

Related errors

• Kinesis backend failed. Wait some time and retry.
• Too many requests to Kinesis. Wait some time and retry.
• Failed to call Rate Limit Service
• Unable to get BigQuery response after retrying %d times usin
• Request failed after exhausting retries. Max retries: {maxRe

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/61e405bd712afbf7. Report an issue: GitHub.