apache/beam · warning

Transient exception occurred.

Error message

Transient exception occurred.

What it means

readLoop caught a TransientKinesisException from a Kinesis call (getRecords/iterator refresh). The exception is logged and the loop continues, retrying the read on the next iteration — the SDK classifies it as retryable. Persistent occurrences indicate ongoing Kinesis-side throttling or limit issues.

Source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/ShardReadersPool.java:187

          // Wait until all records from already closed shard are taken from the buffer and only
          // then start reading successive shards. This guarantees that checkpoints will contain
          // either parent or child shard and never both. Such approach allows for more
          // straightforward checkpoint restoration than in a case when new shards are read
          // immediately.
          waitUntilAllShardRecordsRead(shardRecordsIterator);
          readFromSuccessiveShards(shardRecordsIterator);
          break;
        }
      } catch (KinesisClientThrottledException e) {
        try {
          rateLimiter.onThrottle(e);
        } catch (InterruptedException ex) {
          LOG.warn("Thread was interrupted, finishing the read loop", ex);
          Thread.currentThread().interrupt();
          break;
        }
      } catch (TransientKinesisException e) {
        LOG.warn("Transient exception occurred.", e);
      } catch (InterruptedException e) {
        LOG.warn("Thread was interrupted, finishing the read loop", e);
        Thread.currentThread().interrupt();
        break;
      } catch (Throwable e) {
        LOG.error("Unexpected exception occurred", e);
      }
    }
    LOG.info("Kinesis Shard read loop has finished");
  }

  CustomOptional<KinesisRecord> nextRecord() {
    try {
      KinesisRecord record = recordsQueue.poll(QUEUE_POLL_TIMEOUT_MS, MILLISECONDS);
      if (record == null) {
        return CustomOptional.absent();
      }
      shardIteratorsMap.get().get(record.getShardId()).ackRecord(record);

View on GitHub (pinned to 12126d8942)

Solutions

1. Monitor the frequency; occasional occurrences can be ignored due to automatic retry.
2. Increase stream shard count / consumer capacity if throughput errors recur.
3. Add or strengthen retry backoff configuration in the Kinesis IO / AWS client.
4. Check AWS service health and region status if errors cluster in time.

// Pre-check stream capacity:
awsCli kinesis describe-stream-summary --stream-name ... // compare shards vs consumer load

try {
  records = kinesis.getRecords(request);
} catch (TransientKinesisException e) {
  LOG.warn("Transient exception occurred.", e); // loop retries automatically
}

When it happens

Trigger: ShardReadersPool.readLoop calls Kinesis getRecords / getShardIterator and the AWS client returns a retryable error (LimitExceededException, ProvisionedThroughputExceeded, 5xx) wrapped as TransientKinesisException.

Common situations: Shard hot-spots exceeding Kinesis throughput; too many concurrent consumers on one stream; transient AWS outages; shard iterator expiry scenarios.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Kinesis backend failed. Wait some time and retry.
• Too many requests to Kinesis. Wait some time and retry.
• Retryable failure
• Failed to delete pendingDeletes.size() messages after retrie
• Pool {} - shard {} subscriber got error

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/416712cc9a7a404b. Report an issue: GitHub.