Transient exception occurred during backlog estimation for…

Error message

Transient exception occurred during backlog estimation for stream {}.

What it means

When the reader estimates a split's backlog bytes, it calls getBacklogBytes which may throw TransientKinesisException for retryable Kinesis errors. The reader logs the warning and keeps the last successfully computed backlog value instead of failing the read.

Solutions

No action usually needed — the reader retains the last backlog estimate and retries on the next call
Reduce polling frequency of split/backlog computation in your runner config
Check Kinesis stream metrics for ThrottledRecords / high read throughput
Verify credentials and VPC endpoint connectivity if the warning persists

Defensive patterns

Strategy: retry

Try / catch

try {
  long backlog = kinesis.getBacklogBytes(stream, ts);
} catch (TransientKinesisException e) {
  // keep lastBacklogBytes; backoff and retry on next poll
  sleep(exponentialBackoff());
}

Prevention

Size stream throughput to stay below Kinesis read limits
Poll backlog estimation infrequently
Verify credentials and endpoint reachability before pipeline start
Treat last-known backlog as acceptable for autoscaling decisions

When it happens

Trigger: Kinesis returns a transient error (throttling, 5xx, temporary credential/network failure) during kinesis.getBacklogBytes(streamName, latestRecordTimestamp), typically invoked by the runner for autoscaling or split decisions.

Common situations: Autoscaling runners polling backlog on heavily loaded streams hitting Kinesis throttle limits; momentary network issues or AWS service degradation during a split computation.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/684d468cd4cc1096. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisReader.java:197

          latestRecordTimestamp,
          spec.getUpToDateThreshold());
      return 0L;
    }

    if (backlogBytesLastCheckTime.plus(backlogBytesCheckThreshold).isAfterNow()) {
      LOG.debug(
          "Split backlog bytes for {} stream with latest record timestamp {}: {} (cached value)",
          spec.getStreamName(),
          latestRecordTimestamp,
          lastBacklogBytes);
      return lastBacklogBytes;
    }

    try {
      lastBacklogBytes = kinesis.getBacklogBytes(spec.getStreamName(), latestRecordTimestamp);
      backlogBytesLastCheckTime = Instant.now();
    } catch (TransientKinesisException e) {
      LOG.warn(
          "Transient exception occurred during backlog estimation for stream {}.",
          spec.getStreamName(),
          e);
    }
    LOG.info(
        "Split backlog bytes for {} stream with {} latest record timestamp: {}",
        spec.getStreamName(),
        latestRecordTimestamp,
        lastBacklogBytes);
    return lastBacklogBytes;
  }

  ShardReadersPool createShardReadersPool() throws TransientKinesisException {
    return new ShardReadersPool(spec, kinesis, initCheckpoint);
  }
}

View on GitHub (pinned to 12126d8942)