{"record":{"id":"684d468cd4cc1096","repo":"apache/beam","slug":"transient-exception-occurred-during-backlog-estimation-for","errorCode":null,"errorMessage":"Transient exception occurred during backlog estimation for stream {}.","messagePattern":"Transient exception occurred during backlog estimation for stream (.+?)\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisReader.java","lineNumber":197,"sourceCode":"          latestRecordTimestamp,\n          spec.getUpToDateThreshold());\n      return 0L;\n    }\n\n    if (backlogBytesLastCheckTime.plus(backlogBytesCheckThreshold).isAfterNow()) {\n      LOG.debug(\n          \"Split backlog bytes for {} stream with latest record timestamp {}: {} (cached value)\",\n          spec.getStreamName(),\n          latestRecordTimestamp,\n          lastBacklogBytes);\n      return lastBacklogBytes;\n    }\n\n    try {\n      lastBacklogBytes = kinesis.getBacklogBytes(spec.getStreamName(), latestRecordTimestamp);\n      backlogBytesLastCheckTime = Instant.now();\n    } catch (TransientKinesisException e) {\n      LOG.warn(\n          \"Transient exception occurred during backlog estimation for stream {}.\",\n          spec.getStreamName(),\n          e);\n    }\n    LOG.info(\n        \"Split backlog bytes for {} stream with {} latest record timestamp: {}\",\n        spec.getStreamName(),\n        latestRecordTimestamp,\n        lastBacklogBytes);\n    return lastBacklogBytes;\n  }\n\n  ShardReadersPool createShardReadersPool() throws TransientKinesisException {\n    return new ShardReadersPool(spec, kinesis, initCheckpoint);\n  }\n}\n","sourceCodeStart":179,"sourceCodeEnd":214,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisReader.java#L179-L214","documentation":"When the reader estimates a split's backlog bytes, it calls getBacklogBytes which may throw TransientKinesisException for retryable Kinesis errors. The reader logs the warning and keeps the last successfully computed backlog value instead of failing the read.","triggerScenarios":"Kinesis returns a transient error (throttling, 5xx, temporary credential/network failure) during kinesis.getBacklogBytes(streamName, latestRecordTimestamp), typically invoked by the runner for autoscaling or split decisions.","commonSituations":"Autoscaling runners polling backlog on heavily loaded streams hitting Kinesis throttle limits; momentary network issues or AWS service degradation during a split computation.","solutions":["No action usually needed — the reader retains the last backlog estimate and retries on the next call","Reduce polling frequency of split/backlog computation in your runner config","Check Kinesis stream metrics for ThrottledRecords / high read throughput","Verify credentials and VPC endpoint connectivity if the warning persists"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  long backlog = kinesis.getBacklogBytes(stream, ts);\n} catch (TransientKinesisException e) {\n  // keep lastBacklogBytes; backoff and retry on next poll\n  sleep(exponentialBackoff());\n}","preventionTips":["Size stream throughput to stay below Kinesis read limits","Poll backlog estimation infrequently","Verify credentials and endpoint reachability before pipeline start","Treat last-known backlog as acceptable for autoscaling decisions"],"tags":["kinesis","backlog","transient-error","autoscaling"],"backgroundTag":"request-timeout","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}