{"record":{"id":"8312dafea5a06746","repo":"apache/beam","slug":"failed-to-extend-visibility-timeout-for-messages-size","errorCode":null,"errorMessage":"Failed to extend visibility timeout for messages.size() messages after retries retries","messagePattern":"Failed to extend visibility timeout for messages\\.size\\(\\) messages after retries retries","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsUnboundedReader.java","lineNumber":827,"sourceCode":"  void extendBatch(long nowMsSinceEpoch, List<KV<String, String>> messages, int extensionSec)\n      throws IOException {\n    int retries = 0;\n\n    Function<KV<String, String>, ChangeMessageVisibilityBatchRequestEntry> buildEntry =\n        kv ->\n            ChangeMessageVisibilityBatchRequestEntry.builder()\n                .visibilityTimeout(extensionSec)\n                .id(kv.getKey())\n                .receiptHandle(kv.getValue())\n                .build();\n\n    Map<String, ChangeMessageVisibilityBatchRequestEntry> pendingExtends =\n        messages.stream().collect(toMap(KV::getKey, buildEntry));\n\n    while (!pendingExtends.isEmpty()) {\n\n      if (retries >= BATCH_OPERATION_MAX_RETIRES) {\n        throw new IOException(\n            \"Failed to extend visibility timeout for \"\n                + messages.size()\n                + \" messages after \"\n                + retries\n                + \" retries\");\n      }\n\n      ChangeMessageVisibilityBatchResponse response =\n          sqsClient.changeMessageVisibilityBatch(\n              ChangeMessageVisibilityBatchRequest.builder()\n                  .queueUrl(queueUrl())\n                  .entries(pendingExtends.values())\n                  .build());\n\n      Map<Boolean, Set<String>> failures =\n          response.failed().stream()\n              .collect(partitioningBy(this::isHandleInvalid, mapping(e -> e.id(), toSet())));\n","sourceCodeStart":809,"sourceCodeEnd":845,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsUnboundedReader.java#L809-L845","documentation":"SqsUnboundedReader throws this IOException when a batch ChangeMessageVisibility request to SQS fails to extend visibility timeouts for all messages even after BATCH_OPERATION_MAX_RETIRES retries. Messages whose visibility expires will be redelivered, so the reader aborts rather than silently lose delivery guarantees.","triggerScenarios":"SQS changeMessageVisibilityBatch repeatedly returns per-message failures (or throttles/errors) for the remaining pendingExtends entries across all retry attempts in SqsUnboundedReader.","commonSituations":"SQS throttling under high throughput, messages already deleted or processed by another consumer, IAM policy lacking changeMessageVisibility permission, messages past the 12-hour total visibility extension limit.","solutions":["Verify the AWS credentials/IAM policy grant sqs:ChangeMessageVisibility on the queue.","Check for SQS throttling (ThrottledException) and reduce pipeline parallelism or add backoff.","Ensure no other consumer is deleting the same messages concurrently (duplicate workers sharing a queue).","Increase retries headroom or batch sizes, and check queue visibility timeout settings relative to batch processing time."],"exampleFix":"// before\nthrow new IOException(\"Failed to extend visibility timeout for \" + messages.size() + \" messages after \" + retries + \" retries\");\n// after\n// inspect per-entry failures before giving up, and log/omit only truly failed message IDs\nfor (ChangeMessageVisibilityBatchResult r : results) {\n  failedIds.addAll(r.getFailed()); // surface which messages failed and why\n}","handlingStrategy":"retry","validationCode":"// Pre-flight: ensure IAM policy allows sqs:ChangeMessageVisibility on the queue\naws sqs get-queue-attributes --queue-url $QUEUE_URL --attribute-names Policy","typeGuard":null,"tryCatchPattern":"try { reader.advance(); } catch (IOException e) {\n  if (e.getMessage().contains(\"Failed to extend visibility timeout\")) {\n    // backoff and restart the source; messages will be redelivered\n    Thread.sleep(TimeUnit.MINUTES.toMillis(1));\n  }\n}","preventionTips":["Grant sqs:ChangeMessageVisibility to runner worker roles","Set queue visibility timeout comfortably above batch processing time","Monitor SQS ThrottledException metrics and add client-side rate limiting","Avoid multiple consumers sharing the same queue"],"tags":["aws","sqs","visibility-timeout","retries-exhausted"],"backgroundTag":"api-error-response","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}