apache/beam · error · IOException

Failed to delete pendingDeletes.size() messages after…

Error message

Failed to delete pendingDeletes.size() messages after retries retries

What it means

SqsUnboundedReader.deleteBatch retries deleting message batches up to BATCH_OPERATION_MAX_RETIRES; if pendingDeletes is still non-empty after exhausting retries it throws IOException 'Failed to delete N messages after R retries'. Messages were read but their receipt handles could not be deleted from the queue in time.

Solutions

1. Check the SQS queue permissions and credentials for DeleteMessageBatch
2. Reduce read throughput / increase SQS queue capacity to avoid throttling
3. Ensure messages are deleted within the visibility timeout window
4. Inspect AWS-side failures (CloudWatch metrics) and transient errors; messages will reappear after visibility timeout so downstream processing must be idempotent

Example fix

// before
// visibility timeout of 30s with batch processing taking minutes
// after
// changeVisibilityTimeout(queueUrl, Duration.ofMinutes(10));
// or process/delete batches well within the visibility window

// pre-check queue accessibility before reading
sqs.getQueueUrl(GetQueueUrlRequest.builder().queueName(name).build());

try { reader.deleteBatch(handles); } catch (IOException e) { /* messages will reappear after visibility timeout; ensure idempotent processing */ }

When it happens

Trigger: SQS DeleteMessageBatch repeatedly failing/partially failing (transient AWS errors, invalid/expired receipt handles, throttling) across all retry attempts while the reader keeps retrying.

Common situations: Long-running reads where visibility timeout expired and receipt handles became stale; SQS throttling under high throughput; temporary AWS outage/credential issues during deletes.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Error matching file spec
• Error when writing batch.
• Error writing to ES after
• Error writing to Solr after
• Error writing to Solr (no attempt made to retry)

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4af20b559f21450a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsUnboundedReader.java:589

   * <p>CAUTION: May be invoked from a separate thread.
   */
  private void deleteBatch(List<String> receiptHandles) throws IOException {
    int retries = 0;

    FunctionWithIndex<String, DeleteMessageBatchRequestEntry> buildEntry =
        (handle, id) ->
            DeleteMessageBatchRequestEntry.builder()
                .id(Long.toString(id))
                .receiptHandle(handle)
                .build();

    Map<String, DeleteMessageBatchRequestEntry> pendingDeletes =
        mapWithIndex(receiptHandles.stream(), buildEntry).collect(toMap(e -> e.id(), identity()));

    while (!pendingDeletes.isEmpty()) {

      if (retries >= BATCH_OPERATION_MAX_RETIRES) {
        throw new IOException(
            "Failed to delete "
                + pendingDeletes.size()
                + " messages after "
                + retries
                + " retries");
      }

      DeleteMessageBatchResponse result =
          sqsClient.deleteMessageBatch(
              DeleteMessageBatchRequest.builder()
                  .queueUrl(queueUrl())
                  .entries(pendingDeletes.values())
                  .build());

      Map<Boolean, Set<String>> failures =
          result.failed().stream()
              .collect(partitioningBy(this::isHandleInvalid, mapping(e -> e.id(), toSet())));

View on GitHub (pinned to 12126d8942)