apache/beam · critical · IOException

Error writing to DynamoDB. Unprocessed items remaining

Error message

Error writing to DynamoDB. Unprocessed items remaining

What it means

DynamoDBIO's batch write retries unprocessed items using the configured backoff policy; after exhausting retries, if some items are still unprocessed, it increments the failure counter, logs them, and throws an IOException. This means DynamoDB persistently rejected part of the batch (throughput limits, item size, throttling).

Solutions

Enable DynamoDB auto-scaling or increase write capacity on target tables
Reduce batch size / rate limit the write via a rate limit policy in the pipeline
Check the RESUME_ERROR_LOG output to identify which items failed and why
Retry the failed pipeline range / use Beam retry plus a dead-letter queue for persistently failing items

Example fix

// before
DynamoDBIO.<T>write().withWriteResultMapper(...) // default backoff, large batches
// after
.apply("DynamoDB write", DynamoDBIO.<T>write()
    .withMaxBatchSize(20)
    .withRetryPolicy(_RETRY_POLICY)); // longer exponential backoff

Defensive patterns

Strategy: retry

Validate before calling

// pre-check capacity: use DescribeTable / CloudWatch WriteThrottleEvents before large writes

Try / catch

try { result = p.run().waitUntilFinish(); } catch (Exception e) { inspect DYNAMO_DB_WRITE_FAILURES metric; backfill failed range; }

Prevention

Enable auto-scaling / on-demand capacity on target tables
Limit batch size and add rate limiting upstream
Monitor CloudWatch WriteThrottleEvents for the table
Route persistent failures to a dead-letter queue

When it happens

Trigger: Calling flushBatch (from processElement or finishBundle) when batchWriteItem keeps returning non-empty unprocessedItems after all BackOff retries are exhausted.

Common situations: DynamoDB table provisioned throughput exceeded; hot partitions; very large batches near 25-item/16MB limits; sustained write rates above capacity; AWS throttling during traffic spikes.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6322b3ded6c0f909. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIO.java:463

          // Group values KV<tableName, writeRequest> by tableName
          // Note: The original order of arrival is lost reading the map entries.
          Map<String, List<WriteRequest>> writesPerTable =
              batch.values().stream()
                  .collect(groupingBy(KV::getKey, mapping(KV::getValue, toList())));

          // Backoff used to resume from partial failures
          BackOff resume = resumeBackoff.backoff();
          do {
            BatchWriteItemRequest batchRequest =
                BatchWriteItemRequest.builder().requestItems(writesPerTable).build();
            // If unprocessed items remain, we have to resume the operation (with backoff)
            writesPerTable = client.batchWriteItem(batchRequest).unprocessedItems();
          } while (!writesPerTable.isEmpty() && BackOffUtils.next(Sleeper.DEFAULT, resume));

          if (!writesPerTable.isEmpty()) {
            DYNAMO_DB_WRITE_FAILURES.inc();
            LOG.error(RESUME_ERROR_LOG, writesPerTable);
            throw new IOException(ERROR_UNPROCESSED_ITEMS);
          }
        } finally {
          batch.clear();
        }
      }

      @Teardown
      public void tearDown() {
        if (client != null) {
          client.close();
        }
      }
    }
  }
}

View on GitHub (pinned to 12126d8942)