risingwavelabs/risingwave · critical · SinkError::DynamoDb

failed to write {} unprocessed items to DynamoDB sink after

Error message

failed to write {} unprocessed items to DynamoDB sink after {} retries

What it means

The sink writes rows via BatchWriteItem; items that come back as UNPROCESSED are retried up to batch_write_retry_times. If items remain unprocessed after all retries, write_chunk aborts with this error indicating the chunk was only partially written.

Source

Thrown at src/connector/src/sink/dynamodb.rs:593

                            match result {
                                Ok(output) => {
                                    let unprocessed_items =
                                        output.unprocessed_items().cloned().unwrap_or_default();
                                    if unprocessed_items.is_empty() {
                                        if retry_count > 0 {
                                            tracing::warn!(
                                                retry_count,
                                                consumed_capacity = ?output.consumed_capacity(),
                                                "DynamoDB batch write retry succeeded"
                                            );
                                        }
                                        return Ok(());
                                    }

                                    req_items = unprocessed_items.into_values().flatten().collect();
                                    if retry_count >= batch_write_retry_times {
                                        return Err(SinkError::DynamoDb(anyhow!(
                                            "failed to write {} unprocessed items to DynamoDB sink after {} retries",
                                            req_items.len(),
                                            batch_write_retry_times,
                                        )));
                                    }
                                }
                                Err(e) => {
                                    return Err(SinkError::DynamoDb(
                                        anyhow!(e).context("failed to write items to DynamoDB sink"),
                                    ));
                                }
                            }

                            retry_count += 1;
                            let Some(delay) = retry_backoff.next() else {
                                return Err(SinkError::DynamoDb(anyhow!(
                                    "failed to write {} unprocessed items to DynamoDB sink after {} retries",
                                    req_items.len(),

View on GitHub (pinned to 6469eb736d)

Solutions

Increase the table's write capacity (WCU or switch to on-demand/billing mode PAY_PER_REQUEST).
Retry the sink/restart it — already-written items may be deduplicated if the sink is idempotent; otherwise verify data consistency.
Avoid hot partition keys by revising the key design, and spread backfill load.
Check CloudWatch ThrottledRequests/WriteThrottleEvents metrics to confirm throttling and raise batch_write_retry_times if configured limits are too low.

Example fix

// before: table with 5 WCU, bulk backfill -> throttled unprocessed items
aws dynamodb update-table --table-name events --billing-mode PAY_PER_REQUEST
// after: restart the sink; unprocessed items can be written with adequate capacity

Defensive patterns

Strategy: retry

Validate before calling

// check throttle metrics before heavy writes
aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name WriteThrottleEvents --dimensions TableName=events ...

Try / catch

// classify as throttling/partial-write; back off and resume the sink
match write_result {
  Err(e) if e.to_string().contains("unprocessed items") => {
    alert_partially_written_chunk();
    sleep(exponential_backoff());
    resume_sink_from_checkpoint();
  }
  other => other,
}

Prevention

Provision adequate WCU or use PAY_PER_REQUEST billing before bulk backfills.
Avoid hot partition keys; spread writes across key space.
Monitor WriteThrottleEvents/UnprocessedItems CloudWatch metrics.
Schedule large backfills during low-traffic windows and increase retry budget.

When it happens

Trigger: BatchWriteItem repeatedly returns UnprocessedItems — typically due to provisioned/adaptive capacity exhaustion, throttling, or item-size/validation errors — across all retry rounds in write_chunk.

Common situations: Backfilling a large sink into a table with low write capacity (WCU); hot partition keys causing throttling; on-demand table in throttled state; DynamoDB returning transient 500s sustained beyond the retry budget.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/debe8c3bd1327880. Report an issue: GitHub.