{"record":{"id":"debe8c3bd1327880","repo":"risingwavelabs/risingwave","slug":"failed-to-write-unprocessed-items-to-dynamodb-s","errorCode":null,"errorMessage":"failed to write {} unprocessed items to DynamoDB sink after {} retries","messagePattern":"failed to write (.+?) unprocessed items to DynamoDB sink after (.+?) retries","errorType":"error_code","errorClass":"SinkError::DynamoDb","httpStatus":null,"severity":"critical","filePath":"src/connector/src/sink/dynamodb.rs","lineNumber":593,"sourceCode":"\n                            match result {\n                                Ok(output) => {\n                                    let unprocessed_items =\n                                        output.unprocessed_items().cloned().unwrap_or_default();\n                                    if unprocessed_items.is_empty() {\n                                        if retry_count > 0 {\n                                            tracing::warn!(\n                                                retry_count,\n                                                consumed_capacity = ?output.consumed_capacity(),\n                                                \"DynamoDB batch write retry succeeded\"\n                                            );\n                                        }\n                                        return Ok(());\n                                    }\n\n                                    req_items = unprocessed_items.into_values().flatten().collect();\n                                    if retry_count >= batch_write_retry_times {\n                                        return Err(SinkError::DynamoDb(anyhow!(\n                                            \"failed to write {} unprocessed items to DynamoDB sink after {} retries\",\n                                            req_items.len(),\n                                            batch_write_retry_times,\n                                        )));\n                                    }\n                                }\n                                Err(e) => {\n                                    return Err(SinkError::DynamoDb(\n                                        anyhow!(e).context(\"failed to write items to DynamoDB sink\"),\n                                    ));\n                                }\n                            }\n\n                            retry_count += 1;\n                            let Some(delay) = retry_backoff.next() else {\n                                return Err(SinkError::DynamoDb(anyhow!(\n                                    \"failed to write {} unprocessed items to DynamoDB sink after {} retries\",\n                                    req_items.len(),","sourceCodeStart":575,"sourceCodeEnd":611,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/sink/dynamodb.rs#L575-L611","documentation":"The sink writes rows via BatchWriteItem; items that come back as UNPROCESSED are retried up to batch_write_retry_times. If items remain unprocessed after all retries, write_chunk aborts with this error indicating the chunk was only partially written.","triggerScenarios":"BatchWriteItem repeatedly returns UnprocessedItems — typically due to provisioned/adaptive capacity exhaustion, throttling, or item-size/validation errors — across all retry rounds in write_chunk.","commonSituations":"Backfilling a large sink into a table with low write capacity (WCU); hot partition keys causing throttling; on-demand table in throttled state; DynamoDB returning transient 500s sustained beyond the retry budget.","solutions":["Increase the table's write capacity (WCU or switch to on-demand/billing mode PAY_PER_REQUEST).","Retry the sink/restart it — already-written items may be deduplicated if the sink is idempotent; otherwise verify data consistency.","Avoid hot partition keys by revising the key design, and spread backfill load.","Check CloudWatch ThrottledRequests/WriteThrottleEvents metrics to confirm throttling and raise batch_write_retry_times if configured limits are too low."],"exampleFix":"// before: table with 5 WCU, bulk backfill -> throttled unprocessed items\naws dynamodb update-table --table-name events --billing-mode PAY_PER_REQUEST\n// after: restart the sink; unprocessed items can be written with adequate capacity","handlingStrategy":"retry","validationCode":"// check throttle metrics before heavy writes\naws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name WriteThrottleEvents --dimensions TableName=events ...","typeGuard":null,"tryCatchPattern":"// classify as throttling/partial-write; back off and resume the sink\nmatch write_result {\n  Err(e) if e.to_string().contains(\"unprocessed items\") => {\n    alert_partially_written_chunk();\n    sleep(exponential_backoff());\n    resume_sink_from_checkpoint();\n  }\n  other => other,\n}","preventionTips":["Provision adequate WCU or use PAY_PER_REQUEST billing before bulk backfills.","Avoid hot partition keys; spread writes across key space.","Monitor WriteThrottleEvents/UnprocessedItems CloudWatch metrics.","Schedule large backfills during low-traffic windows and increase retry budget."],"tags":["dynamodb","sink","throttling","retries","write-failure"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}