risingwavelabs/risingwave · error · SinkError::Kinesis

failed to send records. sent {} out of {}

Error message

failed to send records. sent {} out of {}

What it means

The PutRecords API call itself returned Err (not per-record entry failures). After decrementing and exhausting the no-progress retry counter, the sink returns the SDK error wrapped with a context showing how many records had been sent so far.

Source

Thrown at src/connector/src/sink/kinesis.rs:362

                                            total_count,
                                            "failed to send records. code: [{}], message: [{}]",
                                            result_entry.error_code.unwrap_or_default(),
                                            result_entry.error_message.unwrap_or_default()
                                        )
                                    }
                                }
                            } else {
                                start_idx += record_count;
                                // reset retry count when having progress
                                remaining_no_progress_retry_count = MAX_NO_PROGRESS_RETRY_COUNT;
                                // reset throttle delay when records can be fully sent.
                                throttle_delay = None;
                            }
                        }
                        Err(e) => {
                            remaining_no_progress_retry_count -= 1;
                            if remaining_no_progress_retry_count == 0 {
                                return Err(SinkError::Kinesis(anyhow!(e).context(format!(
                                    "failed to send records. sent {} out of {}",
                                    start_idx, total_count,
                                ))));
                            } else {
                                warn!(
                                    remaining_no_progress_retry_count,
                                    sent = start_idx,
                                    total_count,
                                    "failed to send records. err: [{:?}]",
                                    e.as_report(),
                                )
                            }
                        }
                    }
                }
                Ok(())
            }
            .boxed()

View on GitHub (pinned to 6469eb736d)

Solutions

1. Reduce batch size / check batch limits (max 500 records, 5MB per PutRecords call)
2. Check network connectivity and AWS endpoint reachability from the compute node
3. Refresh/fix AWS credentials and region configuration
4. Retry; the sink resumes from the checkpoint (records already sent are counted in the message)

// pre-flight: verify endpoint reachability and creds
nc -zv <kinesis-endpoint-host> 443
aws kinesis list-streams --region <region>

match sink.finish().await {
  Err(e) if e.to_string().contains("failed to send records") => {
    // transient SDK/transport error: rely on checkpoint replay
    retry_with_backoff(|| sink.write_all(), 5).await
  }
  r => r,
}

When it happens

Trigger: `put_records(...).send().await` returning Err during `finish` — network failures, timeouts, auth errors, or throttling of the API call itself — repeatedly until retries run out.

Common situations: Network partitions between RisingWave and AWS; oversized PutRecords batches exceeding 500 records/5MB limits returning validation errors; expired credentials; VPC endpoint issues.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• failed to send records. sent {} out of {}, last err: code: [
• failed to list kinesis shards
• failed to get row stream from mysql query
• Vault API returned error status: {} - {}
• Failed to get secret from Vault

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/5bc44465c89d5f2c. Report an issue: GitHub.