risingwavelabs/risingwave · error · SinkError::Config

S3 configuration is required for redshift s3 sink

Error message

S3 configuration is required for redshift s3 sink

What it means

The periodic COPY task (`run_periodic_query_task`) requires S3 as the staging area between RisingWave and Redshift; this is thrown when `config.s3_inner` is None at the copy timer tick. `s3_inner` is populated only when the sink is configured with `with_s3`/S3 options, so this indicates the Redshift sink was configured for periodic COPY without valid S3 settings.

Source

Thrown at src/connector/src/sink/snowflake_redshift/redshift.rs:598

        mut shutdown_receiver: tokio::sync::mpsc::UnboundedReceiver<()>,
    ) {
        let mut copy_timer = interval(Duration::from_secs(write_intermediate_interval_seconds));
        copy_timer.set_missed_tick_behavior(MissedTickBehavior::Skip);
        let mut merge_timer = interval(Duration::from_secs(writer_target_interval_seconds));
        merge_timer.set_missed_tick_behavior(MissedTickBehavior::Skip);

        loop {
            tokio::select! {
                _ = shutdown_receiver.recv() => break,
                _ = merge_timer.tick(), if merge_into_sql.is_some() => {
                    if let Some(sql) = &merge_into_sql && let Err(e) = client.execute_sql_sync(sql.clone()).await {
                        tracing::warn!("Failed to execute periodic query for table {}: {}", config.table, e.as_report());
                    }
                },
                _ = copy_timer.tick(), if need_copy_into => {
                    if let Err(e) = async {
                        let s3_inner = config.s3_inner.as_ref().ok_or_else(|| {
                            SinkError::Config(anyhow!("S3 configuration is required for redshift s3 sink"))
                        })?;
                        Self::flush_manifest_to_redshift(&client, &config,s3_inner, is_append_only).await?;
                        Ok::<(),SinkError>(())
                    }.await {
                        tracing::error!("Failed to execute copy into task for sink id {}: {}", sink_id, e.as_report());
                    }
                }
            }
        }
        tracing::info!("Periodic query task stopped for sink id {}", sink_id);
    }
}

impl Drop for RedshiftSinkCommitter {
    fn drop(&mut self) {
        // Send shutdown signal to the periodic task
        if let Some(shutdown_sender) = &self.shutdown_sender
            && let Err(e) = shutdown_sender.send(())

View on GitHub (pinned to 6469eb736d)

Solutions

1. Add the required S3 `with` options (s3.bucket, s3.path, s3.region, aws credentials) to the sink
2. Fix typo'd option keys so `s3_inner` gets populated
3. If COPY via S3 is not intended, remove the copy/s3-related options so the periodic copy task is not scheduled

Example fix

// before
WITH ('connector'='redshift', 'copy.interval'='60s')
// after
WITH ('connector'='redshift', 'copy.interval'='60s',
      's3.bucket'='my-bucket', 's3.path'='rw-sink', 's3.region'='us-east-1',
      'aws.credentials.access_key_id'='...', 'aws.credentials.secret_access_key'='...')

if need_copy_into && config.s3_inner.is_none() {
    return Err("S3 options are required when copy is enabled for Redshift sink");
}

fn s3_ready(cfg: &RedshiftConfig) -> bool { cfg.s3_inner.is_some() }

When it happens

Trigger: The periodic copy task ticks (`need_copy_into == true`) but `s3_inner` is absent because the sink lacks S3 options or the S3 config failed to deserialize into `s3_inner`.

Common situations: Redshift sink DDL missing `s3.bucket`/`s3.path`/`s3.region`/credentials; typo in S3 option keys so they are not parsed; copy interval configured while S3 options omitted.

Understand the failure class

Background: "X is required", "must be set", "cannot be empty": the missing-required-config error family, from Vertex AI project/location to WeChat keys — this error's family across 18 libraries.

Related errors

• {e}
• `{}` must be {}, or {}
• SinkError::Config(anyhow!(e))
• intermediate.table.name is required for append-only sink
• Primary key columns not found. Please set the `primary_key`

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/da5ac04918da185c. Report an issue: GitHub.