risingwavelabs/risingwave · error · SinkError::Config

intermediate.table.name is required for non-append-only sink

Error message

intermediate.table.name is required for non-append-only sink

What it means

For a non-append-only Snowflake/Redshift S3 sink, writes go to an intermediate CDC table rather than the final table; `new` therefore requires `config.cdc_table` (`intermediate.table.name`) to be present. When it is None for a non-append-only sink, this SinkError::Config is returned.

Source

Thrown at src/connector/src/sink/snowflake_redshift/redshift.rs:330

        let schema = param.schema();
        if config.with_s3 {
            let s3_writer = SnowflakeRedshiftSinkS3Writer::new(
                config.s3_inner.ok_or_else(|| {
                    SinkError::Config(anyhow!("S3 configuration is required for S3 sink"))
                })?,
                schema,
                is_append_only,
                config.table,
            )?;
            Ok(Self::S3(s3_writer))
        } else {
            let (writer_schema, writer_table) = if is_append_only {
                (config.schema.clone(), config.table.clone())
            } else {
                (
                    config.intermediate_schema.clone().or(config.schema.clone()),
                    config.cdc_table.clone().ok_or_else(|| {
                        SinkError::Config(anyhow!(
                            "intermediate.table.name is required for non-append-only sink"
                        ))
                    })?,
                )
            };
            param.properties.remove("schema");
            param.properties.remove("schema.name");
            if let Some(writer_schema) = writer_schema {
                param.properties.insert("schema".to_owned(), writer_schema);
            }
            let jdbc_writer = SnowflakeRedshiftSinkJdbcWriter::new(
                is_append_only,
                writer_param,
                param,
                writer_table,
            )
            .await?;
            Ok(Self::Jdbc(jdbc_writer))

View on GitHub (pinned to 6469eb736d)

Solutions

1. Set `intermediate.table.name` (and `intermediate.schema.name` if needed) in the sink properties
2. Create the intermediate table in the target database with the sink schema
3. If updates are never expected, define the sink as append-only to use the final table directly

Example fix

// before
WITH ( connector = 'redshift', table = 'final_table', primary_key = 'id' )
-- after
WITH ( connector = 'redshift', table = 'final_table', primary_key = 'id', 'intermediate.table.name' = 'rw_cdc_staging' )

// before creating a non-append-only snowflake/redshift sink
if !is_append_only && !props.contains_key("intermediate.table.name") {
    return Err("intermediate.table.name is required for non-append-only sink");
}

When it happens

Trigger: Creating a non-append-only sink (data has updates/deletes) targeting Snowflake/Redshift with S3 staging but without the `intermediate.table.name` property; falling through the else branch of the writer schema/table selection.

Common situations: Upsert sinks missing the intermediate table config; users copying an append-only sink definition and adding a primary key (making it non-append-only) without adding the intermediate table.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• SinkError::Config(anyhow!(e))
• intermediate.table.name is required for append-only sink
• Primary key columns not found. Please set the `primary_key`
• S3 configuration is required for S3 sink
• should have meta client

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/4b65d962903f588c. Report an issue: GitHub.