risingwavelabs/risingwave · error · anyhow

Es sink only supports single pk or pk with delimiter option

Error message

Es sink only supports single pk or pk with delimiter option

What it means

When validating a remote Elasticsearch sink, `validate_remote_sink` requires that the downstream primary key either be a single column or, if it is a multi-column PK, that the `delimiter` option be provided so ES document ids can be built by joining PK values. Otherwise validation bails with this error.

Solutions

1. Add `delimiter = '<char>'` to the sink properties so multi-column PKs can be joined into document ids.
2. Restructure the sink to have a single-column primary key (e.g. via a view with a synthetic id).
3. If the sink is not actually Elasticsearch (`is_remote_es_sink` check), verify the sink name/type to confirm which validation path applies.

Example fix

// before
CREATE SINK es_sink FROM t WITH (connector='elasticsearch', type='append-only');  -- composite pk
// after
CREATE SINK es_sink FROM t WITH (connector='elasticsearch', type='append-only', delimiter='-');

if (connector === 'elasticsearch' && pkColumns.length > 1 && !opts.delimiter) throw new Error('composite pk requires delimiter option');

try { await createSink(cfg); } catch (e) { if (/single pk or pk with delimiter/.test(String(e))) { cfg.delimiter = '-'; await createSink(cfg); } else throw e; }

When it happens

Trigger: Create a remote Elasticsearch sink whose table has a composite (2+) primary key and no `delimiter` option in the sink properties.

Common situations: Users sink tables with natural composite keys (e.g. (user_id, event_id)) to ES without realizing document id generation needs a delimiter; frameworks auto-generating sinks from wide PK schemas.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• auth.method=key_pair_file must not set `password`
• auth.method=key_pair_file must not set `private_key_pem`
• auth.method=key_pair_file requires `private_key_file`
• auth.method=key_pair_object requires `private_key_pem`
• auth.method=password must not set…

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/fd6c1cc89e4c81b4. Report an issue: GitHub.

Appendix: source

Thrown at src/connector/src/sink/remote.rs:192

    }

    async fn validate(&self) -> Result<()> {
        validate_remote_sink(&self.param, Self::SINK_NAME).await?;
        Ok(())
    }
}

async fn validate_remote_sink(param: &SinkParam, sink_name: &str) -> ConnectorResult<()> {
    // if sink_name == OpenSearchJavaSink::SINK_NAME {
    //     risingwave_common::license::Feature::OpenSearchSink
    //         .check_available()
    //         .map_err(|e| anyhow::anyhow!(e))?;
    // }
    if is_remote_es_sink(sink_name)
        && param.downstream_pk_or_empty().len() > 1
        && !param.properties.contains_key(ES_OPTION_DELIMITER)
    {
        bail!("Es sink only supports single pk or pk with delimiter option");
    }
    // FIXME: support struct and array in stream sink
    param.columns.iter().try_for_each(|col| {
        match &col.data_type {
            DataType::Int16
                    | DataType::Int32
                    | DataType::Int64
                    | DataType::Float32
                    | DataType::Float64
                    | DataType::Boolean
                    | DataType::Decimal
                    | DataType::Timestamp
                    | DataType::Timestamptz
                    | DataType::Varchar
                    | DataType::Date
                    | DataType::Time
                    | DataType::Interval
                    | DataType::Jsonb

View on GitHub (pinned to 6469eb736d)