risingwavelabs/risingwave · error · SinkError::Http

Turbopuffer document id column must be an integer or varchar

Error message

Turbopuffer document id column must be an integer or varchar

What it means

The Turbopuffer sink derives each document's id from the primary key column of the streamed row. Turbopuffer only accepts unsigned 64-bit integers, UUIDs, or strings up to 64 bytes, so RisingWave maps Int16/Int32/Int64/Serial/VARCHAR primary keys to document ids. When the primary key datum is any other type (e.g. Float, Decimal, Boolean, Struct), id_for_row throws this error. The caller (write_chunk) logs a warning and skips the row, so the data is silently dropped.

Source

Thrown at src/connector/src/sink/turbopuffer.rs:534

    // RisingWave UUID IDs can be represented with varchar.
    fn id_for_row(&self, row: &impl Row) -> Result<DocumentId> {
        let datum = row.datum_at(self.pk_index).ok_or_else(|| {
            SinkError::Http(anyhow!("Turbopuffer document id column cannot be null"))
        })?;
        match datum {
            ScalarRefImpl::Int16(value) => Ok(document_id_from_i64(value as i64)),
            ScalarRefImpl::Int32(value) => Ok(document_id_from_i64(value as i64)),
            ScalarRefImpl::Int64(value) => Ok(document_id_from_i64(value)),
            ScalarRefImpl::Serial(value) => Ok(document_id_from_i64(value.into_inner())),
            ScalarRefImpl::Utf8(value) => {
                if value.len() > 64 {
                    return Err(SinkError::Http(anyhow!(
                        "Turbopuffer string document id exceeds 64 bytes"
                    )));
                }
                Ok(DocumentId::String(value.to_owned()))
            }
            _ => Err(SinkError::Http(anyhow!(
                "Turbopuffer document id column must be an integer or varchar"
            ))),
        }
    }

    fn upsert_row(&self, row: &impl Row, id: DocumentId) -> Result<Map<String, Value>> {
        let mut value = self.row_encoder.encode(row)?;
        value.insert(
            "id".to_owned(),
            serde_json::to_value(id).expect("serialize document id"),
        );
        Ok(value)
    }

    fn request_body(
        &self,
        upsert_rows: Vec<Map<String, Value>>,
        deletes: Vec<DocumentId>,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Change the sink's source table/materialized view primary key to BIGINT or VARCHAR.
2. Cast the primary key to VARCHAR or BIGINT in an intermediate materialized view before sinking.
3. Use a different sink that supports arbitrary primary key types.

Example fix

// before: PRIMARY KEY (price DECIMAL) sinking to turbopuffer
// after
CREATE MATERIALIZED VIEW mv_ids AS SELECT price::VARCHAR AS price_id, * FROM source;
-- sink from mv_ids with PRIMARY KEY (price_id)

// ensure the sink PK column is BIGINT or VARCHAR before creating the sink
-- SELECT data_type FROM rw_catalog.rw_columns WHERE name = '<pk_col>' AND relation_id = <table_id>;
fn is_valid_tpuf_pk(t: &str) -> bool {
    matches!(t, "SMALLINT" | "INTEGER" | "BIGINT" | "SERIAL" | "VARCHAR")
}

fn valid_doc_id(v: &ScalarRefImpl) -> bool {
    matches!(v, ScalarRefImpl::Int16(_) | ScalarRefImpl::Int32(_) | ScalarRefImpl::Int64(_) | ScalarRefImpl::Serial(_) | ScalarRefImpl::Utf8(_))
}

When it happens

Trigger: Creating a Turbopuffer sink whose primary key column is not an integer type (SMALLINT/INT/BIGINT/SERIAL) or VARCHAR — e.g. a FLOAT, DECIMAL, BOOLEAN, or STRUCT primary key — and writing a row through write_chunk.

Common situations: Defining a materialized view or table with a decimal or float primary key for data that will be sunk to Turbopuffer; forgetting that Turbopuffer ids must be integers or short strings; legacy schemas with composite or non-scalar primary keys.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• The key encode is BYTES, but the primary key column {} has t
• The key encode is TEXT, but the primary key column {} has ty
• `{MEMBER_NAME}` must be set to `varchar` and `primary_key`
• Turbopuffer sink requires exactly one primary_key column
• Turbopuffer document id column must be an integer or varchar

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/da0510d304aed16f. Report an issue: GitHub.