risingwavelabs/risingwave · error · SinkError::LanceDb

(schema conversion error)

Error message

(schema conversion error)

What it means

During sink validation, RisingWave converts its internal schema to an Arrow schema via LanceDbConvert::rw_schema_to_arrow_schema; if any RisingWave column type has no Arrow/Lance representation in the converter, the conversion fails and validation aborts.

Source

Thrown at src/connector/src/sink/lancedb.rs:244

        // Validate connection
        let conn = self.config.common.create_connection().await?;

        // Validate table exists and schema is compatible
        let table = self.config.common.open_table(&conn).await?;

        // Get the Lance table schema (Arrow schema)
        let lance_schema = table
            .schema()
            .await
            .context("failed to get LanceDB table schema")
            .map_err(SinkError::LanceDb)?;

        // Convert RW schema to arrow schema and compare
        let rw_schema = self.param.schema();
        let rw_arrow_schema = LanceDbConvert
            .rw_schema_to_arrow_schema(&rw_schema)
            .map_err(|e| SinkError::LanceDb(anyhow!(e)))?;

        validate_ordered_schema(&rw_arrow_schema, lance_schema.as_ref())?;

        Ok(())
    }

    fn is_coordinated_sink(&self) -> bool {
        true
    }

    async fn new_coordinator(
        &self,
        _iceberg_compact_stat_sender: Option<UnboundedSender<IcebergSinkCompactionUpdate>>,
    ) -> Result<SinkCommitCoordinator> {
        let committer =
            LanceDbSinkCommitter::new(self.config.clone(), self.param.sink_id.to_string()).await?;
        if Self::is_exactly_once(&self.param.properties)? {
            Ok(SinkCommitCoordinator::TwoPhase(Box::new(committer)))

View on GitHub (pinned to 6469eb736d)

Solutions

1. Read the wrapped conversion message to identify the unsupported column/type
2. Change the offending column to a supported type (int, bigint, varchar, boolean, float, timestamp, etc.)
3. Cast unsupported columns to supported types in the sink's query (SELECT col::varchar ...)
4. Upgrade RisingWave in case support for the type was added later

Example fix

// before
CREATE SINK s FROM mv_t WITH ('connector'='lancedb', ...); -- mv_t has INTERVAL column
// after
CREATE SINK s AS SELECT i::varchar AS i FROM mv_t WITH ('connector'='lancedb', ...);

// Validate column types before creating the sink
const supported = ['boolean','smallint','integer','bigint','real','double precision','varchar','timestamp','date','bytea'];
for (const col of sinkColumns)
  if (!supported.includes(col.type)) throw new Error(`unsupported type on ${col.name}: ${col.type}`);

catch (SinkError::LanceDb(e)) if e.to_string().contains("schema") { adjust offending column type or cast it in the sink SELECT }

When it happens

Trigger: Creating a LanceDB sink whose source schema contains an RW data type unsupported by the LanceDB arrow conversion (e.g., certain struct/list/interval types).

Common situations: Sinking tables with exotic types (nested structs, map types, intervals) into LanceDB; after upgrading RW and adding new types not yet mapped in LanceDbConvert.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Columns mismatch. RisingWave schema has {} fields, LanceDB t
• column order mismatch at position {}: LanceDB column is '{}'
• column '{}' type mismatch: LanceDB type is {:?}, RisingWave
• Join key types are not aligned: LHS: {outer_type:?}, RHS: {i
• Join key types are not aligned: LHS: {outer_type:?}, RHS: {i

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/89d853eafad7d987. Report an issue: GitHub.