risingwavelabs/risingwave · error · SinkError::BigQuery

MAP is not supported for BigQuery sink.

Error message

MAP is not supported for BigQuery sink.

What it means

The BigQuery sink's type mapping function `get_string_and_check_support_from_datatype` encountered a RisingWave `DataType::Map` column, which has no BigQuery equivalent in this sink's mapping table. BigQuery has no native MAP type, so the sink refuses to proceed rather than silently coercing data. The error is wrapped as `SinkError::BigQuery` and aborts sink creation or type validation.

Source

Thrown at src/connector/src/sink/big_query.rs:430

                        Self::get_string_and_check_support_from_datatype(datatype)?;
                    elements_vec.push(format!("{} {}", name, element_string));
                }
                Ok(format!("STRUCT<{}>", elements_vec.join(", ")))
            }
            DataType::List(l) => {
                let element_string = Self::get_string_and_check_support_from_datatype(l.elem())?;
                Ok(format!("ARRAY<{}>", element_string))
            }
            DataType::Bytea => Ok("BYTES".to_owned()),
            DataType::Jsonb => Ok("JSON".to_owned()),
            DataType::Variant => Err(SinkError::BigQuery(anyhow::anyhow!(
                "VARIANT is not supported for BigQuery sink."
            ))),
            DataType::Serial => Ok("INT64".to_owned()),
            DataType::Int256 => Err(SinkError::BigQuery(anyhow::anyhow!(
                "INT256 is not supported for BigQuery sink."
            ))),
            DataType::Map(_) => Err(SinkError::BigQuery(anyhow::anyhow!(
                "MAP is not supported for BigQuery sink."
            ))),
            DataType::Vector(_) => Err(SinkError::BigQuery(anyhow::anyhow!(
                "VECTOR is not supported for BigQuery sink."
            ))),
        }
    }

    fn map_field(rw_field: &Field) -> Result<TableFieldSchema> {
        let tfs = match &rw_field.data_type {
            DataType::Boolean => TableFieldSchema::bool(&rw_field.name),
            DataType::Int16 | DataType::Int32 | DataType::Int64 | DataType::Serial => {
                TableFieldSchema::integer(&rw_field.name)
            }
            DataType::Float32 => {
                return Err(SinkError::BigQuery(anyhow::anyhow!(
                    "REAL is not supported for BigQuery sink. Please convert to FLOAT64 or other supported types."
                )));

View on GitHub (pinned to 6469eb736d)

Solutions

  1. Change the column type to JSONB before sinking (e.g. recreate the source/materialized view with the map cast or stored as JSONB).
  2. Restructure the query feeding the sink so map data is unnested into rows or flattened into separate scalar columns.
  3. Cast the MAP column to VARCHAR/JSONB in the sink's query definition if the sink supports a SELECT/query-based definition.

Example fix

// before: sink references a MAP column
CREATE SINK s FROM mv WITH (...) TYPE bigquery;
-- mv has col map_col MAP<VARCHAR,INT>

// after: cast to JSONB in the materialized view feeding the sink
CREATE MATERIALIZED VIEW mv2 AS SELECT map_to_json(map_col) AS map_col FROM mv;
CREATE SINK s FROM mv2 WITH (...) TYPE bigquery;
Defensive patterns

Strategy: validation

Validate before calling

-- Before creating the sink, verify no MAP columns exist:
SELECT column_name, data_type
FROM rw_catalog.rw_columns
WHERE relation_id = ('<schema>.<mv_name>')::regclass
  AND data_type ILIKE 'map%';

Type guard

fn is_bigquery_supported(dt: &DataType) -> bool {
    !matches!(dt, DataType::Map(_) | DataType::Vector(_) | DataType::Int256 | DataType::Variant)
}

Prevention

When it happens

Trigger: Creating or validating a BigQuery sink whose schema contains a column of RisingWave type MAP (e.g. `MAP<VARCHAR, INT>`), either at top level or encountered during schema resolution.

Common situations: Sinking a materialized view or table that uses map-typed columns (often produced by aggregations or JSON parsing) into BigQuery; users assume maps serialize like JSON but this sink rejects them outright.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/c409bf7346ca6785. Report an issue: GitHub.