risingwavelabs/risingwave · error · SinkError

convert parquet batch of {path}

Error message

convert parquet batch of {path}

What it means

Thrown in `scan_input_pks_at_positions` when `IcebergArrowConvert::chunk_from_record_batch` cannot convert a decoded Arrow `RecordBatch` into a RisingWave `DataChunk`. The parquet read succeeded, but the Arrow batch does not match what the converter expects (unsupported data type in a column, mismatched schema vs. expectation, or nullability/layout issues), so the batch cannot be projected onto the pk columns.

Source

Thrown at src/stream/src/executor/iceberg_with_pk_index/compaction_resolver.rs:434

    let metadata = input_file.metadata().await?;
    let reader = input_file.reader().await?;
    let builder = ParquetRecordBatchStreamBuilder::new(ParquetFileReader::new(metadata, reader))
        .await
        .map_err(|e| anyhow!(e).context(format!("open parquet reader for {path}")))?;
    let (projection, pk_order) = pk_projection(builder.parquet_schema(), pk_indices)?;
    let mut stream = builder
        .with_projection(projection)
        .build()
        .map_err(|e| anyhow!(e).context(format!("build parquet stream for {path}")))?;

    let mut base_pos = 0;
    let mut iter = want_positions.iter().peekable();
    while let Some(batch) = stream.next().await {
        let batch =
            batch.map_err(|e| anyhow!(e).context(format!("read parquet batch of {path}")))?;
        let chunk = IcebergArrowConvert
            .chunk_from_record_batch(&batch)
            .map_err(|e| anyhow!(e).context(format!("convert parquet batch of {path}")))?
            .project(&pk_order);
        while let Some(&iter_pos) = iter.peek() {
            let chunk_pos = iter_pos as usize - base_pos;
            if chunk_pos >= chunk.capacity() {
                break;
            }
            let pk = chunk.row_at(chunk_pos).0.to_owned_row();
            results.push(pk);
            iter.next();
        }
        base_pos += chunk.capacity();
    }

    Ok(results)
}

#[try_stream(ok = DataChunk, error = SinkError)]
async fn scan_output_file_inner<'a>(

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the conversion error to find the offending column/type; check RisingWave's supported Iceberg type mappings.
2. Rewrite the offending file with a writer that emits supported types (e.g. via Iceberg rewrite action).
3. Verify pk columns use plain supported types (int/long/string/etc.) in the table schema.
4. If a RisingWave version issue, upgrade to a version with broader IcebergArrowConvert type support.

Example fix

// before: table schema with unsupported type in a column reached during projection
-- column updated_at TIMESTAMPNSTZ (unsupported)

// after
-- column updated_at TIMESTAMPTZ or TIMESTAMP (supported mapping)

fn supports_arrow_types(schema: &arrow_schema::SchemaRef) -> bool {
    schema.fields().iter().all(|f| matches!(f.data_type(),
        arrow_schema::DataType::Boolean
        | arrow_schema::DataType::Int32 | arrow_schema::DataType::Int64
        | arrow_schema::DataType::Float32 | arrow_schema::DataType::Float64
        | arrow_schema::DataType::Utf8 | arrow_schema::DataType::Binary
        | arrow_schema::DataType::Timestamp(_, _)))
}

let chunk = match IcebergArrowConvert.chunk_from_record_batch(&batch) {
    Ok(c) => c,
    Err(e) => return Err(SinkError::Iceberg(anyhow!(e).context("unsupported arrow type in parquet batch"))),
};

When it happens

Trigger: `resolve` → `scan_input_pks_at_positions` converts each parquet batch to a chunk and projects it with `pk_order`. Triggered when the parquet file contains Arrow types the IcebergArrowConvert does not support (e.g. exotic logical types, dictionary-encoded columns the converter rejects), or the projected batch shape does not match the expected schema.

Common situations: Files written by an external Iceberg writer using data types RisingWave's converter does not yet support; schema evolution introduced a new type into pk-adjacent columns; parquet logical/converted-type written inconsistently by a different writer version.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• Failed to convert Arrow type to Iceberg type
• failed to convert {}: {} to arrow type
• pk index {pk_index} is out of range for parquet schema with
• duplicate pk index {pk_index}
• open parquet reader for {path}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/c0009413baf733f1. Report an issue: GitHub.