risingwavelabs/risingwave · error · anyhow::Error

RecordBatch::try_new failed: {e}

Error message

RecordBatch::try_new failed: {e}

What it means

Arrow's `RecordBatch::try_new` validates that every provided column array matches the given schema (field count, types, nullability, length equality). When the path StringArray or pos Int64Array disagrees with the arrow schema's declared fields, try_new returns an ArrowError which this code rethrows as an anyhow error prefixed with 'RecordBatch::try_new failed'.

Source

Thrown at src/connector/src/sink/iceberg/position_delete.rs:294

        .build()
        .context("Failed to build position-delete file metadata")
}

/// Writes one chunk of `positions` as a `(file_path, pos)` batch into `writer`. Every row shares
/// `data_file_path` because the delete file is file-scoped.
async fn write_position_delete_chunk(
    writer: &mut ParquetWriter,
    arrow_schema: &ArrowSchemaRef,
    data_file_path: &str,
    positions: Vec<i64>,
) -> Result<()> {
    let path_column: ArrayRef = Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
        data_file_path,
        positions.len(),
    )));
    let pos_column: ArrayRef = Arc::new(Int64Array::from(positions));
    let batch = RecordBatch::try_new(arrow_schema.clone(), vec![path_column, pos_column])
        .map_err(|e| anyhow!(e))?;
    writer.write(&batch).await?;
    Ok(())
}

View on GitHub (pinned to 6469eb736d)

Solutions

1. Log/print the arrow_schema fields and the array data types at failure; confirm the schema is exactly [Utf8, Int64] in that order
2. Ensure the schema used to build the writer (from PositionDeleteWriterBuilder) is the same Arc<arrow_schema> passed here rather than a locally constructed one
3. Pin/upgrade iceberg-rs and RisingWave together so both agree on the position-delete record format
4. Build the batch with RecordBatch::try_new_with_options only after asserting arrays.len() == schema.fields().len() and matching data_type() per column

Example fix

// before
let batch = RecordBatch::try_new(arrow_schema.clone(), vec![path_column, pos_column])
    .map_err(|e| anyhow!(e))?;
// after
let batch = RecordBatch::try_new(
    arrow_schema.clone(),
    vec![Arc::clone(&path_column), Arc::clone(&pos_column)],
)
.inspect_err(|e| tracing::error!(?e, schema = ?arrow_schema, "position-delete batch build failed"))
.map_err(|e| anyhow!(e))?;

assert_eq!(arrow_schema.fields().len(), 2);
assert_eq!(arrow_schema.field(0).data_type(), &arrow::datatypes::DataType::Utf8);
assert_eq!(arrow_schema.field(1).data_type(), &arrow::datatypes::DataType::Int64);

fn batch_matches(schema: &arrow::datatypes::SchemaRef, cols: &[ArrayRef]) -> bool {
    schema.fields().len() == cols.len()
        && schema.fields().iter().zip(cols).all(|(f, a)| f.data_type() == a.data_type())
        && cols.iter().all(|a| a.len() == cols[0].len())
}

let batch = RecordBatch::try_new(arrow_schema.clone(), vec![path_column, pos_column])
    .map_err(|e| anyhow!("RecordBatch::try_new failed: {e}; schema = {:?}", arrow_schema))?;

When it happens

Trigger: Calling write_position_delete_chunk with an arrow_schema whose fields are not exactly [String (file_path), Int64 (pos)] — e.g. after an iceberg-rs upgrade changed PositionDeleteWriterBuilder's expected schema — or passing an empty positions vec to a schema marked non-nullable.

Common situations: Version mismatch between the RisingWave connector and the iceberg-rs crate where the position-delete schema ordering/types changed; a refactor accidentally swapping the two columns; schema generated from a different source than the arrays.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• failed to convert {}: {} to arrow type
• failed to convert arrow schema to iceberg schema
• error converting Iceberg schema to Arrow schema: {err}
• support negative scale for arrow decimal
• Invalid struct encoding: {0}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/8e494b8560c921dc. Report an issue: GitHub.