risingwavelabs/risingwave · error · ArrayError

support negative scale for arrow decimal

Error message

support negative scale for arrow decimal

What it means

Converting an Arrow Decimal128 array into RisingWave's DecimalArray is only implemented for non-negative scale. Arrow (and Iceberg) can represent decimals with negative scale (value scaled by 10^|scale|), but RW decimal conversion does not support that representation, so the conversion bails.

Source

Thrown at src/common/src/array/arrow/arrow_impl.rs:1583

impl From<&DecimalArray> for arrow_array::StringArray {
    fn from(array: &DecimalArray) -> Self {
        let mut builder =
            arrow_array::builder::StringBuilder::with_capacity(array.len(), array.len() * 8);
        for value in array.iter() {
            builder.append_option(value.map(|d| d.to_string()));
        }
        builder.finish()
    }
}

// This arrow decimal type is used by iceberg source to read iceberg decimal into RW decimal.
impl TryFrom<&arrow_array::Decimal128Array> for DecimalArray {
    type Error = ArrayError;

    fn try_from(array: &arrow_array::Decimal128Array) -> Result<Self, Self::Error> {
        if array.scale() < 0 {
            bail!("support negative scale for arrow decimal")
        }

        // Calculate the max value based on the Arrow decimal's precision
        // When writing Inf to Arrow Decimal128(precision, scale), we use 10^precision - 1
        let precision = array.precision();
        let max_value = 10_i128.pow(precision as u32) - 1;

        let from_arrow = |value| {
            const NAN: i128 = i128::MIN + 1;
            let res = match value {
                // Check for special values using Arrow Decimal's max value, not i128::MAX
                NAN => Decimal::NaN,
                v if v == max_value => Decimal::PositiveInf,
                v if v == -max_value => Decimal::NegativeInf,
                i128::MAX => Decimal::PositiveInf, // Fallback for old data
                i128::MIN => Decimal::NegativeInf, // Fallback for old data
                _ => Decimal::truncated_i128_and_scale(value, array.scale() as u32)
                    .ok_or_else(|| ArrayError::from_arrow("decimal overflow"))?,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Avoid negative-scale decimals in the source table schema (use scale >= 0)
2. Rescale the column upstream (e.g. cast to scale 0 or higher before writing/reading)
3. Pre-process the Arrow array with a cast to a non-negative-scale decimal type before conversion

Example fix

// before
let decimal = DecimalArray::try_from(arrow_decimal_array)?;
// after
let casted = arrow_decimal_array.reinterpret_cast(arrow::datatypes::DataType::Decimal128(20, 0));
let decimal = DecimalArray::try_from(&casted)?;

fn check_arrow_decimal_ok(dt: &arrow::datatypes::DataType) -> Result<(), String> {
    if let arrow::datatypes::DataType::Decimal128(p, s) = dt {
        if *s < 0 { return Err(format!("negative scale {} unsupported", s)); }
    }
    Ok(())
}

fn is_supported_decimal(a: &arrow_array::Decimal128Array) -> bool { a.scale() >= 0 }

match DecimalArray::try_from(&arrow_arr) {
    Err(e) if e.to_string().contains("negative scale") => {
        let casted = arrow_arr.reinterpret_cast(arrow::datatypes::DataType::Decimal128(20, 0));
        DecimalArray::try_from(&casted)
    }
    other => other,
}

When it happens

Trigger: TryFrom<&arrow_array::Decimal128Array> for DecimalArray is invoked (primarily by the Iceberg source reading decimal columns) when array.scale() < 0, e.g. Arrow type Decimal128(5, -2).

Common situations: Iceberg table column defined with a negative decimal scale; a writer exported data with scaled-truncated decimals; upstream schema evolution introduced negative-scale decimals.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Failed to convert Arrow type to Iceberg type
• failed to convert {}: {} to arrow type
• failed to convert arrow schema to iceberg schema
• Snowflake catalog only supports iceberg sources
• RecordBatch::try_new failed: {e}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/044ee54c87a3b7bf. Report an issue: GitHub.