risingwavelabs/risingwave · error · SinkError::Iceberg

error from iceberg writer builder

Error message

error from iceberg writer builder: {err}

What it means

The iceberg writer_builder.build() call failed while constructing the append-only DataFileWriter from the ParquetWriterBuilder/RollingFileWriterBuilder stack. The iceberg-rs Error is wrapped as SinkError::Iceberg. This indicates the writer configuration (schema, partition spec, writer properties) was rejected by the iceberg library at build time.

Solutions

1. Read the wrapped iceberg library error (it carries the root cause) and address the specific build failure it reports
2. Verify the parquet writer properties (compression, page size) are values the pinned iceberg-rs version supports
3. Confirm the schema and partition_spec passed to the DataFileWriterBuilder match table.metadata().current_schema() and its default spec
4. Pin/upgrade iceberg-rs to a version compatible with the RisingWave connector code

Example fix

// before
writer_builder
    .build()
    .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,
// after
writer_builder.build().map_err(|err| {
    SinkError::Iceberg(anyhow::anyhow!(
        "error from iceberg writer builder: {err} (schema: {:?}, partition: {:?})",
        schema,
        partition_spec
    ))
})?

// Pre-check schema/spec consistency before building
anyhow::ensure!(schema.fields().len() > 0, "empty schema passed to writer builder");
anyhow::ensure!(partition_spec.fields().len() == table.metadata().default_partition_spec().map(|s| s.fields().len()).unwrap_or(0), "partition spec mismatch");

writer_builder.build().map_err(|err| {
    SinkError::Iceberg(anyhow!("error from iceberg writer builder: {err}"))
})?

When it happens

Trigger: Calling build_append_only when the Arrow/Iceberg schema pair, partition spec clone, or parquet writer properties are inconsistent — e.g. a schema the ParquetWriterBuilder cannot support, or an invalid partition spec combination passed to the DataFileWriterBuilder.

Common situations: Sinking columns whose types cannot be encoded as parquet by the configured writer properties; mismatched schema/partition-spec after table evolution; misconfigured parquet writer properties (e.g. invalid compression or dictionary settings).

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• error from iceberg library
• bounded compaction branch
• bounded compaction head sequence
• bounded compaction is not supported for copy-on-write tasks
• bounded compaction requires Iceberg format V2 or V3

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/66f87e79ac8ed5e3. Report an issue: GitHub.

Appendix: source

Thrown at src/connector/src/sink/iceberg/writer.rs:419

            ),
        );
        let data_file_builder = DataFileWriterBuilder::new(rolling_builder);
        let monitored_builder = MonitoredGeneralWriterBuilder::new(
            data_file_builder,
            write_qps.clone(),
            write_latency.clone(),
        );
        let writer_builder = TaskWriterBuilderWrapper::new(
            monitored_builder,
            fanout_enabled,
            schema.clone(),
            partition_spec.clone(),
            true,
        );
        let inner_writer = Some(Box::new(
            writer_builder
                .build()
                .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,
        ) as Box<dyn IcebergWriter>);
        Ok(Self {
            arrow_schema: Arc::new(
                schema_to_arrow_schema(table.metadata().current_schema())
                    .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,
            ),
            metrics: IcebergWriterMetrics {
                _write_qps: write_qps,
                _write_latency: write_latency,
                write_bytes,
            },
            writer: IcebergWriterDispatch::Append {
                writer: inner_writer,
                writer_builder,
            },
            table,
            actor_id: actor_id.to_string(),
            sink_id: sink_id.to_string(),

View on GitHub (pinned to 6469eb736d)