risingwavelabs/risingwave · error

output column must exist in the source catalog

Error message

output column must exist in the source catalog

What it means

While mapping the node's output columns back to the original Iceberg source catalog, `iceberg_side_fields` expects every output column name to exist in the source catalog. A missing name means the intermediate scan produces a column the source doesn't have — an internal consistency violation, panicked via `.expect` during predicate pushdown.

Source

Thrown at src/frontend/src/optimizer/plan_node/logical_iceberg_intermediate_scan.rs:181

    /// Fields carrying the iceberg-side column types (before the engine-table Hummock
    /// type remapping), for predicate pushdown. Derived from the source catalog, which
    /// the remapping never touches.
    fn iceberg_side_fields(&self) -> Vec<Field> {
        let catalog = self
            .core
            .catalog
            .as_ref()
            .expect("iceberg intermediate scan must have a source catalog");
        let by_name: HashMap<&str, &ColumnCatalog> =
            catalog.columns.iter().map(|c| (c.name(), c)).collect();
        self.core
            .column_catalog
            .iter()
            .map(|col| {
                let source_col = by_name
                    .get(col.name())
                    .expect("output column must exist in the source catalog");
                Field::from(&source_col.column_desc)
            })
            .collect()
    }

    pub fn output_columns(&self) -> impl ExactSizeIterator<Item = &str> {
        self.core.column_catalog.iter().map(|c| c.name.as_str())
    }

    pub fn add_predicate(
        &self,
        iceberg_predicate: Predicate,
        extracted_condition: Condition,
    ) -> Self {
        LogicalIcebergIntermediateScan {
            iceberg_predicate: self.iceberg_predicate.clone().and(iceberg_predicate),
            hummock_rewrite: self.hummock_rewrite.add_predicate(extracted_condition),
            ..self.clone()

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify that every output column of the intermediate scan is derived from the source catalog and names are kept in sync through rewrites.
2. Return a Result with a descriptive error instead of `.expect` to aid diagnosis.
3. Capture the failing plan (EXPLAIN) and minimal query, and file a bug against the Iceberg intermediate scan rewrite passes.

Example fix

// before
let source_col = by_name.get(col.name())
    .expect("output column must exist in the source catalog");

// after
let source_col = by_name.get(col.name()).ok_or_else(|| {
    anyhow!("output column {:?} not found in iceberg source catalog", col.name())
})?;

// before pushdown, verify column alignment
for col in &scan.core.column_catalog {
    if !scan.core.catalog.as_ref().unwrap().columns.iter().any(|c| c.name() == col.name()) {
        return Err(format!("column {} missing in source catalog", col.name()));
    }
}

When it happens

Trigger: `predicate_pushdown` on a LogicalIcebergIntermediateScan where a column in `core.column_catalog` has no matching name in the source catalog's columns (e.g. after a rewrite added/renamed columns without updating the scan node).

Common situations: Optimizer rewrite bugs around Iceberg intermediate scans (column pruning/renaming passes) or mismatched node construction in tests.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• iceberg intermediate scan must have a source catalog
• required column should be kept
• table {} not found
• register_table is not supported in mock catalog
• iceberg sink metadata should have schema_id

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/ae9c42723f4e140a. Report an issue: GitHub.