risingwavelabs/risingwave · error

new column id not found in scan output

Error message

new column id not found in scan output

What it means

`rewrite_project_node` appends InputRefs for each newly added column by looking up its column id in `scan_rewrite.new_output_index_by_column_id` built from the rewritten StreamScan output. If a newly added column's id is absent from the scan's new output, the mapping is inconsistent and the rewrite fails, preventing a Project that references a column the scan does not produce.

Source

Thrown at src/meta/src/stream/stream_graph/fragment.rs:729

                continue;
            };
            new_expr.rex_node = Some(expr_node::RexNode::InputRef(new_index));
        } else if !removed_column_ids.is_empty() {
            return Err(anyhow!(
                "auto schema change with drop column only supports Project with InputRef"
            )
            .into());
        }
        new_select_list.push(new_expr);
        new_project_fields.push(project_node.fields[index].clone());
    }

    for col in newly_added_columns {
        let Some(&new_index) = scan_rewrite
            .new_output_index_by_column_id
            .get(&col.column_id())
        else {
            return Err(anyhow!("new column id not found in scan output").into());
        };
        new_select_list.push(PbExprNode {
            function_type: expr_node::Type::Unspecified as i32,
            return_type: Some(col.data_type().to_protobuf()),
            rex_node: Some(expr_node::RexNode::InputRef(new_index)),
        });
        new_project_fields.push(
            Field::new(
                format!("{}.{}", upstream_table_name, col.column_desc.name),
                col.data_type().clone(),
            )
            .to_prost(),
        );
    }

    project_node_body.select_list = new_select_list;
    project_node.fields = new_project_fields;
    Ok(())

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the new column id is part of the upstream table's column ids passed to rewrite_stream_scan_and_merge.
2. Check that newly_added_columns passed to rewrite_refresh_schema_sink_fragment match what the scan rewrite was told to add.
3. File/inspect a RisingWave bug; this indicates the scan rewrite and project rewrite were given divergent column sets.

// Rust
match rewrite_refresh_schema_sink_fragment(...) {
    Err(e) if e.to_string().contains("new column id not found in scan output") => {
        // fall back to drop-and-recreate of the downstream MV
    }
    other => other?,
}

When it happens

Trigger: ALTER TABLE ADD COLUMN rewrite where `rewrite_stream_scan_and_merge` did not include the new column's id in the scan output (e.g. the added column was filtered or omitted during scan rewrite), while the Project extension loop in `rewrite_refresh_schema_sink_fragment` tries to append it.

Common situations: Internal inconsistency during schema-change graph rewriting; typically a bug or a scan rewrite that dropped the new column (e.g. column not in upstream_column_ids), hit when adding columns to tables with downstream MVs containing Projects.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• auto schema change with drop column only supports Project wi
• Different number of internal tables. New: {}, Old: {}
• must be either all snapshot_backfill or no snapshot_backfill
• Project node must have exactly 1 input for auto schema chang
• expect PbNodeBody::StreamScan or PbNodeBody::Project but got

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/c496a5bf67904501. Report an issue: GitHub.