{"record":{"id":"c496a5bf67904501","repo":"risingwavelabs/risingwave","slug":"new-column-id-not-found-in-scan-output","errorCode":null,"errorMessage":"new column id not found in scan output","messagePattern":"new column id not found in scan output","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/stream/stream_graph/fragment.rs","lineNumber":729,"sourceCode":"                continue;\n            };\n            new_expr.rex_node = Some(expr_node::RexNode::InputRef(new_index));\n        } else if !removed_column_ids.is_empty() {\n            return Err(anyhow!(\n                \"auto schema change with drop column only supports Project with InputRef\"\n            )\n            .into());\n        }\n        new_select_list.push(new_expr);\n        new_project_fields.push(project_node.fields[index].clone());\n    }\n\n    for col in newly_added_columns {\n        let Some(&new_index) = scan_rewrite\n            .new_output_index_by_column_id\n            .get(&col.column_id())\n        else {\n            return Err(anyhow!(\"new column id not found in scan output\").into());\n        };\n        new_select_list.push(PbExprNode {\n            function_type: expr_node::Type::Unspecified as i32,\n            return_type: Some(col.data_type().to_protobuf()),\n            rex_node: Some(expr_node::RexNode::InputRef(new_index)),\n        });\n        new_project_fields.push(\n            Field::new(\n                format!(\"{}.{}\", upstream_table_name, col.column_desc.name),\n                col.data_type().clone(),\n            )\n            .to_prost(),\n        );\n    }\n\n    project_node_body.select_list = new_select_list;\n    project_node.fields = new_project_fields;\n    Ok(())","sourceCodeStart":711,"sourceCodeEnd":747,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/stream/stream_graph/fragment.rs#L711-L747","documentation":"`rewrite_project_node` appends InputRefs for each newly added column by looking up its column id in `scan_rewrite.new_output_index_by_column_id` built from the rewritten StreamScan output. If a newly added column's id is absent from the scan's new output, the mapping is inconsistent and the rewrite fails, preventing a Project that references a column the scan does not produce.","triggerScenarios":"ALTER TABLE ADD COLUMN rewrite where `rewrite_stream_scan_and_merge` did not include the new column's id in the scan output (e.g. the added column was filtered or omitted during scan rewrite), while the Project extension loop in `rewrite_refresh_schema_sink_fragment` tries to append it.","commonSituations":"Internal inconsistency during schema-change graph rewriting; typically a bug or a scan rewrite that dropped the new column (e.g. column not in upstream_column_ids), hit when adding columns to tables with downstream MVs containing Projects.","solutions":["Verify the new column id is part of the upstream table's column ids passed to rewrite_stream_scan_and_merge.","Check that newly_added_columns passed to rewrite_refresh_schema_sink_fragment match what the scan rewrite was told to add.","File/inspect a RisingWave bug; this indicates the scan rewrite and project rewrite were given divergent column sets."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Rust\nmatch rewrite_refresh_schema_sink_fragment(...) {\n    Err(e) if e.to_string().contains(\"new column id not found in scan output\") => {\n        // fall back to drop-and-recreate of the downstream MV\n    }\n    other => other?,\n}","preventionTips":["Keep newly_added_columns consistent with what the scan rewrite receives.","Treat this as a bug signal; capture the fragment and column ids when reporting."],"tags":["meta-service","schema-change","internal-inconsistency","streaming-graph"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}