{"record":{"id":"0af98c6c83a01d0e","repo":"risingwavelabs/risingwave","slug":"column-not-found-in-source-backfill-state-tab","errorCode":null,"errorMessage":"`{}` column not found in source backfill state table schema","messagePattern":"`(.+?)` column not found in source backfill state table schema","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/frontend/src/optimizer/rule/table_function_to_internal_source_backfill_progress.rs","lineNumber":160,"sourceCode":"struct SourceBackfillInfo {\n    job_id: JobId,\n    fragment_id: FragmentId,\n    table_id: TableId,\n    partition_id_column_index: usize,\n    backfill_progress_column_index: usize,\n}\n\nimpl SourceBackfillInfo {\n    fn new(table: &TableCatalog) -> anyhow::Result<Self> {\n        let Some(job_id) = table.job_id else {\n            bail!(\"`job_id` column not found in source backfill table catalog\");\n        };\n        let Some(backfill_progress_column_index) = table\n            .columns\n            .iter()\n            .position(|c| c.name() == StreamSourceScan::BACKFILL_PROGRESS_COLUMN_NAME)\n        else {\n            bail!(\n                \"`{}` column not found in source backfill state table schema\",\n                StreamSourceScan::BACKFILL_PROGRESS_COLUMN_NAME\n            );\n        };\n        let Some(partition_id_column_index) = table\n            .columns\n            .iter()\n            .position(|c| c.name() == StreamSourceScan::PARTITION_ID_COLUMN_NAME)\n        else {\n            bail!(\n                \"`{}` column not found in source backfill state table schema\",\n                StreamSourceScan::PARTITION_ID_COLUMN_NAME\n            );\n        };\n        let fragment_id = table.fragment_id;\n        let table_id = table.id;\n\n        Ok(Self {","sourceCodeStart":142,"sourceCodeEnd":178,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/frontend/src/optimizer/rule/table_function_to_internal_source_backfill_progress.rs#L142-L178","documentation":"When building the plan for `internal_source_backfill_progress()`, each source backfill state table must contain a column named BACKFILL_PROGRESS_COLUMN_NAME so the rule can project its index. If the column lookup by name fails, the table schema does not match the expected internal source-backfill table shape. This is an internal schema invariant of the auto-generated state table.","triggerScenarios":"Running `SELECT * FROM internal_source_backfill_progress()` while a source-backfill state table in the catalog lacks the backfill progress column — e.g. the table name matched `is_source_backfill_table` but was created by an older/other code path without the progress column.","commonSituations":"Cluster upgraded from a RisingWave version whose source-backfill state tables had a different schema while old internal tables persist; manual edits or migrations of internal tables; a misclassified table passing the name-based `is_source_backfill_table` filter.","solutions":["Upgrade/complete the in-flight source backfill jobs so their state tables are recreated with the current schema; check `rw_databases`/internal table schema via `SHOW COLUMNS FROM` on the internal table id.","Drop and recreate the source/backfilling materialized view or table whose internal state table has the stale schema.","If a version-skew left orphaned tables, clean them up (drop leftover jobs) and re-run the progress query.","For developers: harden the filter in `get_source_backfilling_tables` to also verify the expected columns before building SourceBackfillInfo."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"-- verify internal state table schema before querying progress\nSHOW COLUMNS FROM <internal_backfill_table_id>;","typeGuard":"// Rust: check schema shape up front\nfn has_required_columns(t: &TableCatalog) -> bool {\n    t.columns.iter().any(|c| c.name() == StreamSourceScan::BACKFILL_PROGRESS_COLUMN_NAME)\n}","tryCatchPattern":"// catch and report which table failed\nif let Err(e) = SourceBackfillInfo::new(&table) { return Err(e.context(format!(\"table {}: {}\", table.id, e))) }","preventionTips":["Complete version upgrades before inspecting backfill progress","Drop orphaned internal tables from interrupted DDL","Verify expected columns instead of relying on name-based table classification"],"tags":["frontend","optimizer","schema","backfill"],"backgroundTag":"schema-validation-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}