risingwavelabs/risingwave · error

`job_id` column not found in source backfill table catalog

Error message

`job_id` column not found in source backfill table catalog

What it means

SourceBackfillInfo::new is constructing metadata for an internal source backfill state table while rewriting `internal_source_backfill_progress()` into a scan plan. It expects `table.job_id` (Option<JobId>) to be Some; if it is None the table catalog entry lacks the job id association. This is an internal invariant violation: any table returned by `iter_backfilling_internal_tables()` filtered by `is_source_backfill_table` should carry a job_id.

Source

Thrown at src/frontend/src/optimizer/rule/table_function_to_internal_source_backfill_progress.rs:153

impl TableFunctionToInternalSourceBackfillProgressRule {
    pub fn create() -> BoxedRule {
        Box::new(TableFunctionToInternalSourceBackfillProgressRule {})
    }
}

struct SourceBackfillInfo {
    job_id: JobId,
    fragment_id: FragmentId,
    table_id: TableId,
    partition_id_column_index: usize,
    backfill_progress_column_index: usize,
}

impl SourceBackfillInfo {
    fn new(table: &TableCatalog) -> anyhow::Result<Self> {
        let Some(job_id) = table.job_id else {
            bail!("`job_id` column not found in source backfill table catalog");
        };
        let Some(backfill_progress_column_index) = table
            .columns
            .iter()
            .position(|c| c.name() == StreamSourceScan::BACKFILL_PROGRESS_COLUMN_NAME)
        else {
            bail!(
                "`{}` column not found in source backfill state table schema",
                StreamSourceScan::BACKFILL_PROGRESS_COLUMN_NAME
            );
        };
        let Some(partition_id_column_index) = table
            .columns
            .iter()
            .position(|c| c.name() == StreamSourceScan::PARTITION_ID_COLUMN_NAME)
        else {
            bail!(
                "`{}` column not found in source backfill state table schema",

View on GitHub (pinned to 6469eb736d)

Solutions

1. Retry the `internal_source_backfill_progress()` query after the ongoing DDL (CREATE MATERIALIZED VIEW / CREATE TABLE with backfill) finishes, so the table catalog is fully published.
2. Check catalog consistency: verify the offending backfill table has an associated streaming job via internal system catalogs; if a job was dropped mid-creation, drop the leftover objects and recreate.
3. If reproducible on a healthy cluster, report as a frontend/meta bug — the filter `is_source_backfill_table` should exclude tables without job_id; fix `get_source_backfilling_tables` to skip None job_id entries.

Example fix

// before
let Some(job_id) = table.job_id else {
    bail!("`job_id` column not found in source backfill table catalog");
};
// after (skip such tables defensively in the caller)
let backfill_info = match SourceBackfillInfo::new(&table) {
    Ok(info) => info,
    Err(_) => continue, // skip stale/partially published backfill tables
};

-- ensure no DDL backfill is mid-flight before querying progress
SELECT * FROM rw_catalog.rw_table_name WHERE name LIKE '%source_backfill%';
-- only query internal_source_backfill_progress() after CREATE ... finishes

// Rust: narrow before use
fn has_job_id(t: &TableCatalog) -> bool { t.job_id.is_some() }

// match on Result and skip rather than fail the whole query
match SourceBackfillInfo::new(&table) { Ok(i) => push(i), Err(_) => continue }

When it happens

Trigger: Calling `SELECT * FROM internal_source_backfill_progress()` (or the optimizer rule applying to it) while the catalog contains a source-backfill internal table whose TableCatalog.job_id is None, e.g. a partially-published or stale catalog entry.

Common situations: Racing a catalog read against a streaming job creation/removal so a backfill table is observed before its job id is linked; metadata corruption or a version skew between meta node and frontend where the job_id field was not persisted.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• `job_id` column not found in backfill table
• `{}` column not found in source backfill state table schema
• {object_type} not found: {name}
• {object_type} named {name} already exists{}
• `{}` column not found in backfill table

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/b3fd4e21ad74e09b. Report an issue: GitHub.