risingwavelabs/risingwave · error

`job_id` column not found in backfill table

Error message

`job_id` column not found in backfill table

What it means

`BackfillInfo::new` extracts metadata from an internal backfill table catalog. If the table catalog lacks a `job_id`, the constructor bails because backfill progress planning cannot identify the owning background job.

Source

Thrown at src/frontend/src/optimizer/rule/table_function_to_internal_backfill_progress.rs:186

impl TableFunctionToInternalBackfillProgressRule {
    pub fn create() -> BoxedRule {
        Box::new(TableFunctionToInternalBackfillProgressRule {})
    }
}

struct BackfillInfo {
    job_id: JobId,
    fragment_id: FragmentId,
    table_id: TableId,
    row_count_column_index: usize,
    epoch_column_index: Option<usize>,
}

impl BackfillInfo {
    fn new(table: &TableCatalog) -> anyhow::Result<Self> {
        let Some(job_id) = table.job_id else {
            bail!("`job_id` column not found in backfill table");
        };
        let Some(row_count_column_index) = table
            .columns
            .iter()
            .position(|c| c.name() == StreamTableScan::ROW_COUNT_COLUMN_NAME)
        else {
            bail!(
                "`{}` column not found in backfill table",
                StreamTableScan::ROW_COUNT_COLUMN_NAME
            );
        };
        let epoch_column_index = table
            .columns
            .iter()
            .position(|c| c.name() == StreamTableScan::EPOCH_COLUMN_NAME);
        let fragment_id = table.fragment_id;
        let table_id = table.id;

View on GitHub (pinned to 6469eb736d)

Solutions

1. Upgrade the cluster/tables so internal backfill tables carry job_id metadata.
2. Verify you are targeting a real backfill table (created during an ongoing backfill) rather than an unrelated internal table.
3. Recreate the table/MV so a fresh backfill job registers job_id.
4. Check the catalog via internal system tables to confirm job metadata exists.

// Catch catalog errors when reading internal backfill tables
match BackfillInfo::new(&table) {
    Ok(info) => /* use info */,
    Err(e) => eprintln!("backfill table lacks job metadata: {e:#}"),
}

When it happens

Trigger: The rule `table_function_to_internal_backfill_progress` converts an internal table-function scan and finds the resolved internal backfill TableCatalog has `job_id == None`.

Common situations: Querying internal backfill progress tables/functions against tables created by older RisingWave versions before job_id was recorded, or catalogs whose internal metadata was not populated.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• `{}` column not found in backfill table
• `job_id` column not found in source backfill table catalog
• distribution key {:?} must be a subset of primary key {:?}
• {e}
• failed to create iceberg table

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/cb6c7b3a71d4fdfe. Report an issue: GitHub.