risingwavelabs/risingwave · error

Delete field id {} not found in schema

Error message

Delete field id {} not found in schema

What it means

After collecting equality delete files, the planner resolves each equality delete field id to a column name using the table schema (`schema.name_by_field_id`). If a delete field id has no corresponding field in the current schema, the id cannot be projected and the function bails with this error naming the missing id.

Source

Thrown at src/connector/src/source/iceberg/mod.rs:422

                    iceberg::spec::DataContentType::PositionDeletes => {
                        if position_delete_files_set.insert(delete_file.file_path.clone()) {
                            position_delete_files.push(delete_file.to_file_scan_task(&task));
                        }
                    }
                }
            }

            // Top-level scan tasks always represent data files. Keep their delete
            // descriptors intact so the SDK reader can apply them when requested.
            data_files.push(task);
        }
        let schema = table_schema.clone();
        let equality_delete_columns = equality_delete_ids
            .unwrap_or_default()
            .into_iter()
            .map(|id| match schema.name_by_field_id(id) {
                Some(name) => Ok::<std::string::String, ConnectorError>(name.to_owned()),
                None => bail!("Delete field id {} not found in schema", id),
            })
            .collect::<ConnectorResult<Vec<_>>>()?;

        Ok(IcebergListResult {
            data_files,
            equality_delete_files,
            position_delete_files,
            equality_delete_columns,
            format_version,
            schema,
        })
    }

    /// Uniformly distribute scan tasks to compute nodes.
    /// It's deterministic so that it can best utilize the data locality.
    ///
    /// # Arguments
    /// * `file_scan_tasks`: The file scan tasks to be split.

View on GitHub (pinned to 6469eb736d)

Solutions

1. Restore/re-add the missing field to the Iceberg schema so the field id resolves again.
2. Compact/rewrite delete files so stale deletes referencing dropped fields are removed.
3. Avoid dropping columns that are used as equality-delete keys, or first rewrite the affected files.
4. Verify the source loads the correct table/schema (right table identifier and catalog).

let missing: Vec<i64> = equality_ids.iter()
    .filter(|id| schema.name_by_field_id(**id).is_none())
    .copied().collect();
if !missing.is_empty() { /* heal schema or rewrite deletes before scanning */ }

fn ids_resolvable(schema: &Schema, ids: &[i64]) -> bool {
    ids.iter().all(|id| schema.name_by_field_id(*id).is_some())
}

match list_scan_tasks(...) {
    Err(e) if e.to_string().contains("not found in schema") => {
        // re-add the field or compact stale delete files
    }
    r => r,
}

When it happens

Trigger: `list_scan_tasks` resolving `equality_delete_ids` where one of the ids was removed from the Iceberg schema (dropped column) or comes from a delete file written against an incompatible schema version.

Common situations: Iceberg schema evolution that dropped a column previously used in equality deletes; delete files from an old schema applied to the current schema; mismatch between table schema loaded and the schema the deletes were written with.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• schema_id and partition_spec_id should be the same in all wr
• iceberg sink: schema evolution not supported; expect schema
• table {} not found
• register_table is not supported in mock catalog
• iceberg sink metadata should have schema_id

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/6bef837a136085bc. Report an issue: GitHub.