risingwavelabs/risingwave · error

Delete field id not found in schema

Error message

Delete field id {} not found in schema

What it means

After collecting equality delete files, the planner resolves each equality delete field id to a column name using the table schema (`schema.name_by_field_id`). If a delete field id has no corresponding field in the current schema, the id cannot be projected and the function bails with this error naming the missing id.

Solutions

  1. Restore/re-add the missing field to the Iceberg schema so the field id resolves again.
  2. Compact/rewrite delete files so stale deletes referencing dropped fields are removed.
  3. Avoid dropping columns that are used as equality-delete keys, or first rewrite the affected files.
  4. Verify the source loads the correct table/schema (right table identifier and catalog).
Defensive patterns

Strategy: validation

Validate before calling

let missing: Vec<i64> = equality_ids.iter()
    .filter(|id| schema.name_by_field_id(**id).is_none())
    .copied().collect();
if !missing.is_empty() { /* heal schema or rewrite deletes before scanning */ }

Type guard

fn ids_resolvable(schema: &Schema, ids: &[i64]) -> bool {
    ids.iter().all(|id| schema.name_by_field_id(*id).is_some())
}

Try / catch

match list_scan_tasks(...) {
    Err(e) if e.to_string().contains("not found in schema") => {
        // re-add the field or compact stale delete files
    }
    r => r,
}

Prevention

When it happens

Trigger: `list_scan_tasks` resolving `equality_delete_ids` where one of the ids was removed from the Iceberg schema (dropped column) or comes from a delete file written against an incompatible schema version.

Common situations: Iceberg schema evolution that dropped a column previously used in equality deletes; delete files from an old schema applied to the current schema; mismatch between table schema loaded and the schema the deletes were written with.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/6bef837a136085bc. Report an issue: GitHub.

Appendix: source

Thrown at src/connector/src/source/iceberg/mod.rs:422

                    iceberg::spec::DataContentType::PositionDeletes => {
                        if position_delete_files_set.insert(delete_file.file_path.clone()) {
                            position_delete_files.push(delete_file.to_file_scan_task(&task));
                        }
                    }
                }
            }

            // Top-level scan tasks always represent data files. Keep their delete
            // descriptors intact so the SDK reader can apply them when requested.
            data_files.push(task);
        }
        let schema = table_schema.clone();
        let equality_delete_columns = equality_delete_ids
            .unwrap_or_default()
            .into_iter()
            .map(|id| match schema.name_by_field_id(id) {
                Some(name) => Ok::<std::string::String, ConnectorError>(name.to_owned()),
                None => bail!("Delete field id {} not found in schema", id),
            })
            .collect::<ConnectorResult<Vec<_>>>()?;

        Ok(IcebergListResult {
            data_files,
            equality_delete_files,
            position_delete_files,
            equality_delete_columns,
            format_version,
            schema,
        })
    }

    /// Uniformly distribute scan tasks to compute nodes.
    /// It's deterministic so that it can best utilize the data locality.
    ///
    /// # Arguments
    /// * `file_scan_tasks`: The file scan tasks to be split.

View on GitHub (pinned to 6469eb736d)