risingwavelabs/risingwave · error · SinkError::Iceberg

Can't find schema by id {}

Error message

Can't find schema by id {}

What it means

During an Iceberg sink commit, RisingWave looks up the table schema by the schema_id recorded in the data files' write results. The reloaded table metadata does not contain a schema with that id, meaning the writer and the current table metadata have diverged. This is thrown from commit_data_impl in commit.rs when schema_by_id(expect_schema_id) returns None.

Source

Thrown at src/connector/src/sink/iceberg/commit.rs:692

            schema_id = expect_schema_id,
            partition_spec_id = expect_partition_spec_id,
            data_file_count,
            retry_num = self.commit_retry_num,
            "iceberg_sink_commit_applying",
        );

        // Load the latest table to avoid concurrent modification with the best effort.
        self.table = commit_retry::reload_table(
            self.catalog.as_ref(),
            self.table.identifier(),
            expect_schema_id,
            expect_partition_spec_id,
        )
        .await
        .map_err(SinkError::Iceberg)?;

        let Some(schema) = self.table.metadata().schema_by_id(expect_schema_id) else {
            return Err(SinkError::Iceberg(anyhow!(
                "Can't find schema by id {}",
                expect_schema_id
            )));
        };
        let partition_type = resolve_partition_type(&self.table, expect_partition_spec_id, schema)?;

        let data_files = write_results
            .into_iter()
            .flat_map(|r| {
                r.data_files.into_iter().map(|f| {
                    f.try_into(expect_partition_spec_id, &partition_type, schema)
                        .map_err(|err| SinkError::Iceberg(anyhow!(err)))
                })
            })
            .collect::<Result<Vec<DataFile>>>()?;

        // # TODO:
        // This retry behavior should be revert and do in iceberg-rust when it supports retry(Track in: https://github.com/apache/iceberg-rust/issues/964)

View on GitHub (pinned to 6469eb736d)

Solutions

1. Reload the table and verify the schema id with `SELECT ... metadata.current_schema_id` or inspect the catalog; ensure the sink's expected schema id still exists.
2. If the table was recreated or replaced, recreate the RisingWave sink so it re-reads the current table metadata.
3. Check for concurrent external writers performing incompatible schema operations; pause them or enable schema evolution handling.
4. Upgrade RisingWave / iceberg-rust if a known metadata-versioning bug is involved.

Example fix

// before
let Some(schema) = self.table.metadata().schema_by_id(expect_schema_id) else {
    return Err(SinkError::Iceberg(anyhow!("Can't find schema by id {}", expect_schema_id)));
};
// after: reload once more before failing, then surface a clearer message
let table = commit_retry::reload_table(self.catalog.as_ref(), self.table.identifier(), expect_schema_id, expect_partition_spec_id).await?;
let Some(schema) = table.metadata().schema_by_id(expect_schema_id) else {
    return Err(SinkError::Iceberg(anyhow!("schema id {} not found in table {} metadata; table may have been replaced", expect_schema_id, self.table.identifier())));
};

// Before relying on the sink, check the table schema id still exists:
let table = catalog.load_table(&table_ident).await?;
let schema = table.metadata().schema_by_id(expected_schema_id);
if schema.is_none() {
    // table was replaced or schema id is stale: recreate the sink
}

fn schema_exists(table: &Table, schema_id: i32) -> bool {
    table.metadata().schema_by_id(schema_id).is_some()
}

When it happens

Trigger: The table's metadata was replaced (e.g., schema dropped/rewritten by an external process or catalog swap) so the schema_id captured at write time no longer exists; or stale/buggy write results carry an invalid schema_id; or the catalog loaded a different table version concurrently.

Common situations: Another engine (Spark/Trino/Flink) replaced or expired table metadata under RisingWave; a `CREATE OR REPLACE` / drop-and-recreate of the target table; pointing the sink at a table that was recreated with different schema ids; iceberg catalog backends with eventual consistency returning older metadata.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Fail to convert version_hint from utf8 to string: {}
• Can't create iceberg sink write result from empty data!
• iceberg sink metadata should be an object
• schema_id should be a u64
• apply iceberg fast_append

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/564a90e6007fb545. Report an issue: GitHub.