risingwavelabs/risingwave · error · BackupError

{k} not found in system_parameter table

Error message

{k} not found in system_parameter table

What it means

In restore overwrite, each persisted system parameter is applied by looking it up in the `system_parameter` table and updating its value. If the key `k` is absent from the table, the code raises a MetaStorage error rather than inserting, because overwrite assumes the restore already seeded the parameter rows.

Source

Thrown at src/meta/src/backup_restore/restore_impl/v2.rs:196

        new_storage_url: &str,
        new_storage_dir: &str,
        new_backup_url: &str,
        new_backup_dir: &str,
    ) -> BackupResult<()> {
        let kvs = [
            ("state_store", new_storage_url),
            ("data_directory", new_storage_dir),
            ("backup_storage_url", new_backup_url),
            ("backup_storage_directory", new_backup_dir),
        ];
        for (k, v) in kvs {
            let Some(model) = risingwave_meta_model::system_parameter::Entity::find_by_id(k)
                .one(&self.meta_store.conn)
                .await
                .map_err(map_db_err)?
            else {
                return Err(BackupError::MetaStorage(
                    anyhow::anyhow!("{k} not found in system_parameter table").into(),
                ));
            };
            let mut kv: risingwave_meta_model::system_parameter::ActiveModel = model.into();
            kv.value = sea_orm::ActiveValue::Set(v.to_owned());
            risingwave_meta_model::system_parameter::Entity::update(kv)
                .exec(&self.meta_store.conn)
                .await
                .map_err(map_db_err)?;
        }
        Ok(())
    }
}

async fn ensure_all_meta_store_tables_are_empty(
    db: &impl sea_orm::ConnectionTrait,
) -> BackupResult<()> {
    macro_rules! ensure_entity_empty {
        ($($entity_mod:ident),* $(,)?) => {

View on GitHub (pinned to 6469eb736d)

Solutions

  1. Align versions: restore the backup into a meta node at the same or newer version than the backup was taken from.
  2. Insert the missing key into the system_parameter table manually (with its default value) before/instead of failing.
  3. Check the reported key name and confirm it is a valid system parameter for this cluster version.

Example fix

// before: fail when row missing
let Some(model) = Entity::find_by_id(k).one(&conn).await? else {
    return Err(...);
};
// after: upsert instead of failing
let kv = match Entity::find_by_id(k).one(&conn).await? {
    Some(model) => { let mut m: ActiveModel = model.into(); m.value = Set(v.to_owned()); m }
    None => ActiveModel { name: Set(k.clone()), value: Set(v.to_owned()), ..Default::default() },
};
Defensive patterns

Strategy: try-catch

Validate before calling

-- check the key exists before overwrite
SELECT name FROM system_parameter WHERE name = 'backup_write_bandwidth_limit';

Try / catch

match restore_result {
    Err(e) if e.to_string().contains("not found in system_parameter table") => {
        // version skew: run meta node >= backup version, or insert default row for the key, then retry
    }
    other => other?,
}

Prevention

When it happens

Trigger: Restoring a backup whose system_parameter key (e.g. a newer config name) does not exist as a row in the restored meta database — typically a version skew where the backup contains a parameter name unknown to the running meta node.

Common situations: Restoring a backup from a newer RisingWave version into an older meta node; manually pruning the system_parameter table; fresh meta schema that did not initialize default parameter rows.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/fc0df93d0f0ac0c4. Report an issue: GitHub.