risingwavelabs/risingwave · error · BackupError

expect data_directory

Error message

expect data_directory

What it means

MetaSnapshotV2::storage_directory extracts the data_directory system parameter and requires exactly one entry via Itertools::exactly_one; otherwise it throws "expect data_directory". The snapshot must record the cluster's data directory so restore can rebuild the catalog path.

Source

Thrown at src/storage/backup/src/meta_snapshot_v2.rs:232

        storage_url_from_snapshot
            .strip_prefix("hummock+")
            .map(|s| s.to_owned())
            .ok_or_else(|| {
                BackupError::Other(anyhow!(
                    "invalid state_store from metadata snapshot: {}",
                    storage_url_from_snapshot
                ))
            })
    }

    fn storage_directory(&self) -> BackupResult<String> {
        Itertools::exactly_one(self.system_parameters.iter().filter_map(|m| {
            if m.name == "data_directory" {
                return Some(m.value.clone());
            }
            None
        }))
        .map_err(|_| BackupError::Other(anyhow!("expect data_directory")))
    }

    fn table_change_log_object_ids(&self) -> HashSet<HummockRawObjectId> {
        self.hummock_table_change_logs
            .iter()
            .flat_map(|m| {
                // We cannot use `change_log_ssts` here because `to_table_change_log` returns an owned value, not a reference.
                let EpochNewChangeLog {
                    new_value,
                    old_value,
                    ..
                } = to_table_change_log(m);
                new_value
                    .into_iter()
                    .chain(old_value)
                    .map(|t| t.object_id.as_raw())
            })
            .collect()

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the snapshot's system_parameters and confirm a single data_directory entry exists before restoring.
2. Recreate the backup from a healthy cluster so data_directory is recorded.
3. If migrating a legacy snapshot, inject the data_directory parameter into the decoded model before calling storage_directory.
4. Deduplicate system_parameter rows in the source cluster (keep the one from the latest config version) and re-snapshot.

Example fix

// before: assuming presence
let dir = snapshot.storage_directory()?;
// after: fallback to restore-time config
let dir = snapshot.storage_directory()
    .unwrap_or_else(|_| restore_config.data_directory.clone());

let n = snapshot.system_parameters.iter().filter(|m| m.name == "data_directory").count();
if n != 1 { return Err(anyhow!("snapshot has {} data_directory entries, expected 1", n)); }

fn has_single_param<'a>(ps: &'a [SystemParam], name: &str) -> Option<&'a str> {
    let mut it = ps.iter().filter(|p| p.name == name).map(|p| p.value.as_str());
    match (it.next(), it.next()) { (Some(v), None) => Some(v), _ => None }
}

match snapshot.storage_directory() {
    Ok(dir) => dir,
    Err(e) if e.to_string().contains("expect data_directory") => {
        log::warn!("snapshot lacks data_directory; using restore config");
        restore_config.data_directory.clone()
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling storage_directory() on a V2 snapshot whose system_parameters list has zero or multiple entries named "data_directory" — parameters missing from the backup, duplicated rows, or a corrupted parameters section.

Common situations: Backups taken from clusters where data_directory was never persisted; duplicate system_parameter rows from config migration bugs; legacy snapshots predating the parameter's capture; tests decoding snapshots with empty parameter lists.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• expect state_store
• snapshot id {} not found
• {k} not found in system_parameter table
• Meta storage is not empty before being restored
• invalid state_store from metadata snapshot: {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/b94022d4a54f6b99. Report an issue: GitHub.