risingwavelabs/risingwave · error · BackupError

expect state_store

Error message

expect state_store

What it means

MetaSnapshotV2::storage_url extracts the state_store system parameter from the snapshot's system_parameters list and requires exactly one entry named "state_store"; otherwise it throws "expect state_store". The snapshot must record the Hummock state store URL for restore to know where storage lives.

Source

Thrown at src/storage/backup/src/meta_snapshot_v2.rs:213

    }

    fn hummock_version_ref(&self) -> &HummockVersion {
        &self.hummock_version
    }

    fn hummock_version(self) -> HummockVersion {
        self.hummock_version
    }

    fn storage_url(&self) -> BackupResult<String> {
        let storage_url_from_snapshot =
            Itertools::exactly_one(self.system_parameters.iter().filter_map(|m| {
                if m.name == "state_store" {
                    return Some(m.value.clone());
                }
                None
            }))
            .map_err(|_| BackupError::Other(anyhow!("expect state_store")))?;
        storage_url_from_snapshot
            .strip_prefix("hummock+")
            .map(|s| s.to_owned())
            .ok_or_else(|| {
                BackupError::Other(anyhow!(
                    "invalid state_store from metadata snapshot: {}",
                    storage_url_from_snapshot
                ))
            })
    }

    fn storage_directory(&self) -> BackupResult<String> {
        Itertools::exactly_one(self.system_parameters.iter().filter_map(|m| {
            if m.name == "data_directory" {
                return Some(m.value.clone());
            }
            None
        }))

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the backup's system_parameters section contains exactly one state_store entry (inspect with risectl or decode the snapshot).
2. Recreate the backup from a healthy cluster so system parameters (state_store, data_directory) are captured.
3. If restoring from a legacy snapshot, use the matching older restore path or supply the state store URL manually.
4. Check for duplicated parameter rows in the source cluster's system_parameter table before snapshotting.

Example fix

// before: assuming the field exists
let url = snapshot.storage_url()?;
// after: check presence and fall back to operator-supplied config
let url = match snapshot.storage_url() {
    Ok(u) => u,
    Err(_) => read_state_store_from_restore_config()?,
};

let n = snapshot.system_parameters.iter().filter(|m| m.name == "state_store").count();
if n != 1 { return Err(anyhow!("snapshot has {} state_store entries, expected 1", n)); }

fn has_single_param<'a>(ps: &'a [SystemParam], name: &str) -> Option<&'a str> {
    let mut it = ps.iter().filter(|p| p.name == name).map(|p| p.value.as_str());
    match (it.next(), it.next()) { (Some(v), None) => Some(v), _ => None }
}

match snapshot.storage_url() {
    Ok(url) => url,
    Err(e) if e.to_string().contains("expect state_store") => {
        log::warn!("snapshot lacks state_store; using restore config");
        restore_config.state_store.clone()
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling storage_url() on a decoded V2 meta snapshot whose system_parameters list has zero or multiple entries named "state_store" — e.g. parameters were never persisted, an older snapshot omitted them, or a corrupted/duplicated parameters section.

Common situations: Restoring a backup created without system parameters recorded; a backup made from a cluster configured unusually (parameters split across nodes) producing duplicates; hand-edited or partially decoded snapshots used in tests; old backup format missing the field.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• expect data_directory
• snapshot id {} not found
• {k} not found in system_parameter table
• Meta storage is not empty before being restored
• invalid state_store from metadata snapshot: {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/1235e03c37462c1f. Report an issue: GitHub.