What causes "fill secrets for iceberg" in risingwavelabs/risingwave?

List the sink's secret_refs and confirm each referenced secret exists; recreate missing secrets with the same refs.

fill secrets for iceberg

Error message

fill secrets for iceberg

What it means

build_iceberg_config resolves the sink's secret references through LocalSecretManager::fill_secrets before constructing the Iceberg catalog config. If a referenced secret cannot be read (missing from the store, bad ref id), the error is wrapped with this context so the caller knows secret resolution — not parsing — failed while building the coordinator's IcebergConfig.

Solutions

List the sink's secret_refs and confirm each referenced secret exists; recreate missing secrets with the same refs.
Recreate the sink (DROP SINK + CREATE SINK) so secret refs are re-registered and synced.
Check secret-manager logs for the exact ref id that failed and fix the ref name/id in the sink definition.
If it is a sync issue after recovery, restart the meta node so it reloads secret state.

Example fix

-- before: sink references a dropped secret
CREATE SINK s FROM mv WITH (connector='iceberg', secret=iceberg_secret_ref);
-- after: recreate the secret first, then the sink
CREATE SECRET iceberg_secret WITH (backend='meta', properties={'access_key':'...','secret_key':'...'});
CREATE SINK s FROM mv WITH (connector='iceberg', secret=iceberg_secret_ref);

Defensive patterns

Strategy: validation

Validate before calling

// before building the coordinator, ensure all refs resolve
for ref_id in sink.secret_refs.keys() {
    if LocalSecretManager::global().read_secret(ref_id).is_none() {
        return Err(anyhow!("secret {} referenced by sink is missing", ref_id));
    }
}

Try / catch

let cfg = match build_iceberg_config(&sink) {
    Ok(cfg) => cfg,
    Err(e) if e.to_string().contains("fill secrets for iceberg") => {
        // recreate the missing secret then retry
        recreate_secrets(&sink).await?;
        build_iceberg_config(&sink)?
    }
    Err(e) => return Err(e),
};

Prevention

Create secrets before the sink that references them.
Never drop a secret while a sink still references it.
Verify secret sync state on meta nodes after recovery.

When it happens

Trigger: Coordinator creation from a PbSink whose secret_refs entry cannot be filled — the referenced secret id does not exist or is unreadable on this meta node (secret dropped, not synced, or wrong ref in the sink definition).

Common situations: Secret deleted while the sink still references it; wrong secret ref name given at CREATE SINK; meta node missing synced secret state after restore/recovery.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/f5efb1c6d450179e. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/manager/iceberg_pk_index_sink/mod.rs:58

        .get(UPSTREAM_SOURCE_KEY)
        .map(|v| v.eq_ignore_ascii_case("iceberg"))
        .unwrap_or(false);
    let pk_index_enabled = properties
        .get(ENABLE_PK_INDEX)
        .map(|v| v.eq_ignore_ascii_case("true"))
        .unwrap_or(false);
    connector_match && pk_index_enabled
}

/// Build an [`IcebergConfig`] from a [`PbSink`], filling secret refs along the
/// way. Used at CREATE SINK time and during recovery to (re-)register the
/// commit coordinator.
pub fn build_iceberg_config(pb_sink: &PbSink) -> anyhow::Result<IcebergConfig> {
    let properties: BTreeMap<String, String> = pb_sink.properties.clone().into_iter().collect();
    let secret_refs: BTreeMap<_, _> = pb_sink.secret_refs.clone().into_iter().collect();
    let with_secrets = LocalSecretManager::global()
        .fill_secrets(properties, secret_refs)
        .map_err(|e| anyhow!(e).context("fill secrets for iceberg"))?;
    IcebergConfig::from_btreemap(with_secrets)
        .map_err(|e| anyhow!(e).context("parse iceberg config"))
}

View on GitHub (pinned to 6469eb736d)