risingwavelabs/risingwave · error · SinkError::BigQuery

`bigquery.local.path` and `bigquery.s3.path` set at least on

Error message

`bigquery.local.path` and `bigquery.s3.path` set at least one, configure as needed.

What it means

`get_auth_json_from_path` resolves the BigQuery service-account credentials JSON from one of three sources: inline `bigquery.credentials`, a local file path (`bigquery.local.path`), or an S3 path (`bigquery.s3.path`). If none of these properties is set, there is no way to authenticate and the sink throws this configuration error at sink creation time.

Source

Thrown at src/connector/src/sink/big_query.rs:248

        StorageWriterClient::new(credentials_file).await
    }

    async fn get_auth_json_from_path(&self, aws_auth_props: &AwsAuthProps) -> Result<String> {
        if let Some(credentials) = &self.credentials {
            Ok(credentials.clone())
        } else if let Some(local_path) = &self.local_path {
            std::fs::read_to_string(local_path)
                .map_err(|err| SinkError::BigQuery(anyhow::anyhow!(err)))
        } else if let Some(s3_path) = &self.s3_path {
            let url =
                Url::parse(s3_path).map_err(|err| SinkError::BigQuery(anyhow::anyhow!(err)))?;
            let auth_vec = load_file_descriptor_from_s3(&url, aws_auth_props)
                .await
                .map_err(|err| SinkError::BigQuery(anyhow::anyhow!(err)))?;
            Ok(String::from_utf8(auth_vec).map_err(|e| SinkError::BigQuery(e.into()))?)
        } else {
            Err(SinkError::BigQuery(anyhow::anyhow!(
                "`bigquery.local.path` and `bigquery.s3.path` set at least one, configure as needed."
            )))
        }
    }
}

#[serde_as]
#[derive(Clone, Debug, Deserialize, WithOptions)]
pub struct BigQueryConfig {
    #[serde(flatten)]
    pub common: BigQueryCommon,
    #[serde(flatten)]
    pub aws_auth_props: AwsAuthProps,
    pub r#type: String, // accept "append-only" or "upsert"

    #[serde(flatten)]
    pub unknown_fields: std::collections::HashMap<String, String>,
}

View on GitHub (pinned to 6469eb736d)

Solutions

1. Set `bigquery.local.path` to the local path of the service-account JSON file (works when the file is accessible on the compute node)
2. Set `bigquery.s3.path` to an s3:// URL pointing at the credentials file (requires AWS auth props)
3. Alternatively set `bigquery.credentials` inline with the service-account JSON (it is enforced as a secret)
4. Re-create the sink after adding the property; it cannot be fixed at runtime

Example fix

// before
WITH (
  connector = 'bigquery',
  type = 'append-only'
)
// after
WITH (
  connector = 'bigquery',
  type = 'append-only',
  bigquery.local.path = '/secrets/gcp_sa.json'
)

let has_creds = props.contains_key("bigquery.credentials")
    || props.contains_key("bigquery.local.path")
    || props.contains_key("bigquery.s3.path");
assert!(has_creds, "must set bigquery.credentials, bigquery.local.path, or bigquery.s3.path");

// Rust
match BigQueryConfig::from_btreemap(props) {
    Err(e) if e.to_string().contains("set at least one") => {
        return Err(anyhow!("configure bigquery.local.path or bigquery.s3.path before creating the sink"));
    }
    other => other.map(|_| ()),
}

When it happens

Trigger: Creating a BigQuery sink whose properties omit `bigquery.credentials`, `bigquery.local.path`, and `bigquery.s3.path` (or they are set to empty/null), so all three `if let Some(...)` branches fall through to the else.

Common situations: Users set the GCS/S3 path for data but forget the credential file location; users assume Workload Identity or environment-based auth is picked up automatically when the connector requires explicit credentials; properties renamed in a newer connector version.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• No auth method specified for Vault backend
• Both `access_key` and `secret_key` must be provided
• NATS connect mode `user_and_password` requires both `user` a
• NATS connect mode `credential` requires both `nkey` and `jwt
• `{}` must be {}, or {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/c3a88070a9ee4e1e. Report an issue: GitHub.