risingwavelabs/risingwave · error · SinkError

Google Pub/Sub error: {0}

Error message

Google Pub/Sub error: {0}

What it means

SinkError::GooglePubSub(anyhow::Error) in src/connector/src/sink/mod.rs:1149 wraps failures from the Google Cloud Pub/Sub sink connector, preserving the source anyhow::Error and backtrace from the google-cloud-pubsub client. Raised during client construction, authentication, or when publishing messages to a topic fails.

Source

Thrown at src/connector/src/sink/mod.rs:1148

    #[error("Http error: {0}")]
    Http(
        #[source]
        #[backtrace]
        anyhow::Error,
    ),
    #[error("Mqtt error: {0}")]
    Mqtt(
        #[source]
        #[backtrace]
        anyhow::Error,
    ),
    #[error("Nats error: {0}")]
    Nats(
        #[source]
        #[backtrace]
        anyhow::Error,
    ),
    #[error("Google Pub/Sub error: {0}")]
    GooglePubSub(
        #[source]
        #[backtrace]
        anyhow::Error,
    ),
    #[error("Doris/Starrocks connect error: {0}")]
    DorisStarrocksConnect(
        #[source]
        #[backtrace]
        anyhow::Error,
    ),
    #[error("Doris error: {0}")]
    Doris(String),
    #[error("DeltaLake error: {0}")]
    DeltaLake(
        #[source]
        #[backtrace]
        anyhow::Error,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify credentials: ensure GOOGLE_APPLICATION_CREDENTIALS points to a valid service-account JSON key (or ADC/workload identity is configured), then recreate/restart the sink.
2. Grant roles/pubsub.publisher on the topic to the service account and confirm project ID and topic name in the sink options.
3. Test with gcloud pubsub topics publish <topic> --message=hi using the same credentials.
4. Check network egress (DNS/TLS to pubsub.googleapis.com) from the RisingWave host or configure a proxy.
5. If publishing large messages, keep payloads under Pub/Sub limits or compress/shrink the encoded rows.

Example fix

// before: credentials not visible to RisingWave
CREATE SINK gcp_sink FROM mv WITH (
  connector = 'google_pubsub', topic_id = 'projects/my-proj/topics/events'
);
// after: export valid credentials before starting
// export GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa.json
CREATE SINK gcp_sink FROM mv WITH (
  connector = 'google_pubsub', topic_id = 'projects/my-proj/topics/events'
);

# Validate credentials and publish permission before creating the sink
[[ -n "$GOOGLE_APPLICATION_CREDENTIALS" && -f "$GOOGLE_APPLICATION_CREDENTIALS" ]] || echo 'missing SA key'
gcloud pubsub topics publish "$TOPIC_ID" --message=check --project="$PROJECT" || echo 'publish not permitted / topic missing'

fn pubsub_sink_config_ok(topic_id: &str, creds_path: Option<&str>) -> bool {
    topic_id.starts_with("projects/") && topic_id.contains("/topics/")
        && creds_path.map(|p| std::path::Path::new(p).exists()).unwrap_or(true)
}

// Map auth vs topic vs transport failures to different actions
match sink_result {
    Err(SinkError::GooglePubSub(e)) if is_auth_error(&e) => refresh_credentials(),
    Err(SinkError::GooglePubSub(e)) if is_not_found(&e) => verify_topic_exists(),
    Err(SinkError::GooglePubSub(e)) => retry_with_backoff(&e),
    Ok(v) => process(v),
}

When it happens

Trigger: CREATE SINK ... WITH (connector='google_pubsub') when: GOOGLE_APPLICATION_CREDENTIALS is unset/invalid, the service account lacks pubsub.publisher on the topic, the topic ID/project is wrong or nonexistent, gRPC to pubsub.googleapis.com fails, or a publish batch exceeds limits or times out.

Common situations: Missing or malformed service-account JSON key; workload identity/ADC not configured in the deployment environment; topic deleted or renamed after sink creation; private clusters without egress to Google APIs; message size over the 10MB Pub/Sub limit.

Related errors

• BigQuery error: {0}
• credentials must be set if not using the pubsub emulator
• `{CHANNEL}` and `{CHANNEL_COLUMN}` only one can be set
• Cannot find '{VALUE_FORMAT}',please set it.
• Google Pub/Sub sink only support append-only mode

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/d338707bee223902. Report an issue: GitHub.