apache/seatunnel · error · CatalogException

Failed to list BigQuery databases (datasets)

Error message

Failed to list BigQuery databases (datasets)

What it means

BigQueryCatalog.listDatabases() wraps any exception thrown while calling the Google BigQuery client's bigquery.listDatasets() (and iterating results) into a generic CatalogException. It signals that the metadata listing RPC against BigQuery failed; the underlying cause (auth, network, quota, permissions) is attached as the cause. The catalog itself does not interpret the failure, it only reports that datasets could not be listed.

Source

Thrown at seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/catalog/BigQueryCatalog.java:163

    @Override
    public boolean databaseExists(String databaseName) throws CatalogException {
        if (databaseName == null || databaseName.trim().isEmpty()) {
            databaseName = config.get(BigQuerySinkOptions.DATASET_ID);
        }
        return bigquery.getDataset(databaseName) != null;
    }

    @Override
    public List<String> listDatabases() throws CatalogException {
        List<String> databases = new ArrayList<>();
        try {
            Page<Dataset> datasets = bigquery.listDatasets();
            for (Dataset dataset : datasets.iterateAll()) {
                databases.add(dataset.getDatasetId().getDataset());
            }
        } catch (Exception e) {
            throw new CatalogException("Failed to list BigQuery databases (datasets)", e);
        }
        return databases;
    }

    @Override
    public List<String> listTables(String databaseName)
            throws CatalogException, DatabaseNotExistException {
        if (databaseName == null || databaseName.trim().isEmpty()) {
            databaseName = config.get(BigQuerySinkOptions.DATASET_ID);
        }
        if (!databaseExists(databaseName)) {
            throw new DatabaseNotExistException(catalogName, databaseName);
        }
        List<String> tables = new ArrayList<>();
        try {
            Page<Table> bqTables = bigquery.listTables(databaseName);
            for (Table table : bqTables.iterateAll()) {
                tables.add(table.getTableId().getTable());

View on GitHub (pinned to cf67b549a7)

Solutions

1. Verify credentials: ensure GOOGLE_APPLICATION_CREDENTIALS (or the configured service account key) is valid by running `gcloud auth application-default login` or `bq ls` with the same key.
2. Check the service account has at least `roles/bigquery.metadataViewer` (and `roles/bigquery.user`) on the target project.
3. Confirm the project id in the catalog options is correct and the BigQuery API (`bigquery.googleapis.com`) is enabled via `gcloud services enable bigquery.googleapis.com`.
4. Inspect the wrapped cause (`e.getCause()`) in logs — it distinguishes 401/403 auth issues from 429 quota or socket timeouts.
5. If on a restricted network, open egress to bigquery.googleapis.com:443 or configure a proxy for the BigQuery client.

Example fix

// before: running with expired/missing key
// GOOGLE_APPLICATION_CREDENTIALS=/old/revoked-key.json

// after: point to a valid service account key
export GOOGLE_APPLICATION_CREDENTIALS=/opt/seatunnel/keys/bq-sa.json
# and grant the SA: gcloud projects add-iam-policy-binding PROJECT_ID \
#   --member='serviceAccount:bq-sa@PROJECT.iam.gserviceaccount.com' \
#   --role='roles/bigquery.metadataViewer'

// Java, before calling catalog.listDatabases()
import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.ServiceOptions;

GoogleCredentials creds = GoogleCredentials.getApplicationDefault()
    .createScoped("https://www.googleapis.com/auth/cloud-platform");
creds.refreshIfExpired(); // throws early if key is invalid/expired
String project = ServiceOptions.getDefaultProjectId();
if (project == null) {
    throw new IllegalStateException("No GCP project configured; set GOOGLE_CLOUD_PROJECT");
}

static boolean hasValidBqCredentials() {
    return System.getenv("GOOGLE_APPLICATION_CREDENTIALS") != null
        && java.nio.file.Files.isReadable(java.nio.file.Path.of(System.getenv("GOOGLE_APPLICATION_CREDENTIALS")));
}

try {
    List<String> dbs = catalog.listDatabases();
} catch (CatalogException e) {
    Throwable cause = e.getCause();
    if (cause instanceof IOException || cause.getMessage().contains("401") || cause.getMessage().contains("403")) {
        // refresh credentials / fix IAM, then retry once
    } else if (cause instanceof BigQueryException be && be.getCode() == 429) {
        // backoff and retry
    }
    throw e;
}

When it happens

Trigger: Any exception inside `bigquery.listDatasets().iterateAll()`: invalid or expired Google Cloud credentials (service account JSON), missing bigquery.jobs.list / dataset metadata permissions on the project, network failure reaching bigquery.googleapis.com, project not set or wrong project id, BigQuery API not enabled, or quota/rate-limit (429) responses.

Common situations: Running SeaTunnel on a machine whose GOOGLE_APPLICATION_CREDENTIALS points to a missing or revoked key; the service account lacks BigQuery Metadata Viewer at project level; a typo'd project id in the catalog config; firewalled clusters (e.g. on-prem) without egress to googleapis.com; project where the BigQuery API was disabled after being previously used.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Failed to list tables in dataset:
• Failed to fetch metadata from Gravitino for metadata: %s
• Failed to open BigQueryCatalog
• Failed to create BigQuery table:
• Failed to drop BigQuery table:

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/b9e1da76286ded76. Report an issue: GitHub.