{"record":{"id":"b9e1da76286ded76","repo":"apache/seatunnel","slug":"failed-to-list-bigquery-databases-datasets","errorCode":null,"errorMessage":"Failed to list BigQuery databases (datasets)","messagePattern":"Failed to list BigQuery databases \\(datasets\\)","errorType":"exception","errorClass":"CatalogException","httpStatus":null,"severity":"error","filePath":"seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/catalog/BigQueryCatalog.java","lineNumber":163,"sourceCode":"\n    @Override\n    public boolean databaseExists(String databaseName) throws CatalogException {\n        if (databaseName == null || databaseName.trim().isEmpty()) {\n            databaseName = config.get(BigQuerySinkOptions.DATASET_ID);\n        }\n        return bigquery.getDataset(databaseName) != null;\n    }\n\n    @Override\n    public List<String> listDatabases() throws CatalogException {\n        List<String> databases = new ArrayList<>();\n        try {\n            Page<Dataset> datasets = bigquery.listDatasets();\n            for (Dataset dataset : datasets.iterateAll()) {\n                databases.add(dataset.getDatasetId().getDataset());\n            }\n        } catch (Exception e) {\n            throw new CatalogException(\"Failed to list BigQuery databases (datasets)\", e);\n        }\n        return databases;\n    }\n\n    @Override\n    public List<String> listTables(String databaseName)\n            throws CatalogException, DatabaseNotExistException {\n        if (databaseName == null || databaseName.trim().isEmpty()) {\n            databaseName = config.get(BigQuerySinkOptions.DATASET_ID);\n        }\n        if (!databaseExists(databaseName)) {\n            throw new DatabaseNotExistException(catalogName, databaseName);\n        }\n        List<String> tables = new ArrayList<>();\n        try {\n            Page<Table> bqTables = bigquery.listTables(databaseName);\n            for (Table table : bqTables.iterateAll()) {\n                tables.add(table.getTableId().getTable());","sourceCodeStart":145,"sourceCodeEnd":181,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/catalog/BigQueryCatalog.java#L145-L181","documentation":"BigQueryCatalog.listDatabases() wraps any exception thrown while calling the Google BigQuery client's bigquery.listDatasets() (and iterating results) into a generic CatalogException. It signals that the metadata listing RPC against BigQuery failed; the underlying cause (auth, network, quota, permissions) is attached as the cause. The catalog itself does not interpret the failure, it only reports that datasets could not be listed.","triggerScenarios":"Any exception inside `bigquery.listDatasets().iterateAll()`: invalid or expired Google Cloud credentials (service account JSON), missing bigquery.jobs.list / dataset metadata permissions on the project, network failure reaching bigquery.googleapis.com, project not set or wrong project id, BigQuery API not enabled, or quota/rate-limit (429) responses.","commonSituations":"Running SeaTunnel on a machine whose GOOGLE_APPLICATION_CREDENTIALS points to a missing or revoked key; the service account lacks BigQuery Metadata Viewer at project level; a typo'd project id in the catalog config; firewalled clusters (e.g. on-prem) without egress to googleapis.com; project where the BigQuery API was disabled after being previously used.","solutions":["Verify credentials: ensure GOOGLE_APPLICATION_CREDENTIALS (or the configured service account key) is valid by running `gcloud auth application-default login` or `bq ls` with the same key.","Check the service account has at least `roles/bigquery.metadataViewer` (and `roles/bigquery.user`) on the target project.","Confirm the project id in the catalog options is correct and the BigQuery API (`bigquery.googleapis.com`) is enabled via `gcloud services enable bigquery.googleapis.com`.","Inspect the wrapped cause (`e.getCause()`) in logs — it distinguishes 401/403 auth issues from 429 quota or socket timeouts.","If on a restricted network, open egress to bigquery.googleapis.com:443 or configure a proxy for the BigQuery client."],"exampleFix":"// before: running with expired/missing key\n// GOOGLE_APPLICATION_CREDENTIALS=/old/revoked-key.json\n\n// after: point to a valid service account key\nexport GOOGLE_APPLICATION_CREDENTIALS=/opt/seatunnel/keys/bq-sa.json\n# and grant the SA: gcloud projects add-iam-policy-binding PROJECT_ID \\\n#   --member='serviceAccount:bq-sa@PROJECT.iam.gserviceaccount.com' \\\n#   --role='roles/bigquery.metadataViewer'","handlingStrategy":"try-catch","validationCode":"// Java, before calling catalog.listDatabases()\nimport com.google.auth.oauth2.GoogleCredentials;\nimport com.google.cloud.ServiceOptions;\n\nGoogleCredentials creds = GoogleCredentials.getApplicationDefault()\n    .createScoped(\"https://www.googleapis.com/auth/cloud-platform\");\ncreds.refreshIfExpired(); // throws early if key is invalid/expired\nString project = ServiceOptions.getDefaultProjectId();\nif (project == null) {\n    throw new IllegalStateException(\"No GCP project configured; set GOOGLE_CLOUD_PROJECT\");\n}","typeGuard":"static boolean hasValidBqCredentials() {\n    return System.getenv(\"GOOGLE_APPLICATION_CREDENTIALS\") != null\n        && java.nio.file.Files.isReadable(java.nio.file.Path.of(System.getenv(\"GOOGLE_APPLICATION_CREDENTIALS\")));\n}","tryCatchPattern":"try {\n    List<String> dbs = catalog.listDatabases();\n} catch (CatalogException e) {\n    Throwable cause = e.getCause();\n    if (cause instanceof IOException || cause.getMessage().contains(\"401\") || cause.getMessage().contains(\"403\")) {\n        // refresh credentials / fix IAM, then retry once\n    } else if (cause instanceof BigQueryException be && be.getCode() == 429) {\n        // backoff and retry\n    }\n    throw e;\n}","preventionTips":["Smoke-test the key with `bq ls` from the same host before running SeaTunnel jobs","Grant metadataViewer at project level, not per-dataset, for catalog operations","Keep service-account keys rotated and mounted with correct file permissions","Verify BigQuery API is enabled and egress to bigquery.googleapis.com:443 is open","Log e.getCause() so root cause is never hidden by the CatalogException wrapper"],"tags":["bigquery","gcp","authentication","network","metadata"],"backgroundTag":"api-request-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}