apache/beam · error

Failed to get table {} with {}

Error message

Failed to get table {} with {}

What it means

UpdateSchemaDestination.startZeroLoadJob fetches the destination table via BigQueryServices.DatasetService.getTable(); if this call throws IOException or InterruptedException, the code logs this warning and rethrows as RuntimeException, failing the update-schema-ahead-of-write step. InterruptedException triggers are re-interrupted implicitly by rethrow (caller should check), and IOException generally means an API/network failure or missing table permission.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/UpdateSchemaDestination.java:272

              .collect(Collectors.toList());
      loadConfig.setSchemaUpdateOptions(options);
    }
    if (!loadConfig
            .getWriteDisposition()
            .equals(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE.toString())
        && !loadConfig
            .getWriteDisposition()
            .equals(BigQueryIO.Write.WriteDisposition.WRITE_APPEND.toString())) {
      return null;
    }
    final Table destinationTable;
    try {
      destinationTable = datasetService.getTable(tableReference);
      if (destinationTable == null) {
        return null; // no need to update schema ahead if table does not exist
      }
    } catch (IOException | InterruptedException e) {
      LOG.warn("Failed to get table {} with {}", tableReference, e.toString());
      throw new RuntimeException(e);
    }
    // no need to update schema ahead if provided schema already matches destination schema
    // or when destination schema is null (the write will set the schema)
    // or when provided schema is null (e.g. when using CREATE_NEVER disposition)
    TableSchema destinationSchema = destinationTable.getSchema();
    if (destinationSchema == null
        || destinationSchema.isEmpty()
        || schema == null
        || destinationSchema.equals(schema)) {
      return null;
    }
    loadConfig.setSchema(schema);
    if (timePartitioning != null) {
      loadConfig.setTimePartitioning(timePartitioning);
    }

    if (clustering != null) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Read e.toString() in the log: if 403, grant the service account bigquery.tables.get (Data Viewer/Editor).
2. If IOException/timeouts, check network egress to bigquery.googleapis.com and BigQuery quotas.
3. Retry the pipeline; transient API errors are usually resolved on rerun.
4. If InterruptedException, check for pipeline cancellation racing with schema update and drain gracefully.

Example fix

// before: table not visible due to missing grant
// 'Failed to get table myproj:ds.tbl with ... 403'
// after: grant IAM to the runner service account
gsutil // gcloud projects add-iam-policy-binding proj --member=serviceAccount:sa@proj.iam.gserviceaccount.com --role=roles/bigquery.dataEditor

// pre-check table visibility with the same credentials
bq.getTable(projectId, datasetId, tableId); // null => it won't exist downstream either
// check access: bq.getDataset(ds).get(tableId)

When it happens

Trigger: datasetService.getTable(tableReference) throws — network failure calling BigQuery API, quota/timeout, permission denied, or the thread interrupted during the blocking call.

Common situations: Service account lacking bigquery.tables.get; transient Google API errors during heavy pipelines; VPC without access to bigquery.googleapis.com; worker interruption during teardown.

Related errors

• Unable to get project number
• Schema update load job {} failed with {}
• Copy job {} failed.
• Unable to verify project with ID ${projectId}
• Unable to confirm BigQuery %1$s presence for table "%2$s". I

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e22a4b2845b9f302. Report an issue: GitHub.