apache/beam · error · IllegalArgumentException

Table not found

Error message

Table not found

What it means

BigQueryStorageTableSource.getTargetTable fetches table metadata via DatasetService.getTable to build the read session. If BigQuery returns null (the dataset or table doesn't exist, or isn't visible to the credentials), an IllegalArgumentException is thrown. Note the message concatenates the (null) table object, so it prints 'Table not foundnull' unless the table ref was intended.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageTableSource.java:223

        checkState(
            !Strings.isNullOrEmpty(options.getProject()),
            "No project ID set in %s or %s, cannot construct a complete %s",
            TableReference.class.getSimpleName(),
            BigQueryOptions.class.getSimpleName(),
            TableReference.class.getSimpleName());
        LOG.info(
            "Project ID not set in {}. Using default project from {}.",
            TableReference.class.getSimpleName(),
            BigQueryOptions.class.getSimpleName());
        tableReference.setProjectId(
            options.getBigQueryProject() == null
                ? options.getProject()
                : options.getBigQueryProject());
      }
      try (DatasetService datasetService = bqServices.getDatasetService(options)) {
        Table table = datasetService.getTable(tableReference);
        if (table == null) {
          throw new IllegalArgumentException("Table not found" + table);
        }
        cachedTable.compareAndSet(null, table);
        return table;
      }
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the fully-qualified table (project:dataset.table) exists: run `bq show project:dataset.table` with the same credentials the pipeline uses
2. Check the configured project: set --project / --bigQueryProject / BQ_PROJECT explicitly to the project that contains the table
3. Grant the runner service account roles/bigquery.metadataViewer (or at least bigquery.tables.get) on the dataset
4. Recheck that the table was not deleted/renamed between building the pipeline and running it

Example fix

// before
options.setBigQueryProject(null); // falls back to possibly wrong project
// after
pipelineOptions.setProject("my-data-project");
BigQueryOptions bq = pipelineOptions.as(BigQueryOptions.class);
bq.setBigQueryProject("my-data-project");
// pre-check before running the pipeline:
DatasetReference ds = DatasetReference.of("my-data-project", "my_dataset");
// ensure `bq show my-data-project:my_dataset.my_table` succeeds with these credentials

// Pre-flight with the same credentials the pipeline uses:
BigQuery bq = BigQueryOptions.getDefaultInstance().getService();
Table t = bq.getTable("project", "dataset", "table");
if (t == null) throw new IllegalArgumentException("Table project:dataset.table does not exist");

boolean tableExists(BigQuery bq, TableReference ref) {
  return bq.getTable(ref) != null;
}

try {
  pipeline.run().waitUntilFinish();
} catch (IllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Table not found")) {
    throw new IllegalStateException("Check table name, project and IAM access: " + e.getMessage(), e);
  }
  throw e;
}

When it happens

Trigger: Querying the Storage Read API for a table whose projectId/dataset/tableId doesn't exist; wrong project configured (options.getBigQueryProject()); credentials lacking bigquery.tables.get / metadata visibility; stale cached table reference after a table was deleted between pipeline construction and execution.

Common situations: Typo in dataset or table name; running in a different GCP project than expected (BQ_PROJECT unset); using a dataset-view the service account can't see; table dropped by a retention policy before the pipeline runs.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Table not found:
• BigQuery table " + tableReference + " not found. If you want
• Dataset {} does not exist in your project. You have to creat
• BigQuery %1$s not found for table "%2$s" . Please create the
• Query job %s failed, status: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d9ad43e82a74bec1. Report an issue: GitHub.