apache/beam · error · IllegalStateException

Table not found:

Error message

Table not found: 

What it means

BigQueryTableSource.getEstimatedSizeBytes estimates source size by reading the table's numBytes metadata from DatasetService.getTable. If the table lookup returns null, an IllegalStateException is thrown. Like the sibling error in BigQueryStorageTableSource, the message concatenates the null table, so it renders as 'Table not found: null'.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java:81

  @Override
  protected TableReference getTableToExtract(BigQueryOptions bqOptions) throws IOException {
    return tableDef.getTableReference(bqOptions);
  }

  @Override
  public synchronized long getEstimatedSizeBytes(PipelineOptions options) throws Exception {
    Long maybeNumBytes = tableSizeBytes.get();
    if (maybeNumBytes != null) {
      return maybeNumBytes;
    } else {
      BigQueryOptions bqOptions = options.as(BigQueryOptions.class);
      TableReference tableRef = tableDef.getTableReference(bqOptions);
      try (DatasetService datasetService = bqServices.getDatasetService(bqOptions)) {
        Table table = datasetService.getTable(tableRef);

        if (table == null) {
          throw new IllegalStateException("Table not found: " + table);
        }

        Long numBytes = table.getNumBytes();
        if (numBytes == null) {
          // Tables that don't report storage statistics, e.g. Lakehouse runtime catalog
          // (BigLake metastore) tables.
          numBytes = 0L;
        }
        if (table.getStreamingBuffer() != null
            && table.getStreamingBuffer().getEstimatedBytes() != null) {
          numBytes += table.getStreamingBuffer().getEstimatedBytes().longValue();
        }

        tableSizeBytes.compareAndSet(null, numBytes);
        return numBytes;
      }
    }
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Confirm the table exists with the pipeline's credentials (`bq show project:dataset.table`)
2. Set --project/--bigQueryProject to the correct project containing the table
3. Grant the service account bigquery.tables.get / roles/bigquery.metadataViewer on the dataset
4. Guard table lifecycle: ensure the table isn't dropped between pipeline construction and execution (recreate or skip reads of ephemeral tables)

Example fix

// before
pipeline.apply(BigQueryIO.readTableRows().from("proj:ds." + tableName)); // tableName may be stale
// after
TableReference ref = TableReference.from("proj", "ds", tableName);
// pre-flight check with same credentials:
// bqClient.getTable(ref) != null  ->  else fail fast with a clear message
pipeline.apply(BigQueryIO.readTableRows().from(ref));

BigQuery bq = BigQueryOptions.getDefaultInstance().getService();
if (bq.getTable("proj", "ds", "tbl") == null) {
  throw new IllegalArgumentException("proj:ds.tbl not found or not accessible");
}

boolean tableExists(BigQuery bq, TableReference ref) {
  return ref != null && bq.getTable(ref) != null;
}

try {
  source.getEstimatedSizeBytes(options);
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Table not found")) {
    // fall back to a conservative size estimate or fail fast with a clear message
  }
  throw e;
}

When it happens

Trigger: Cost/size estimation phase of a BOUNDED_READ (extract/QUERY) BigQueryIO read when the referenced table is missing, in the wrong project, invisible to the credentials, or deleted after job graph construction.

Common situations: Typo'd dataset/table; pipeline built against a dev dataset but run against prod credentials; service account lacking metadataViewer; BigLake/external tables in datasets the caller can't describe.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Table not found
• BigQuery table " + tableReference + " not found. If you want
• Dataset {} does not exist in your project. You have to creat
• BigQuery %1$s not found for table "%2$s" . Please create the
• Query job %s failed, status: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d92a04e1746027a6. Report an issue: GitHub.