apache/beam · warning

Could not get the row count for the table

Error message

Could not get the row count for the table {}

What it means

BigQueryTable queries BigQuery for the table's row count to produce Calcite table statistics. If that lookup fails with IOException or InterruptedException, the error is logged and BOUNDED_UNKNOWN statistics are returned instead. Query planning still proceeds; only the optimizer's row-count estimate is degraded.

Solutions

1. Check the logged exception cause for auth/network/permission errors and fix (credentials, IAM roles: roles/bigquery.dataViewer + jobUser).
2. Verify the bqLocation identifier points to an existing table.
3. Re-run planning without worker interruption; statistics will populate on next use.
4. Ignore safely if approximate statistics are acceptable — behavior falls back to BOUNDED_UNKNOWN.

Example fix

// before
} catch (IOException | InterruptedException e) {
  LOG.warn("Could not get the row count for the table {}", bqLocation, e);
}
// after
// No change needed by caller; ensure permissions via gcloud:
// gcloud projects add-iam-policy-binding PROJECT --member=user:ME --role=roles/bigquery.jobUser

// Verify table access before planning:
bq.getTable(TableId.from(project, dataset, table)) // throws on missing permission/table

try {
  BeamTableStatistics stats = bigQueryTable.getTableStatistics();
} catch (Exception e) {
  // library already falls back to BOUNDED_UNKNOWN; optionally supply own estimate
  stats = BeamTableStatistics.BOUNDED_UNKNOWN;
}

When it happens

Trigger: Calling getRowCountFromBQ (via getTableStatistics) when the BigQuery tables.get/query API call throws IOException (network, auth, permission) or the calling thread is interrupted while waiting for the response.

Common situations: Missing BigQuery permissions (bigquery.tables.get / bigquery.jobs.create); network or quota issues; pipeline worker shutdown interrupting planning; misconfigured bqLocation (project:dataset.table).

Related errors

• Copy job failed.
• Could not execute the query
• Failed to get table with
• Schema update load job
• Unable to confirm BigQuery %1$s presence for table "%2$s"…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/83c6d4dcab83a23f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java:259

        .withMethod(method)
        .from(bqLocation)
        .withCoder(SchemaCoder.of(schema));
  }

  private static BeamTableStatistics getRowCountFromBQ(PipelineOptions o, String bqLocation) {
    try {
      BigInteger rowCount =
          BigQueryHelpers.getNumRows(
              o.as(BigQueryOptions.class), BigQueryHelpers.parseTableSpec(bqLocation));

      if (rowCount == null) {
        return BeamTableStatistics.BOUNDED_UNKNOWN;
      }

      return BeamTableStatistics.createBoundedTableStatistics(rowCount.doubleValue());

    } catch (IOException | InterruptedException e) {
      LOG.warn("Could not get the row count for the table {}", bqLocation, e);
    }

    return BeamTableStatistics.BOUNDED_UNKNOWN;
  }

  public static class InvalidPropertyException extends UnsupportedOperationException {
    private InvalidPropertyException(String s) {
      super(s);
    }
  }
}

View on GitHub (pinned to 12126d8942)