apache/beam · error · InvalidTableException

Could not resolve table in Data Catalog: {tableName}

Error message

Could not resolve table in Data Catalog: {tableName}

What it means

When resolving a table name through DataCatalog's lookupEntry, the provider converts API errors (invalid argument, permission denied, not found) into Beam's InvalidTableException with the message "Could not resolve table in Data Catalog: <tableName>". The original GCP exception is attached as the cause.

Source

Thrown at sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java:151

    return tableProvider.buildBeamSqlTable(table);
  }

  private Table loadTable(String tableName) {
    if (!tableCache.containsKey(tableName)) {
      tableCache.put(tableName, loadTableFromDC(tableName));
    }

    return tableCache.get(tableName);
  }

  private Table loadTableFromDC(String tableName) {
    try {
      return toCalciteTable(
          tableName,
          dataCatalog.lookupEntry(
              LookupEntryRequest.newBuilder().setSqlResource(tableName).build()));
    } catch (InvalidArgumentException | PermissionDeniedException | NotFoundException e) {
      throw new InvalidTableException("Could not resolve table in Data Catalog: " + tableName, e);
    }
  }

  @Internal
  public static DataCatalogClient createDataCatalogClient(DataCatalogPipelineOptions options) {
    try {
      DataCatalogSettings.Builder builder =
          DataCatalogSettings.newBuilder()
              .setCredentialsProvider(() -> options.as(GcpOptions.class).getGcpCredential())
              .setEndpoint(options.getDataCatalogEndpoint());

      // Retry permission denied errors, they are likely due to sync delay.
      // Limit max retry delay to 1 minute, at that point its probably a legitimate permission error
      // and we should get back to the user.
      builder
          .lookupEntrySettings()
          .setRetryableCodes(
              ImmutableSet.of(Code.PERMISSION_DENIED, Code.DEADLINE_EXCEEDED, Code.UNAVAILABLE))

View on GitHub (pinned to 12126d8942)

Solutions

  1. Verify the table exists in DataCatalog and the name matches the expected SQL resource format (project[.:]dataset.table).
  2. Grant the caller IAM permission to look up the entry (roles/datacatalog.viewer or stronger).
  3. Check credentials/application default credentials point to the right project and inspect the cause exception for which of the three GCP errors fired.

Example fix

// before
Table t = provider.getTable("my_ds.my_table"); // unqualified name

// after
Table t = provider.getTable("my-proj.my_ds.my_table"); // fully qualified
Defensive patterns

Strategy: try-catch

Validate before calling

if (!name.matches("^([a-zA-Z0-9-_]+)[.:][a-zA-Z0-9-_]+[.:][a-zA-Z0-9-_]+$")) {
  throw new IllegalArgumentException("Use fully-qualified project.dataset.table name");
}

Try / catch

try {
  Table t = provider.getTable(name);
} catch (InvalidTableException e) {
  // inspect e.getCause(): NotFoundException | PermissionDeniedException | InvalidArgumentException
}

Prevention

When it happens

Trigger: Calling getTable/loadTable with a tableName that (a) does not exist in DataCatalog, (b) the caller lacks permission to view, or (c) is not a valid SQL resource name (expected format like project.dataset.table or project:dataset.table).

Common situations: Typos in table names, missing IAM roles (datacatalog.viewer) on the project, querying tables from a different GCP project than the credentials can see, or using the wrong fully-qualified name format.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/641fe04e0f70072c. Report an issue: GitHub.