apache/beam · error · IllegalArgumentException

Table specification [ ] is not in one of the expected…

Error message

Table specification [%s] is not in one of the expected formats ( [project_id]:[dataset_id].[table_id], [project_id].[dataset_id].[table_id], [dataset_id].[table_id], [project_id]:[catalog_id].[namespace_id].[table_id], [project_id].[catalog_id].[namespace_id].[table_id])

What it means

BigQueryHelpers.parseTableSpec parses a BigQuery table specification string against the TABLE_SPEC regex; when the string does not match any accepted format it throws this IllegalArgumentException (via invalidTableSpec). Accepted forms include project:dataset.table, project.dataset.table, dataset.table, and the two catalog/namespace variants.

Solutions

1. Print/inspect the offending string and reformat it to one of the accepted patterns, e.g. project:dataset.table.
2. Use BigQueryHelpers.parseTableSpec only after validating with the same pattern, or build a TableReference programmatically instead of parsing a string.
3. If the spec comes from CLI/options, validate it at argument-parse time with a regex or by attempting parse early and failing fast.
4. Escape or strip whitespace/quotes that may have been captured with the value.

Example fix

// before
TableReference ref = BigQueryHelpers.parseTableSpec(input); // input = "mytable"
// after
TableReference ref = input.contains(".")
    ? BigQueryHelpers.parseTableSpec(input)
    : BigQueryHelpers.parseTableSpec("my-project:mydataset." + input);

public static boolean isValidTableSpec(String s) {
  return s != null && java.util.regex.Pattern.compile(
      "((?<project>[\p{L}0-9-.]+):)?(?<dataset>[\p{L}0-9-_.]+)\\.(?<table>[\p{L}0-9-$]+)").matcher(s).matches();
}

static boolean looksLikeTableSpec(String s) {
  return s != null && s.contains(".") && s.chars().filter(c -> c == ':').count() <= 1;
}

try { ref = BigQueryHelpers.parseTableSpec(spec); } catch (IllegalArgumentException e) { throw new IllegalArgumentException("Bad --table option: " + spec, e); }

When it happens

Trigger: Passing a malformed table spec to parseTableSpec (or BigQueryIO read/write .from(...)) — e.g. "mytable" with no dot, "a:b:c.d.t" with stray colons, trailing dots, or illegal characters in identifiers.

Common situations: Hardcoded table strings with typos; pipeline options where the user omitted the dataset; interpolating variables that leave an empty segment; confusing BigQuery legacy vs. newer catalog formats.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Failure parsing TableRow
• table name has empty components
• table name missing components
• Table reference [ ] must include at least a dataset and a…
• A BigQuery table or a query must be specified

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/876d1b3969f386c3. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java:480

  }

  /**
   * Parse a table specification in the form {@code "[project_id]:[dataset_id].[table_id]"} or
   * {@code "[project_id].[dataset_id].[table_id]"} or {@code "[dataset_id].[table_id]"}.
   *
   * <p>Lakehouse runtime catalog (BigLake metastore) tables are referenced with four parts, {@code
   * "[project_id].[catalog_id].[namespace_id].[table_id]"} (or {@code
   * "[project_id]:[catalog_id].[namespace_id].[table_id]"}); these parse to a composite {@code
   * "[catalog_id].[namespace_id]"} dataset id, which is the form the BigQuery APIs accept for such
   * tables. More generally, when a specification contains more than three segments, everything
   * between the project id and the final (table) segment becomes the dataset id.
   *
   * <p>If the project id is omitted, the default project id is used.
   */
  public static TableReference parseTableSpec(String tableSpec) {
    Matcher match = BigQueryIO.TABLE_SPEC.matcher(tableSpec);
    if (!match.matches()) {
      throw invalidTableSpec(tableSpec);
    }

    // Table ids cannot contain '.', so the table is always the segment after
    // the last dot.
    int lastDot = tableSpec.lastIndexOf('.');
    String table = tableSpec.substring(lastDot + 1);
    String prefix = tableSpec.substring(0, lastDot);

    String project = null;
    String dataset;
    long colonCount = prefix.chars().filter(c -> c == ':').count();
    if (colonCount == 0) {
      // No colon means the purely dotted form ("p.d.t", "d.t", "p.catalog.ns.t"): the
      // leading segment is the project id when it is a plausible project id.
      // (Dataset ids may contain characters such as '_' that project ids may
      // not, in which case the whole prefix is the dataset id.)
      // The firstDot < length-1 guard keeps degenerate trailing-dot specs
      // ("pp..t", accepted by the character-set gate with dataset "pp.")

View on GitHub (pinned to 12126d8942)