apache/beam · error · UnsupportedOperationException

Write to Cloud Bigtable is supported for flat schema only.

Error message

Write to Cloud Bigtable is supported for flat schema only.

What it means

BigtableTable only supports writing to Bigtable when the table schema is flat (all columns map directly to a single column family, keyed by the 'key' field). buildIOWriter throws UnsupportedOperationException when useFlatSchema is false, because complex/nested schemas cannot be translated into Bigtable rows by this provider.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTable.java:117

        .apply("BigtableRowToBeamRow", bigtableRowToRow())
        .setRowSchema(schema);
  }

  @Override
  public PCollection<Row> buildIOReader(
      PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames) {
    BigtableIO.Read readTransform = readTransform();
    if (filters instanceof BigtableFilter) {
      BigtableFilter bigtableFilter = (BigtableFilter) filters;
      readTransform = readTransform.withRowFilter(bigtableFilter.getFilters());
    }
    return readTransform.expand(begin).apply(bigtableRowToRow());
  }

  @Override
  public POutput buildIOWriter(PCollection<Row> input) {
    if (!useFlatSchema) {
      throw new UnsupportedOperationException(
          "Write to Cloud Bigtable is supported for flat schema only.");
    }
    BigtableIO.Write write =
        BigtableIO.write().withProjectId(projectId).withInstanceId(instanceId).withTableId(tableId);
    if (!emulatorHost.isEmpty()) {
      write = write.withEmulator(emulatorHost);
    }
    return input.apply(new BeamRowToBigtableMutation(columnsMapping)).apply(write);
  }

  @Override
  public PCollection.IsBounded isBounded() {
    return PCollection.IsBounded.BOUNDED;
  }

  @Override
  public BeamSqlTableFilter constructFilter(List<RexNode> filter) {
    return new BigtableFilter(filter, schema);

View on GitHub (pinned to 12126d8942)

Solutions

1. Write only from tables with flat schemas: keep all fields as primitive types plus the STRING 'key' field.
2. Flatten nested data before writing: project the query to flat columns first, e.g. INSERT INTO flat_bt SELECT key, s.a, s.b FROM ... .
3. If complex writes are genuinely needed, write via BigtableIO.Write directly in a pipeline instead of Beam SQL DML.

Example fix

-- before (nested schema, write fails)
CREATE EXTERNAL TABLE bt (key STRING, data STRUCT<a INT64>) TYPE 'bigtable' LOCATION '...';
INSERT INTO bt SELECT key, STRUCT(1 AS a) FROM t;

-- after (flat schema)
CREATE EXTERNAL TABLE bt (key STRING, data_a INT64) TYPE 'bigtable' LOCATION '...';
INSERT INTO bt SELECT key, 1 FROM t;

boolean flat = table.getSchema().getFields().stream()
    .allMatch(f -> !f.getType().getTypeName().isCompositeType());
if (!flat) {
  throw new IllegalArgumentException("Flatten the schema before writing to Bigtable via Beam SQL");
}

boolean isFlatSchema(Schema s) {
  return s.getFields().stream()
      .noneMatch(f -> f.getType().getTypeName().isCompositeType());
}

try {
  stmt.execute("INSERT INTO bt_table SELECT ... FROM ...");
} catch (UnsupportedOperationException e) {
  if (e.getMessage().contains("flat schema only")) {
    // rewrite the query with a flat projection
  }
}

When it happens

Trigger: Running an INSERT/DELETE/UPDATE (any DML that calls buildIOWriter) against a Bigtable SQL table whose schema is not flat, e.g. the schema has nested STRUCT/ARRAY fields or the columnsMapping implies a non-flat layout.

Common situations: Declaring a table with complex types to read nested Bigtable qualifiers, then attempting INSERT INTO it; schema changes that introduced nested fields after the table was created; trying to persist query output containing STRUCT/ARRAY columns into Bigtable.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• LOCATION is required
• Schema has to contain '%s' field
• key field type should be STRING but was %s
• Bigtable location must be in the following format: 'googleap
• Schema fields count: '%s' does not fit columnsMapping count:

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/47a7e80879b37d04. Report an issue: GitHub.