apache/beam · error · RuntimeException

Input Schema is invalid: Schema should be formatted in one

Error message

Input Schema is invalid: 

Schema should be formatted in one of two ways:
 key": ByteString
"type": String
"value": ByteString
"column_qualifier": ByteString
"family_name": String
"timestamp_micros": Long
"start_timestamp_micros": Long
"end_timestamp_micros": Long

OR

"key": ByteString
("mutations", contains map(String, ByteString) of mutations in the mutation schema format

What it means

BigtableWriteSchemaTransformProvider.expand validates that the input PCollection row schema matches one of two supported layouts for Bigtable writes (row key + explicit cell fields like type/value/column_qualifier, or row key + mutations map). If neither layout is detected, a RuntimeException with this message is thrown naming the offending schema and the two accepted formats.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java:202

        }
        if (inputSchema.hasField("column_qualifier")) {
          validateField(inputSchema, "column_qualifier", Schema.TypeName.BYTES);
        }
        if (inputSchema.hasField("family_name")) {
          validateField(inputSchema, "family_name", Schema.TypeName.STRING);
        }
        if (inputSchema.hasField("timestamp_micros")) {
          validateField(inputSchema, "timestamp_micros", Schema.TypeName.INT64);
        }
        if (inputSchema.hasField("start_timestamp_micros")) {
          validateField(inputSchema, "start_timestamp_micros", Schema.TypeName.INT64);
        }
        if (inputSchema.hasField("end_timestamp_micros")) {
          validateField(inputSchema, "end_timestamp_micros", Schema.TypeName.INT64);
        }
        bigtableMutations = changeMutationInput(input);
      } else {
        throw new RuntimeException(
            "Input Schema is invalid: "
                + inputSchema
                + "\n\nSchema should be formatted in one of two ways:\n "
                + "key\": ByteString\n"
                + "\"type\": String\n"
                + "\"value\": ByteString\n"
                + "\"column_qualifier\": ByteString\n"
                + "\"family_name\": String\n"
                + "\"timestamp_micros\": Long\n"
                + "\"start_timestamp_micros\": Long\n"
                + "\"end_timestamp_micros\": Long\n"
                + "\nOR\n"
                + "\n"
                + "\"key\": ByteString\n"
                + "(\"mutations\", contains map(String, ByteString) of mutations in the mutation schema format");
      }

      if (bigtableMutations != null) {

View on GitHub (pinned to 12126d8942)

Solutions

Build input rows with a 'key' field of type ByteString plus either the mutation fields (type, value, column_qualifier, family_name, timestamp_micros) or a 'mutations' map of String->ByteString
Use Schema.field("key", Schema.FieldType.BYTES) etc. when constructing the row schema
Check the printed inputSchema in the error message and fix field names/types to match one of the two accepted layouts

Example fix

// before
Schema schema = Schema.builder().addStringField("rowKey").addStringField("val").build();
// after
Schema schema = Schema.builder()
    .addByteArrayField("key")
    .addStringField("type")
    .addByteArrayField("value")
    .addByteArrayField("column_qualifier")
    .addStringField("family_name")
    .build();

Defensive patterns

Strategy: validation

Validate before calling

boolean validBigtableInputSchema(Schema s) {
  return s.hasField("key") && s.getField("key").getType().equals(Schema.TypeName.BYTES)
      && (s.hasField("mutations") || s.hasField("type"));
}

Type guard

boolean isBigtableRowSchema(Schema schema) {
  return schema.hasField("key")
      && schema.getField("key").getType().getByteStringType() != null
      && (schema.hasField("mutations") || schema.hasField("value"));
}

Prevention

Define input row schemas exactly as documented (key ByteString + cell fields or mutations map)
Print and diff the failing schema against the accepted layouts from the error message
Add an assertSchema step before applying the transform

When it happens

Trigger: Applying the Bigtable schema transform to a PCollection<Row> whose schema contains none of the expected field combinations — e.g. rows built for a different sink, missing the 'key' ByteString field, or having wrong field names/types.

Common situations: Wiring a Beam Row stream from an upstream transform with a mismatched schema; typos in field names ('keys' vs 'key'); using String instead of ByteString for key/value/qualifier fields.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9c81d179420cd8e6. Report an issue: GitHub.