{"record":{"id":"9c81d179420cd8e6","repo":"apache/beam","slug":"input-schema-is-invalid-schema-should-be-formatted-in-one-of","errorCode":null,"errorMessage":"Input Schema is invalid: \n\nSchema should be formatted in one of two ways:\n key\": ByteString\n\"type\": String\n\"value\": ByteString\n\"column_qualifier\": ByteString\n\"family_name\": String\n\"timestamp_micros\": Long\n\"start_timestamp_micros\": Long\n\"end_timestamp_micros\": Long\n\nOR\n\n\"key\": ByteString\n(\"mutations\", contains map(String, ByteString) of mutations in the mutation schema format","messagePattern":"Input Schema is invalid: \n\nSchema should be formatted in one of two ways:\n key\": ByteString\n\"type\": String\n\"value\": ByteString\n\"column_qualifier\": ByteString\n\"family_name\": String\n\"timestamp_micros\": Long\n\"start_timestamp_micros\": Long\n\"end_timestamp_micros\": Long\n\nOR\n\n\"key\": ByteString\n\\(\"mutations\", contains map\\(String, ByteString\\) of mutations in the mutation schema format","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java","lineNumber":202,"sourceCode":"        }\n        if (inputSchema.hasField(\"column_qualifier\")) {\n          validateField(inputSchema, \"column_qualifier\", Schema.TypeName.BYTES);\n        }\n        if (inputSchema.hasField(\"family_name\")) {\n          validateField(inputSchema, \"family_name\", Schema.TypeName.STRING);\n        }\n        if (inputSchema.hasField(\"timestamp_micros\")) {\n          validateField(inputSchema, \"timestamp_micros\", Schema.TypeName.INT64);\n        }\n        if (inputSchema.hasField(\"start_timestamp_micros\")) {\n          validateField(inputSchema, \"start_timestamp_micros\", Schema.TypeName.INT64);\n        }\n        if (inputSchema.hasField(\"end_timestamp_micros\")) {\n          validateField(inputSchema, \"end_timestamp_micros\", Schema.TypeName.INT64);\n        }\n        bigtableMutations = changeMutationInput(input);\n      } else {\n        throw new RuntimeException(\n            \"Input Schema is invalid: \"\n                + inputSchema\n                + \"\\n\\nSchema should be formatted in one of two ways:\\n \"\n                + \"key\\\": ByteString\\n\"\n                + \"\\\"type\\\": String\\n\"\n                + \"\\\"value\\\": ByteString\\n\"\n                + \"\\\"column_qualifier\\\": ByteString\\n\"\n                + \"\\\"family_name\\\": String\\n\"\n                + \"\\\"timestamp_micros\\\": Long\\n\"\n                + \"\\\"start_timestamp_micros\\\": Long\\n\"\n                + \"\\\"end_timestamp_micros\\\": Long\\n\"\n                + \"\\nOR\\n\"\n                + \"\\n\"\n                + \"\\\"key\\\": ByteString\\n\"\n                + \"(\\\"mutations\\\", contains map(String, ByteString) of mutations in the mutation schema format\");\n      }\n\n      if (bigtableMutations != null) {","sourceCodeStart":184,"sourceCodeEnd":220,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java#L184-L220","documentation":"BigtableWriteSchemaTransformProvider.expand validates that the input PCollection row schema matches one of two supported layouts for Bigtable writes (row key + explicit cell fields like type/value/column_qualifier, or row key + mutations map). If neither layout is detected, a RuntimeException with this message is thrown naming the offending schema and the two accepted formats.","triggerScenarios":"Applying the Bigtable schema transform to a PCollection<Row> whose schema contains none of the expected field combinations — e.g. rows built for a different sink, missing the 'key' ByteString field, or having wrong field names/types.","commonSituations":"Wiring a Beam Row stream from an upstream transform with a mismatched schema; typos in field names ('keys' vs 'key'); using String instead of ByteString for key/value/qualifier fields.","solutions":["Build input rows with a 'key' field of type ByteString plus either the mutation fields (type, value, column_qualifier, family_name, timestamp_micros) or a 'mutations' map of String->ByteString","Use Schema.field(\"key\", Schema.FieldType.BYTES) etc. when constructing the row schema","Check the printed inputSchema in the error message and fix field names/types to match one of the two accepted layouts"],"exampleFix":"// before\nSchema schema = Schema.builder().addStringField(\"rowKey\").addStringField(\"val\").build();\n// after\nSchema schema = Schema.builder()\n    .addByteArrayField(\"key\")\n    .addStringField(\"type\")\n    .addByteArrayField(\"value\")\n    .addByteArrayField(\"column_qualifier\")\n    .addStringField(\"family_name\")\n    .build();","handlingStrategy":"validation","validationCode":"boolean validBigtableInputSchema(Schema s) {\n  return s.hasField(\"key\") && s.getField(\"key\").getType().equals(Schema.TypeName.BYTES)\n      && (s.hasField(\"mutations\") || s.hasField(\"type\"));\n}","typeGuard":"boolean isBigtableRowSchema(Schema schema) {\n  return schema.hasField(\"key\")\n      && schema.getField(\"key\").getType().getByteStringType() != null\n      && (schema.hasField(\"mutations\") || schema.hasField(\"value\"));\n}","tryCatchPattern":null,"preventionTips":["Define input row schemas exactly as documented (key ByteString + cell fields or mutations map)","Print and diff the failing schema against the accepted layouts from the error message","Add an assertSchema step before applying the transform"],"tags":["gcp","bigtable","schema","beam"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}