{"record":{"id":"61c302fa3e26a836","repo":"apache/beam","slug":"error-while-converting-fieldschema-to-hcatfieldschema","errorCode":null,"errorMessage":"Error while converting FieldSchema to HCatFieldSchema","messagePattern":"Error while converting FieldSchema to HCatFieldSchema","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/SchemaUtils.java","lineNumber":68,"sourceCode":"          .put(HCatFieldSchema.Type.VARCHAR, FieldType.STRING)\n          .put(HCatFieldSchema.Type.BINARY, FieldType.BYTES)\n          .put(HCatFieldSchema.Type.DATE, FieldType.DATETIME)\n          .put(HCatFieldSchema.Type.TIMESTAMP, FieldType.DATETIME)\n          .build();\n\n  static Schema toBeamSchema(List<FieldSchema> fields) {\n    return fields.stream().map(SchemaUtils::toBeamField).collect(toSchema());\n  }\n\n  private static Schema.Field toBeamField(FieldSchema field) {\n    String name = field.getName();\n    HCatFieldSchema hCatFieldSchema;\n\n    try {\n      hCatFieldSchema = HCatSchemaUtils.getHCatFieldSchema(field);\n    } catch (HCatException e) {\n      // Converting checked Exception to unchecked Exception.\n      throw new UnsupportedOperationException(\n          \"Error while converting FieldSchema to HCatFieldSchema\", e);\n    }\n\n    switch (hCatFieldSchema.getCategory()) {\n      case PRIMITIVE:\n        {\n          if (!HCAT_TO_BEAM_TYPES_MAP.containsKey(hCatFieldSchema.getType())) {\n            throw new UnsupportedOperationException(\n                \"The Primitive HCat type '\"\n                    + field.getType()\n                    + \"' of field '\"\n                    + name\n                    + \"' cannot be converted to Beam FieldType\");\n          }\n\n          FieldType fieldType = HCAT_TO_BEAM_TYPES_MAP.get(hCatFieldSchema.getType());\n          return Schema.Field.of(name, fieldType).withNullable(true);\n        }","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/SchemaUtils.java#L50-L86","documentation":"SchemaUtils.toBeamField converts a Hive/HCard FieldSchema into a Beam Schema.Field. HCatSchemaUtils.getHCatFieldSchema failed with a checked HCatException, so the library wraps it as an UnsupportedOperationException. This means the incoming field definition from the Metastore could not be interpreted as an HCatFieldSchema.","triggerScenarios":"Calling HCatalogIO.read() whose table schema contains a FieldSchema that HCatSchemaUtils rejects (malformed type string, unsupported type descriptor, corrupted Metastore metadata).","commonSituations":"Tables created with non-standard or legacy Hive type strings; Metastore schema drift after a Hive version upgrade; partition-key schemas that differ from the storage schema.","solutions":["Inspect the offending column's type string in the Hive Metastore (DESCRIBE table) and normalize it to a standard Hive type","Upgrade/align hive-hcatalog and parquet-hadoop dependency versions with the Metastore version","Pre-validate the table schema with HCatSchemaUtils.getHCatFieldSchema in a try/catch before running the Beam pipeline","If the column is unusable, project around it or copy the table with only supported types"],"exampleFix":"// before\nSchema schema = SchemaUtils.beamSchemaFromTableSchema(hcatSchema);\n// after\nfor (HCatFieldSchema f : hcatSchema.getFields()) {\n  try { HCatSchemaUtils.getHCatFieldSchema(f); }\n  catch (HCatException e) { throw new IllegalArgumentException(\"Bad column: \" + f.getName(), e); }\n}\nSchema schema = SchemaUtils.beamSchemaFromTableSchema(hcatSchema);","handlingStrategy":"validation","validationCode":"try { HCatSchemaUtils.getHCatFieldSchema(field); } catch (HCatException e) { throw new IllegalArgumentException(\"Unsupported field \" + field.getName(), e); }","typeGuard":"null","tryCatchPattern":"try { Schema s = SchemaUtils.beamSchema(hcatSchema); } catch (UnsupportedOperationException e) { throw new PipelineSchemaException(\"HCatalog schema invalid: \" + e.getMessage(), e); }","preventionTips":["DESCRIBE tables before reading; look for odd type strings","Keep Hive/HCatalog client versions aligned with the Metastore","Pre-validate schemas in a pipeline-construct step","Pin column types to standard Hive primitives"],"tags":["java","schema","hcatalog","hive"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}