apache/flink · error · InvalidRecordException

Unknown type with descriptor

Error message

Unknown type with descriptor "{}" and type "{}."

What it means

The FieldWriter factory hit a FieldDescriptor whose Java type has no writer implementation in PatchedProtoWriteSupport (beyond the primitives, groups, enums, strings, bytes it handles). unknownType raises InvalidRecordException naming the descriptor and JavaType.

Solutions

1. Restrict the proto message to supported field types (varint, floating point, bool, string, bytes, enum, message, group-free maps)
2. Read the exception message: the descriptor name and JavaType identify the offending field - restructure that field (e.g. wrap in a message or convert to string/bytes)
3. If you control the format jar, extend the writer with a custom FieldWriter for that type

Set<FieldDescriptor.JavaType> ok = EnumSet.of(INT, LONG, FLOAT, DOUBLE, BOOLEAN, STRING, BYTE_STRING, ENUM, MESSAGE);
for (FieldDescriptor fd : descriptor.getFields()) { if (!ok.contains(fd.getJavaType())) throw new IllegalArgumentException("unsupported field: " + fd.getFullName()); }

try { writer.write(msg); } catch (InvalidRecordException e) { // parse field name from message; restructure that proto field } 

When it happens

Trigger: A protobuf field type the writer does not implement - in parquet-proto this typically surfaces for unusual JavaType values; most commonly triggered by unsupported combinations or new protobuf features (e.g. editions-era types) reaching createWriter.

Common situations: Exotic proto features or versions of protobuf-java introducing types the patched converter does not cover; custom FieldDescriptors with types outside the supported set.

Related errors

• Expected two fields for the map (key/value), but got
• Can not find column io for parquet reader.
• Cannot convert Protobuf message with extension field(s)
• Corrupted Parquet schema
• Do not support flink type:

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/7984f10f3f724cac. Report an issue: GitHub.

Appendix: source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/protobuf/PatchedProtoWriteSupport.java:853

    }

    class BytesValueWriter extends FieldWriter {
        @Override
        void writeRawValue(Object value) {
            byte[] byteArray = ((BytesValue) value).getValue().toByteArray();
            Binary binary = Binary.fromConstantByteArray(byteArray);
            recordConsumer.addBinary(binary);
        }
    }

    private FieldWriter unknownType(FieldDescriptor fieldDescriptor) {
        String exceptionMsg =
                "Unknown type with descriptor \""
                        + fieldDescriptor
                        + "\" and type \""
                        + fieldDescriptor.getJavaType()
                        + "\".";
        throw new InvalidRecordException(exceptionMsg);
    }
}

/**
 * Minimal schema converter extracting only needed behavior for the patched support. For the test
 * cases we only rely on primitive field mappings, so we can forward directly to the real converter
 * if present; else implement minimal mapping. To minimize risk, we reflectively invoke the original
 * ProtoSchemaConverter if available.
 */
class PatchedProtoSchemaConverter {
    private final ParquetConfiguration configuration;

    PatchedProtoSchemaConverter(ParquetConfiguration configuration) {
        this.configuration = configuration;
    }

    MessageType convert(Descriptor descriptor) {
        try {

View on GitHub (pinned to 2f3c205e92)