apache/druid · error · ParseException

Protobuf message could not be parsed

Error message

Protobuf message could not be parsed

What it means

ProtobufReader.convertMessage delegates record conversion to ProtobufConverter.convertMessage, which can throw InvalidProtocolBufferException when well-known types are malformed (e.g. invalid Value/Struct/wrapper shapes). The reader wraps that failure in a ParseException with this message, aborting conversion of that record.

Source

Thrown at extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/ProtobufReader.java:103

    final Map<String, Object> record;
    final Map<String, Object> plainJava = convertMessage(intermediateRow);
    record = recordFlattener.flatten(plainJava);
    return Collections.singletonList(MapInputRowParser.parse(inputRowSchema, record));
  }

  @Override
  protected List<Map<String, Object>> toMap(DynamicMessage intermediateRow)
  {
    return Collections.singletonList(convertMessage(intermediateRow));
  }

  private static Map<String, Object> convertMessage(Message msg)
  {
    try {
      return ProtobufConverter.convertMessage(msg);
    }
    catch (InvalidProtocolBufferException e) {
      throw new ParseException(null, e, "Protobuf message could not be parsed");
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Inspect the offending record and decode it standalone with protoc to find the malformed well-known type
2. Fix the producer so Value oneofs set exactly one field and wrappers/Struct/ListValue follow the standard layout
3. Align protobuf-java versions across Druid extensions to eliminate descriptor mismatches

Example fix

// before: producer writes two Value oneof members
builder.setNumberValue(1.0); builder.setStringValue("x");
// after: exactly one
builder.setNumberValue(1.0);

// pre-validate a sample record before wiring ingestion
Message sample = DynamicMessage.parseFrom(descriptor, sampleBytes);
Map<String, Object> row = ProtobufConverter.convertMessage(sample); // throws early if malformed

boolean isConvertible(Message msg) {
  try {
    ProtobufConverter.convertMessage(msg);
    return true;
  } catch (InvalidProtocolBufferException e) {
    return false;
  }
}

try {
  Map<String, Object> row = reader.toMap(record);
} catch (ParseException e) {
  log.error("Record failed protobuf conversion; send to DLQ", e);
  deadLetterQueue.send(record, e);
}

When it happens

Trigger: Any record whose message tree contains a well-known type the specialized converters reject (Value with >1 oneof set, descriptors missing 'value'/'fields'/'values'), or a JsonFormat print failure for Any, encountered while reading rows via plainJava or toMap.

Common situations: Malformed records produced upstream; mixed protobuf runtime versions in extensions; hand-written bytes or a corrupt Kafka message.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• Invalid Wrapper type.
• Invalid Struct type.
• Invalid Value type.
• Invalid ListValue type.
• Fail to decode protobuf message!

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/3a15a1dd888a5661. Report an issue: GitHub.