apache/druid · error · ParseException

Fail to decode protobuf message!

Error message

Fail to decode protobuf message!

What it means

Once the descriptor is resolved, the decoder extracts the payload bytes after the 5-byte Confluent header and parses them with DynamicMessage.parseFrom(descriptor, rawMessage). Any exception during this final parse (malformed bytes, wrong schema, truncated record) is wrapped in a ParseException with this message.

Source

Thrown at extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/SchemaRegistryBasedProtobufBytesDecoder.java:204

      LOGGER.error(e.getMessage());
      throw new ParseException(
          null,
          e,
          "Fail to get protobuf schema because of can not connect to registry or failed http request!"
      );
    }
    catch (IOException e) {
      LOGGER.error(e.getMessage());
      throw new ParseException(null, e, "Fail to get protobuf schema because of invalid schema!");
    }
    try {
      byte[] rawMessage = new byte[length];
      bytes.get(rawMessage, 0, length);
      return DynamicMessage.parseFrom(descriptor, rawMessage);
    }
    catch (Exception e) {
      LOGGER.error(e.getMessage());
      throw new ParseException(null, e, "Fail to decode protobuf message!");
    }
  }

  @Override
  public boolean equals(Object o)
  {
    if (this == o) {
      return true;
    }
    if (o == null || getClass() != o.getClass()) {
      return false;
    }

    SchemaRegistryBasedProtobufBytesDecoder that = (SchemaRegistryBasedProtobufBytesDecoder) o;

    return Objects.equals(url, that.url) &&
           Objects.equals(capacity, that.capacity) &&
           Objects.equals(urls, that.urls) &&

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Verify the .proto used by producers matches the schema registered under the subject/id (same field numbers and types)
2. Ensure producers write the Confluent wire format (magic byte 0 + schema id + payload) when this decoder is used
3. Re-publish or skip the corrupt record; check Kafka retention/compression settings for truncation issues

Example fix

// before: raw protobuf bytes without the Confluent header sent to the topic
producer.send(new byte[]{...})
// after: use KafkaProtobufSerializer so the header matches the decoder
props.put(ProducerProperties.VALUE_SERIALIZER, KafkaProtobufSerializer.class);
producer.send(new ProducerRecord<>(topic, msg));

// validate producer/consumer schema compatibility out-of-band
protoc --decode=<Msg> --proto_path=. msg.proto < payload.bin

try {
  byte[] parsed = decoder.parse(bytes);
} catch (ParseException e) {
  if (e.getMessage().contains("Fail to decode protobuf message")) {
    log.error("Payload does not match schema id descriptor; check producer schema wire format", e.getCause());
    deadLetterQueue.send(record, e);
  }
}

When it happens

Trigger: DynamicMessage.parseFrom throws (InvalidProtocolBufferException or anything else) because the payload bytes do not match the descriptor fetched for the envelope's schema id, or the message bytes are truncated/corrupt.

Common situations: Producer and consumer using different .proto definitions for the same subject; a Kafka record written by a different wire format (raw protobuf without Confluent header) so offsets/sizes are wrong; corrupted/truncated messages on the topic.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• Invalid Wrapper type.
• Invalid Struct type.
• Invalid Value type.
• Invalid ListValue type.
• Protobuf message could not be parsed

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/f79ad396a190b513. Report an issue: GitHub.