What causes "%s reading %s from proto3" in openzipkin/zipkin?

Call getCause() on the exception to find the exact protobuf defect (wireType, fieldNumber, truncation, varint size) and byte offset.

openzipkin/zipkin · error · IllegalArgumentException

%s reading %s from proto3

Error message

%s reading %s from proto3

What it means

Zipkin's proto3 codec wraps lower-level protobuf decode failures into an IllegalArgumentException formatted '<cause> reading <type> from proto3'. The wrapped cause is usually one of the 'Malformed: ...' messages from Proto3Fields/ReadBuffer (invalid wireType, fieldNumber zero, truncated buffer, oversized varint). The original exception stays attached as the cause.

Source

Thrown at zipkin/src/main/java/zipkin2/internal/Proto3Codec.java:74

    int length = buffer.available();
    if (length == 0) return false;
    try {
      while (buffer.pos() < length) {
        Span span = SPAN.read(buffer);
        if (span == null) return false;
        out.add(span);
      }
    } catch (RuntimeException e) {
      throw exceptionReading("List<Span>", e);
    }
    return true;
  }

  static IllegalArgumentException exceptionReading(String type, Exception e) {
    String cause = e.getMessage() == null ? "Error" : e.getMessage();
    if (cause.contains("Malformed")) cause = "Malformed";
    String message = format("%s reading %s from proto3", cause, type);
    throw new IllegalArgumentException(message, e);
  }
}

View on GitHub (pinned to 878ce2a1fa)

Solutions

Call getCause() on the exception to find the exact protobuf defect (wireType, fieldNumber, truncation, varint size) and byte offset.
Confirm both sides use the same encoding: encode with SpanBytesEncoder.PROTO3 and decode with SpanBytesDecoder.PROTO3; check the Content-Type of the collector.
Re-encode the payload from a known-good source (e.g. decode as JSON then re-encode to proto3) to rule out producer bugs.
In streaming consumers, isolate and drop the offending message so one bad span cannot stop the collector.

Example fix

// before
List<Span> spans = SpanBytesDecoder.PROTO3.decodeList(bytes);

// after
List<Span> spans;
try {
  spans = SpanBytesDecoder.PROTO3.decodeList(bytes);
} catch (IllegalArgumentException e) {
  LOG.warn("dropping malformed proto3 span batch of {} bytes", bytes.length, e);
  spans = Collections.emptyList();
}

Defensive patterns

Strategy: try-catch

Try / catch

try { spans = SpanBytesDecoder.PROTO3.decodeList(bytes); } catch (IllegalArgumentException e) { LOG.warn("malformed proto3 batch ({} bytes)", bytes.length, e); spans = List.of(); }

Prevention

Use grpc/HTTP2 collectors only with proto3 Content-Type; keep JSON on the JSON endpoint.
Prefer SpanBytesEncoder.PROTO3 over hand serialization.
Log the cause chain — the wrapped 'Malformed: ...' message names the exact defect.

When it happens

Trigger: Calling SpanBytesDecoder.PROTO3.decode/decodeList with bytes that are not length-delimited protobuf spans: wrong wire format, corrupted/truncated payload, or bytes produced by an incompatible schema/field numbering. Also hit by zipkin-server's proto3 collector endpoint receiving garbage.

Common situations: Sending JSON to a grpc/proto3 endpoint; a hand-rolled encoder emitting wrong varint keys; a Kafka/RabbitMQ topic mixing encodings; partial reads from storage where only a prefix of the protobuf blob was stored.

Related errors

AI-assisted analysis of openzipkin/zipkin@878ce2a1fa (2026-08-14). Data as JSON: /api/errors/ef3dc117e1ff46ce. Report an issue: GitHub.