openzipkin/zipkin · error · IllegalArgumentException

Incomplete annotation at {}

Error message

Incomplete annotation at {}

What it means

In V2 JSON span decoding, each element of the "annotations" array must carry both "timestamp" and "value"; after reading the object, V2SpanReader throws 'Incomplete annotation at <jsonPath>' when either is null. JSON_V2 is stricter than a lenient parser: model-required fields must be present even if the JSON itself is well-formed.

Source

Thrown at zipkin/src/main/java/zipkin2/internal/V2SpanReader.java:68

        builder.remoteEndpoint(ENDPOINT_READER.fromJson(reader));
      } else if (nextName.equals("annotations")) {
        reader.beginArray();
        while (reader.hasNext()) {
          reader.beginObject();
          Long timestamp = null;
          String value = null;
          while (reader.hasNext()) {
            nextName = reader.nextName();
            if (nextName.equals("timestamp")) {
              timestamp = reader.nextLong();
            } else if (nextName.equals("value")) {
              value = reader.nextString();
            } else {
              reader.skipValue();
            }
          }
          if (timestamp == null || value == null) {
            throw new IllegalArgumentException("Incomplete annotation at " + reader.getPath());
          }
          reader.endObject();
          builder.addAnnotation(timestamp, value);
        }
        reader.endArray();
      } else if (nextName.equals("tags")) {
        reader.beginObject();
        while (reader.hasNext()) {
          String key = reader.nextName();
          if (reader.peekNull()) {
            throw new IllegalArgumentException("No value at " + reader.getPath());
          }
          builder.putTag(key, reader.nextString());
        }
        reader.endObject();
      } else if (nextName.equals("debug")) {
        if (reader.nextBoolean()) builder.debug(true);
      } else if (nextName.equals("shared")) {

View on GitHub (pinned to 878ce2a1fa)

Solutions

1. Fix the producer to emit both "timestamp" (microseconds) and "value" for every annotation — simplest is to use brave's or zipkin's own encoder.
2. Inspect the JSON path in the message to find the exact offending annotation.
3. Check for serializers configured to skip default values (WRITE_DATES_AS_TIMESTAMPS off / NON_DEFAULT inclusion) that drop timestamp=0.
4. Reject the span at ingest with a clear error back to the producer.

Example fix

// before
{"annotations":[{"value":"sr"}]}

// after
{"annotations":[{"timestamp":1617235200000000,"value":"sr"}]}

boolean isCompleteV2Annotation(JsonNode n) { return n.hasNonNull("timestamp") && n.hasNonNull("value"); }

When it happens

Trigger: SpanBytesDecoder.JSON_V2.decode/decodeList on a span where an annotation object misses timestamp or value — e.g. {"annotations":[{"timestamp":1610000000000000}]} or the fields were renamed (ts/val) by a custom exporter.

Common situations: Custom reporters writing hand-shaped span JSON instead of using SpanBytesEncoder.JSON_V2; pipelines that drop zero-valued timestamps (some serializers omit 0L); field-name drift between brave/zipkin report format versions.

Related errors

• Incomplete annotation at {}
• No value at {}
• Expected json or thrift object, not list encoding
• %s reading %s from json
• No key at {}

AI-assisted analysis of openzipkin/zipkin@878ce2a1fa (2026-08-14). Data as JSON: /api/errors/f22c3a1f251f688a. Report an issue: GitHub.