openzipkin/zipkin · error · IOException

Invalid span, expecting annotations array start, got: {}

Error message

Invalid span, expecting annotations array start, got: {}

What it means

Inside parseSpan, when the field name is 'annotations' the next token must be START_ARRAY ([...]); any other JSON type for that field throws IOException('Invalid span, expecting annotations array start, got: <token>'). The span format is strict: annotations is always an array of {timestamp,value} objects.

Source

Thrown at zipkin-storage/elasticsearch/src/main/java/zipkin2/elasticsearch/internal/JsonSerializers.java:88

          break;
        case "name":
          result.name(parser.getText());
          break;
        case "timestamp":
          result.timestamp(parser.getLongValue());
          break;
        case "duration":
          result.duration(parser.getLongValue());
          break;
        case "localEndpoint":
          result.localEndpoint(parseEndpoint(parser));
          break;
        case "remoteEndpoint":
          result.remoteEndpoint(parseEndpoint(parser));
          break;
        case "annotations":
          if (value != JsonToken.START_ARRAY) {
            throw new IOException("Invalid span, expecting annotations array start, got: " +
              value);
          }
          while (parser.nextToken() != JsonToken.END_ARRAY) {
            Annotation a = parseAnnotation(parser);
            result.addAnnotation(a.timestamp(), a.value());
          }
          break;
        case "tags":
          if (value != JsonToken.START_OBJECT) {
            throw new IOException("Invalid span, expecting tags object, got: " + value);
          }
          while (parser.nextValue() != JsonToken.END_OBJECT) {
            result.putTag(parser.currentName(), parser.getValueAsString());
          }
          break;
        case "debug":
          result.debug(parser.getBooleanValue());
          break;

View on GitHub (pinned to 878ce2a1fa)

Solutions

1. Correct the document: annotations must be an array, e.g. "annotations":[{"timestamp":...,"value":"..."}].
2. Find who wrote the malformed docs (custom ingest job?) and fix its serializer to match the Zipkin span JSON schema.
3. Delete/reindex offending documents so queries stop hitting them.
4. When building spans in code, use Span.Builder.addAnnotation so the shape can never be wrong.

Example fix

// before
{"traceId":"abc","id":"def","annotations":{"timestamp":1,"value":"sr"}}
// -> IOException: Invalid span, expecting annotations array start, got: VALUE_START_OBJECT/START_OBJECT

// after
{"traceId":"abc","id":"def","annotations":[{"timestamp":1470150004000000,"value":"sr"}]}

var span = objectMapper.readTree(spanJson);
if (span.has("annotations") && !span.get("annotations").isArray()) {
  throw new IOException("'annotations' must be an array of {timestamp,value} objects");
}

boolean annotationsWellFormed(JsonNode span) {
  JsonNode a = span.get("annotations");
  return a == null || (a.isArray() && a.allMatch(e -> e.isObject() && e.has("timestamp") && e.has("value")));
}

try { Span s = SPAN_PARSER.parse(p); }
catch (IOException e) {
  if (e.getMessage().startsWith("Invalid span, expecting annotations array start")) {
    // reject the document; fix the producer's serializer
  }
}

When it happens

Trigger: A span document whose 'annotations' field is an object, string, number, or null-token shape other than an array — e.g. {"traceId":"...","annotations":{"ts":1}} or annotations:"none" — reaching SPAN_PARSER from an index or test fixture.

Common situations: Hand-written JSON test fixtures; external systems writing spans directly into Zipkin's ES indices with a wrong schema; schema drift from custom ingest pipelines.

Related errors

• Not a valid JSON object, start token: %s
• End of input while parsing object.
• Invalid span, expecting tags object, got: {}
• Not a valid JSON object, start token: {}
• Incomplete annotation at {}

AI-assisted analysis of openzipkin/zipkin@878ce2a1fa (2026-08-14). Data as JSON: /api/errors/001905d6ad2c5969. Report an issue: GitHub.