stanfordnlp/CoreNLP · error · LossySerializationException

Keys are not being serialized

Error message

Keys are not being serialized: ${keysToSerialize}

What it means

When lossless serialization is enforced, ProtobufAnnotationSerializer converts a CoreLabel to protobuf and checks that every annotation key was consumed by the builder. Any unconsumed keys trigger LossySerializationException because the proto output would silently drop data.

Solutions

1. Identify the unserialized key names from the exception message and remove them before serialization, or serialize them yourself
2. Run the JVM with -DProtobufAnnotationSerializer.enforceLosslessSerialization=false to allow lossy serialization if data loss is acceptable
3. Register/extend the serializer's toProtoBuilder to handle your custom key classes

Example fix

// before
token.set(MyCustomKey.class, value);
proto = serializer.toProto(token, Collections.emptySet()); // throws
// after
proto = serializer.toProto(token, Collections.singleton(MyCustomKey.class)); // skip custom key

Set<Class<?>> unexpected = new HashSet<>(coreLabel.keySetNotNull()); unexpected.removeAll(KNOWN_SERIALIZABLE_KEYS); if (!unexpected.isEmpty() && enforceLossless) { /* strip or skip before serializing */ }

try { proto = serializer.toProto(token, keysToSkip); } catch (ProtobufAnnotationSerializer.LossySerializationException e) { logger.warn("Lossy keys: " + e.getMessage()); proto = serializeWithEnforcementDisabled(token); }

When it happens

Trigger: Serializing a CoreLabel (toProto(CoreLabel, Set<Class<?>>)) that carries custom or uncommon annotation keys not handled by toProtoBuilder while ProtobufAnnotationSerializer.ENFORCE_LOSSLESS_SERIALIZATION (system property) is true.

Common situations: Adding custom CoreMap keys via annotators or post-processing, then serializing with enforceLosslessSerialization=true (default in server mode); upgrading CoreNLP and adding new key types not yet mapped in the serializer.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• RuntimeIOException wrapping IOException
• Failed to save classifier
• ERROR: Invalid dependency node line
• ERROR: Invalid format for dependency graph
• ERROR: Incorrect format for the serialized coref graph

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/20d40dbea545f1c9. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/ProtobufAnnotationSerializer.java:255

    return map.get(key);
  }

  /**
   * Create a CoreLabel proto from a CoreLabel instance.
   * This is not static, as it optionally throws an exception if the serialization is lossy.
   * @param coreLabel The CoreLabel to convert
   * @return A protocol buffer message corresponding to this CoreLabel
   */
  public CoreNLPProtos.Token toProto(CoreLabel coreLabel) {
    return toProto(coreLabel, Collections.emptySet());
  }

  public CoreNLPProtos.Token toProto(CoreLabel coreLabel, Set<Class<?>> keysToSkip) {
    Set<Class<?>> keysToSerialize = new HashSet<>(coreLabel.keySetNotNull());
    CoreNLPProtos.Token.Builder builder = toProtoBuilder(coreLabel, keysToSerialize, keysToSkip);
    // Completeness check
    if (enforceLosslessSerialization && !keysToSerialize.isEmpty()) {
      throw new LossySerializationException("Keys are not being serialized: " + StringUtils.join(keysToSerialize));
    }
    return builder.build();
  }

  /**
   * <p>
   *   The method to extend by subclasses of the Protobuf Annotator if custom additions are added to Tokens.
   *   In contrast to {@link ProtobufAnnotationSerializer#toProto(edu.stanford.nlp.ling.CoreLabel)}, this function
   *   returns a builder that can be extended.
   * </p>
   *
   * @param coreLabel The sentence to save to a protocol buffer
   * @param keysToSerialize A set tracking which keys have been saved. It's important to remove any keys added to the proto
   *                        from this set, as the code tracks annotations to ensure lossless serialization
   */
  protected CoreNLPProtos.Token.Builder toProtoBuilder(CoreLabel coreLabel, Set<Class<?>> keysToSerialize, Set<Class<?>> keysToSkip) {
    CoreNLPProtos.Token.Builder builder = CoreNLPProtos.Token.newBuilder();
    Set<Class<?>> keySet = coreLabel.keySetNotNull();

View on GitHub (pinned to 1b7edd19c4)