{"record":{"id":"20d40dbea545f1c9","repo":"stanfordnlp/CoreNLP","slug":"keys-are-not-being-serialized-keystoserialize","errorCode":null,"errorMessage":"Keys are not being serialized: ${keysToSerialize}","messagePattern":"Keys are not being serialized: (.+?)","errorType":"exception","errorClass":"LossySerializationException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/ProtobufAnnotationSerializer.java","lineNumber":255,"sourceCode":"    return map.get(key);\n  }\n\n  /**\n   * Create a CoreLabel proto from a CoreLabel instance.\n   * This is not static, as it optionally throws an exception if the serialization is lossy.\n   * @param coreLabel The CoreLabel to convert\n   * @return A protocol buffer message corresponding to this CoreLabel\n   */\n  public CoreNLPProtos.Token toProto(CoreLabel coreLabel) {\n    return toProto(coreLabel, Collections.emptySet());\n  }\n\n  public CoreNLPProtos.Token toProto(CoreLabel coreLabel, Set<Class<?>> keysToSkip) {\n    Set<Class<?>> keysToSerialize = new HashSet<>(coreLabel.keySetNotNull());\n    CoreNLPProtos.Token.Builder builder = toProtoBuilder(coreLabel, keysToSerialize, keysToSkip);\n    // Completeness check\n    if (enforceLosslessSerialization && !keysToSerialize.isEmpty()) {\n      throw new LossySerializationException(\"Keys are not being serialized: \" + StringUtils.join(keysToSerialize));\n    }\n    return builder.build();\n  }\n\n  /**\n   * <p>\n   *   The method to extend by subclasses of the Protobuf Annotator if custom additions are added to Tokens.\n   *   In contrast to {@link ProtobufAnnotationSerializer#toProto(edu.stanford.nlp.ling.CoreLabel)}, this function\n   *   returns a builder that can be extended.\n   * </p>\n   *\n   * @param coreLabel The sentence to save to a protocol buffer\n   * @param keysToSerialize A set tracking which keys have been saved. It's important to remove any keys added to the proto\n   *                        from this set, as the code tracks annotations to ensure lossless serialization\n   */\n  protected CoreNLPProtos.Token.Builder toProtoBuilder(CoreLabel coreLabel, Set<Class<?>> keysToSerialize, Set<Class<?>> keysToSkip) {\n    CoreNLPProtos.Token.Builder builder = CoreNLPProtos.Token.newBuilder();\n    Set<Class<?>> keySet = coreLabel.keySetNotNull();","sourceCodeStart":237,"sourceCodeEnd":273,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/ProtobufAnnotationSerializer.java#L237-L273","documentation":"When lossless serialization is enforced, ProtobufAnnotationSerializer converts a CoreLabel to protobuf and checks that every annotation key was consumed by the builder. Any unconsumed keys trigger LossySerializationException because the proto output would silently drop data.","triggerScenarios":"Serializing a CoreLabel (toProto(CoreLabel, Set<Class<?>>)) that carries custom or uncommon annotation keys not handled by toProtoBuilder while ProtobufAnnotationSerializer.ENFORCE_LOSSLESS_SERIALIZATION (system property) is true.","commonSituations":"Adding custom CoreMap keys via annotators or post-processing, then serializing with enforceLosslessSerialization=true (default in server mode); upgrading CoreNLP and adding new key types not yet mapped in the serializer.","solutions":["Identify the unserialized key names from the exception message and remove them before serialization, or serialize them yourself","Run the JVM with -DProtobufAnnotationSerializer.enforceLosslessSerialization=false to allow lossy serialization if data loss is acceptable","Register/extend the serializer's toProtoBuilder to handle your custom key classes"],"exampleFix":"// before\ntoken.set(MyCustomKey.class, value);\nproto = serializer.toProto(token, Collections.emptySet()); // throws\n// after\nproto = serializer.toProto(token, Collections.singleton(MyCustomKey.class)); // skip custom key","handlingStrategy":"try-catch","validationCode":"Set<Class<?>> unexpected = new HashSet<>(coreLabel.keySetNotNull()); unexpected.removeAll(KNOWN_SERIALIZABLE_KEYS); if (!unexpected.isEmpty() && enforceLossless) { /* strip or skip before serializing */ }","typeGuard":null,"tryCatchPattern":"try { proto = serializer.toProto(token, keysToSkip); } catch (ProtobufAnnotationSerializer.LossySerializationException e) { logger.warn(\"Lossy keys: \" + e.getMessage()); proto = serializeWithEnforcementDisabled(token); }","preventionTips":["Don't add custom keys to CoreLabels you plan to proto-serialize losslessly","Know the enforceLosslessSerialization system property before using the server"],"tags":["java","serialization","protobuf","corenlp"],"backgroundTag":"schema-validation-failed","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}