stanfordnlp/CoreNLP · error · Error

Error with entity construction, two tokens had inconsistent…

Error message

Error with entity construction, two tokens had inconsistent NER tags

What it means

ChineseQuantifiableEntityNormalizer concatenates tokens of one entity before normalizing it. In singleEntityToString it verifies every token in the List<CoreMap> carries the same NamedEntityTagAnnotation; if the tags differ, the 'entity' was constructed incorrectly and it throws an Error rather than producing garbage output.

Solutions

  1. Inspect the entity list logged by 'differing NER tags detected in entity:' and fix the code that built/merged it so all tokens share one NER tag
  2. Ensure every CoreLabel passed in has NamedEntityTagAnnotation set (call set(NamedEntityTagAnnotation.class, ...) on all tokens)
  3. Check for and disable custom NER post-processors/mergers that relabel individual tokens within an entity
  4. If intentional mixed input is expected, split the list into entities before calling the normalizer

Example fix

// before: entity list with mixed tags
List<CoreLabel> entity = mergeTokens(nerTokenA, plainTokenB); // tokenB has tag null/O
normalizer.normalize(entity); // throws
// after: set a consistent tag on all tokens
for (CoreLabel tok : entity) tok.set(CoreAnnotations.NamedEntityTagAnnotation.class, "NUMBER");
normalizer.normalize(entity);
Defensive patterns

Strategy: validation

Validate before calling

Set<String> tags = entity.stream()
    .map(w -> w.get(CoreAnnotations.NamedEntityTagAnnotation.class))
    .filter(Objects::nonNull)
    .collect(Collectors.toSet());
if (tags.size() != 1) throw new IllegalStateException("inconsistent NER tags: " + tags);

Prevention

When it happens

Trigger: Calling normalize/normalizeString (or running the NER pipeline feeding it) on a List<CoreMap> entity where tokens have inconsistent NamedEntityTagAnnotation values — e.g. a merge of adjacent tokens with different NER labels, or hand-built CoreMaps missing the tag on some tokens.

Common situations: Custom post-processing that splices tokens from different NER-labeled sentences into one entity list; CoreLabel lists built manually where the NER tag was only set on the first token; bugs in upstream entity merging in a customized pipeline.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/66031a47c8e338b3. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/ie/ChineseQuantifiableEntityNormalizer.java:934

    return s;
  }

  /**
   * Concatenate entity annotations to a String. Note that Chinese does not use space to separate
   * tokens so we will follow this convention here.
   *
   * @param l
   * @param <E>
   * @return
   */
  private static <E extends CoreMap> String singleEntityToString(List<E> l) {
    String entityType = l.get(0).get(CoreAnnotations.NamedEntityTagAnnotation.class);
    StringBuilder sb = new StringBuilder();
    for (E w : l) {
      if(!w.get(CoreAnnotations.NamedEntityTagAnnotation.class).equals(entityType)) {
        log.error("differing NER tags detected in entity: " + l);
        throw new Error("Error with entity construction, two tokens had inconsistent NER tags");
      }
      sb.append(w.get(CoreAnnotations.TextAnnotation.class));
    }
    return sb.toString();
  }

  private static String prettyNumber(String s) {
    if (s == null) {
      return null;
    }
    s = ! s.contains(".") ? s : s.replaceAll("0*$", "").replaceAll("\\.$", "");
    return s;
  }


  /**
   * Fix up the NER sequence in case this is necessary.
   *

View on GitHub (pinned to 1b7edd19c4)