stanfordnlp/CoreNLP · warning

ChineseUtils.normalize warning: non-BMP codepoint U+

Error message

ChineseUtils.normalize warning: non-BMP codepoint U+

What it means

ChineseUtils.normalize (via normalizeBMP) logs a warning when it encounters a valid high surrogate paired with a following code unit — i.e. an astral-plane (non-BMP) character above U+FFFF. Such codepoints fall outside the BMP ranges the Chinese normalization tables handle, so the character is not normalized and is warned about.

Solutions

1. Strip or map non-BMP codepoints to a BMP equivalent before calling ChineseUtils.normalize
2. Pre-filter the input: reject or log-and-skip strings containing supplementary characters
3. Update/patch the normalization tables to cover the needed Plane-2 ideographs
4. Ensure the input encoding/Unicode version matches what the corpus expects; convert with ICU transcoding

Example fix

// before
String norm = ChineseUtils.normalize(raw, false, false);
// after
String cleaned = raw.codePoints().filter(cp -> cp <= 0xFFFF).collect(StringBuilder::new, StringBuilder::appendCodePoint, StringBuilder::append).toString();
String norm = ChineseUtils.normalize(cleaned, false, false);

static boolean isBmpOnly(String s) {
  return s.codePoints().allMatch(cp -> cp <= 0xFFFF);
}
if (!isBmpOnly(input)) throw new IllegalArgumentException("non-BMP codepoint in Chinese input: " + input);

static String stripSupplementary(String s) {
  return s.codePoints().filter(cp -> cp <= 0xFFFF)
      .collect(StringBuilder::new, StringBuilder::appendCodePoint, StringBuilder::append).toString();
}

When it happens

Trigger: Calling ChineseUtils.normalize on a string containing supplementary-plane characters such as rare CJK ideographs in Unicode Plane 2 (U+20000+), emoji, or other non-BMP text.

Common situations: Processing text files containing CJK Extension B/C/D characters common in classical Chinese corpora; mixing in emoji or symbols; corpus data converted to a wider Unicode version than the normalization tables expect.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• ChineseUtils.normalize warning: unmatched high surrogate…
• CTBunk file not found
• CTBunk I/O error
• Sentiment analysis is not implemented for Chinese
• after W derivative, index() != x.length()

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/0c1045aea0d5af0d. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/trees/international/pennchinese/ChineseUtils.java:114

        spaceChar < 0 || spaceChar > MAX_LEGAL) {
      throw new IllegalArgumentException("ChineseUtils: Unknown parameter option");
    }
    if (ONLY_BMP) {
      return normalizeBMP(in, ascii, spaceChar, midDot);
    } else {
      return normalizeUnicode(in, ascii, spaceChar, midDot);
    }
  }


  private static String normalizeBMP(String in, int ascii, int spaceChar, int midDot) {
    StringBuilder out = new StringBuilder();
    int len = in.length();
    for (int i = 0; i < len; i++) {
      char cp = in.charAt(i);
      if (Character.isHighSurrogate(cp)) {
        if (i + 1 < len) {
          log.warn("ChineseUtils.normalize warning: non-BMP codepoint U+" +
                  Integer.toHexString(Character.codePointAt(in, i)) + " in " + in);
        } else {
          log.warn("ChineseUtils.normalize warning: unmatched high surrogate character U+" +
                  Integer.toHexString(Character.codePointAt(in, i)) + " in " + in);
        }
      }
      Character.UnicodeBlock cub = Character.UnicodeBlock.of(cp);
      if (cub == Character.UnicodeBlock.PRIVATE_USE_AREA ||
              cub == Character.UnicodeBlock.SUPPLEMENTARY_PRIVATE_USE_AREA_A ||
              cub == Character.UnicodeBlock.SUPPLEMENTARY_PRIVATE_USE_AREA_B) {
        EncodingPrintWriter.err.println("ChineseUtils.normalize warning: private use area codepoint U+" + Integer.toHexString(cp) + " in " + in);
      }
      boolean delete = false;
      switch (ascii) {
        case LEAVE:
          break;
        case ASCII:
          if (cp >= '\uFF01' && cp <= '\uFF5E') {

View on GitHub (pinned to 1b7edd19c4)