{"record":{"id":"0c1045aea0d5af0d","repo":"stanfordnlp/CoreNLP","slug":"chineseutils-normalize-warning-non-bmp-codepoint","errorCode":null,"errorMessage":"ChineseUtils.normalize warning: non-BMP codepoint U+","messagePattern":"ChineseUtils\\.normalize warning: non-BMP codepoint U\\+","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/edu/stanford/nlp/trees/international/pennchinese/ChineseUtils.java","lineNumber":114,"sourceCode":"        spaceChar < 0 || spaceChar > MAX_LEGAL) {\n      throw new IllegalArgumentException(\"ChineseUtils: Unknown parameter option\");\n    }\n    if (ONLY_BMP) {\n      return normalizeBMP(in, ascii, spaceChar, midDot);\n    } else {\n      return normalizeUnicode(in, ascii, spaceChar, midDot);\n    }\n  }\n\n\n  private static String normalizeBMP(String in, int ascii, int spaceChar, int midDot) {\n    StringBuilder out = new StringBuilder();\n    int len = in.length();\n    for (int i = 0; i < len; i++) {\n      char cp = in.charAt(i);\n      if (Character.isHighSurrogate(cp)) {\n        if (i + 1 < len) {\n          log.warn(\"ChineseUtils.normalize warning: non-BMP codepoint U+\" +\n                  Integer.toHexString(Character.codePointAt(in, i)) + \" in \" + in);\n        } else {\n          log.warn(\"ChineseUtils.normalize warning: unmatched high surrogate character U+\" +\n                  Integer.toHexString(Character.codePointAt(in, i)) + \" in \" + in);\n        }\n      }\n      Character.UnicodeBlock cub = Character.UnicodeBlock.of(cp);\n      if (cub == Character.UnicodeBlock.PRIVATE_USE_AREA ||\n              cub == Character.UnicodeBlock.SUPPLEMENTARY_PRIVATE_USE_AREA_A ||\n              cub == Character.UnicodeBlock.SUPPLEMENTARY_PRIVATE_USE_AREA_B) {\n        EncodingPrintWriter.err.println(\"ChineseUtils.normalize warning: private use area codepoint U+\" + Integer.toHexString(cp) + \" in \" + in);\n      }\n      boolean delete = false;\n      switch (ascii) {\n        case LEAVE:\n          break;\n        case ASCII:\n          if (cp >= '\\uFF01' && cp <= '\\uFF5E') {","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/trees/international/pennchinese/ChineseUtils.java#L96-L132","documentation":"ChineseUtils.normalize (via normalizeBMP) logs a warning when it encounters a valid high surrogate paired with a following code unit — i.e. an astral-plane (non-BMP) character above U+FFFF. Such codepoints fall outside the BMP ranges the Chinese normalization tables handle, so the character is not normalized and is warned about.","triggerScenarios":"Calling ChineseUtils.normalize on a string containing supplementary-plane characters such as rare CJK ideographs in Unicode Plane 2 (U+20000+), emoji, or other non-BMP text.","commonSituations":"Processing text files containing CJK Extension B/C/D characters common in classical Chinese corpora; mixing in emoji or symbols; corpus data converted to a wider Unicode version than the normalization tables expect.","solutions":["Strip or map non-BMP codepoints to a BMP equivalent before calling ChineseUtils.normalize","Pre-filter the input: reject or log-and-skip strings containing supplementary characters","Update/patch the normalization tables to cover the needed Plane-2 ideographs","Ensure the input encoding/Unicode version matches what the corpus expects; convert with ICU transcoding"],"exampleFix":"// before\nString norm = ChineseUtils.normalize(raw, false, false);\n// after\nString cleaned = raw.codePoints().filter(cp -> cp <= 0xFFFF).collect(StringBuilder::new, StringBuilder::appendCodePoint, StringBuilder::append).toString();\nString norm = ChineseUtils.normalize(cleaned, false, false);","handlingStrategy":"validation","validationCode":"static boolean isBmpOnly(String s) {\n  return s.codePoints().allMatch(cp -> cp <= 0xFFFF);\n}\nif (!isBmpOnly(input)) throw new IllegalArgumentException(\"non-BMP codepoint in Chinese input: \" + input);","typeGuard":"static String stripSupplementary(String s) {\n  return s.codePoints().filter(cp -> cp <= 0xFFFF)\n      .collect(StringBuilder::new, StringBuilder::appendCodePoint, StringBuilder::append).toString();\n}","tryCatchPattern":null,"preventionTips":["Normalize/strip supplementary-plane characters before Chinese text preprocessing","Pin the Unicode version of your corpus and tooling; Plane-2 CJK ideographs are the usual culprit","Sanity-check corpus files with a scanner for codepoints > U+FFFF"],"tags":["nlp","unicode","chinese","normalization"],"backgroundTag":"invalid-argument-value","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}