{"record":{"id":"a95b1ea63093218c","repo":"stanfordnlp/CoreNLP","slug":"chineseutils-normalize-warning-unmatched-high-sur","errorCode":null,"errorMessage":"ChineseUtils.normalize warning: unmatched high surrogate character U+","messagePattern":"ChineseUtils\\.normalize warning: unmatched high surrogate character U\\+","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/edu/stanford/nlp/trees/international/pennchinese/ChineseUtils.java","lineNumber":117,"sourceCode":"    if (ONLY_BMP) {\n      return normalizeBMP(in, ascii, spaceChar, midDot);\n    } else {\n      return normalizeUnicode(in, ascii, spaceChar, midDot);\n    }\n  }\n\n\n  private static String normalizeBMP(String in, int ascii, int spaceChar, int midDot) {\n    StringBuilder out = new StringBuilder();\n    int len = in.length();\n    for (int i = 0; i < len; i++) {\n      char cp = in.charAt(i);\n      if (Character.isHighSurrogate(cp)) {\n        if (i + 1 < len) {\n          log.warn(\"ChineseUtils.normalize warning: non-BMP codepoint U+\" +\n                  Integer.toHexString(Character.codePointAt(in, i)) + \" in \" + in);\n        } else {\n          log.warn(\"ChineseUtils.normalize warning: unmatched high surrogate character U+\" +\n                  Integer.toHexString(Character.codePointAt(in, i)) + \" in \" + in);\n        }\n      }\n      Character.UnicodeBlock cub = Character.UnicodeBlock.of(cp);\n      if (cub == Character.UnicodeBlock.PRIVATE_USE_AREA ||\n              cub == Character.UnicodeBlock.SUPPLEMENTARY_PRIVATE_USE_AREA_A ||\n              cub == Character.UnicodeBlock.SUPPLEMENTARY_PRIVATE_USE_AREA_B) {\n        EncodingPrintWriter.err.println(\"ChineseUtils.normalize warning: private use area codepoint U+\" + Integer.toHexString(cp) + \" in \" + in);\n      }\n      boolean delete = false;\n      switch (ascii) {\n        case LEAVE:\n          break;\n        case ASCII:\n          if (cp >= '\\uFF01' && cp <= '\\uFF5E') {\n            cp -= (0xFF00 - 0x0020);\n          }\n          break;","sourceCodeStart":99,"sourceCodeEnd":135,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/trees/international/pennchinese/ChineseUtils.java#L99-L135","documentation":"ChineseUtils.normalize (via normalizeBMP) logs a warning when the input ends with (or contains an unpaired) high surrogate — a lead surrogate U+D800–U+DBFF with no trailing surrogate after it. The string is malformed UTF-16; the lone surrogate cannot denote a real codepoint and is not normalized.","triggerScenarios":"Calling ChineseUtils.normalize on a string containing an unpaired high surrogate, typically produced by truncated multi-byte character decoding, bad charset conversion, or byte-level slicing in the middle of a 4-byte UTF-8 sequence decoded to UTF-16.","commonSituations":"Reading corpora with the wrong charset (e.g. decoding UTF-8 bytes as a fixed 2-byte charset); truncated files or network streams cut mid-character; manual substring/indexOf operations that split surrogate pairs.","solutions":["Fix the decoding step: read input as UTF-8 with CodingErrorAction.REPORT/REPLACE instead of whatever charset produced the lone surrogate","Re-encode the malformed string with CharsetDecoder set to REPLACE malformed input with U+FFFD before normalizing","Check for byte-level truncation: substring calls that split surrogate pairs (use codePoint-aware offsets)","Validate input with Character.isHighSurrogate/isSurrogatePair before calling normalize"],"exampleFix":"// before\nString norm = ChineseUtils.normalize(truncatedLine, false, false);\n// after\nCharsetDecoder dec = StandardCharsets.UTF_8.newDecoder().onMalformedInput(CodingErrorAction.REPLACE);\nString clean = dec.decode(ByteBuffer.wrap(bytes)).toString();\nString norm = ChineseUtils.normalize(clean, false, false);","handlingStrategy":"validation","validationCode":"static boolean hasLoneSurrogate(String s) {\n  for (int i = 0; i < s.length(); i++) {\n    if (Character.isHighSurrogate(s.charAt(i)) && (i + 1 >= s.length() || !Character.isLowSurrogate(s.charAt(i + 1)))) return true;\n  }\n  return false;\n}","typeGuard":"static String replaceMalformed(String s) {\n  return s.replaceAll(\"[\\\\uD800-\\\\uDBFF](?![\\\\uDC00-\\\\uDFFF])|(?<![\\\\uD800-\\\\uDBFF])[\\\\uDC00-\\\\uDFFF]\", \"\\\\uFFFD\");\n}","tryCatchPattern":null,"preventionTips":["Always decode files/streams with an explicit charset (UTF-8) and REPLACE/REPORT malformed input","Avoid cutting strings at arbitrary char offsets; use codePoint-aware boundaries","Detect truncated reads: file sizes and stream ends should land on character boundaries"],"tags":["nlp","unicode","encoding","surrogate","chinese"],"backgroundTag":"invalid-argument-format","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}