ChineseUtils.normalize warning: unmatched high surrogate…

Error message

ChineseUtils.normalize warning: unmatched high surrogate character U+

What it means

ChineseUtils.normalize (via normalizeBMP) logs a warning when the input ends with (or contains an unpaired) high surrogate — a lead surrogate U+D800–U+DBFF with no trailing surrogate after it. The string is malformed UTF-16; the lone surrogate cannot denote a real codepoint and is not normalized.

Solutions

Fix the decoding step: read input as UTF-8 with CodingErrorAction.REPORT/REPLACE instead of whatever charset produced the lone surrogate
Re-encode the malformed string with CharsetDecoder set to REPLACE malformed input with U+FFFD before normalizing
Check for byte-level truncation: substring calls that split surrogate pairs (use codePoint-aware offsets)
Validate input with Character.isHighSurrogate/isSurrogatePair before calling normalize

Example fix

// before
String norm = ChineseUtils.normalize(truncatedLine, false, false);
// after
CharsetDecoder dec = StandardCharsets.UTF_8.newDecoder().onMalformedInput(CodingErrorAction.REPLACE);
String clean = dec.decode(ByteBuffer.wrap(bytes)).toString();
String norm = ChineseUtils.normalize(clean, false, false);

Defensive patterns

Strategy: validation

Validate before calling

static boolean hasLoneSurrogate(String s) {
  for (int i = 0; i < s.length(); i++) {
    if (Character.isHighSurrogate(s.charAt(i)) && (i + 1 >= s.length() || !Character.isLowSurrogate(s.charAt(i + 1)))) return true;
  }
  return false;
}

Type guard

static String replaceMalformed(String s) {
  return s.replaceAll("[\\uD800-\\uDBFF](?![\\uDC00-\\uDFFF])|(?<![\\uD800-\\uDBFF])[\\uDC00-\\uDFFF]", "\\uFFFD");
}

Prevention

Always decode files/streams with an explicit charset (UTF-8) and REPLACE/REPORT malformed input
Avoid cutting strings at arbitrary char offsets; use codePoint-aware boundaries
Detect truncated reads: file sizes and stream ends should land on character boundaries

When it happens

Trigger: Calling ChineseUtils.normalize on a string containing an unpaired high surrogate, typically produced by truncated multi-byte character decoding, bad charset conversion, or byte-level slicing in the middle of a 4-byte UTF-8 sequence decoded to UTF-16.

Common situations: Reading corpora with the wrong charset (e.g. decoding UTF-8 bytes as a fixed 2-byte charset); truncated files or network streams cut mid-character; manual substring/indexOf operations that split surrogate pairs.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/a95b1ea63093218c. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/trees/international/pennchinese/ChineseUtils.java:117

    if (ONLY_BMP) {
      return normalizeBMP(in, ascii, spaceChar, midDot);
    } else {
      return normalizeUnicode(in, ascii, spaceChar, midDot);
    }
  }


  private static String normalizeBMP(String in, int ascii, int spaceChar, int midDot) {
    StringBuilder out = new StringBuilder();
    int len = in.length();
    for (int i = 0; i < len; i++) {
      char cp = in.charAt(i);
      if (Character.isHighSurrogate(cp)) {
        if (i + 1 < len) {
          log.warn("ChineseUtils.normalize warning: non-BMP codepoint U+" +
                  Integer.toHexString(Character.codePointAt(in, i)) + " in " + in);
        } else {
          log.warn("ChineseUtils.normalize warning: unmatched high surrogate character U+" +
                  Integer.toHexString(Character.codePointAt(in, i)) + " in " + in);
        }
      }
      Character.UnicodeBlock cub = Character.UnicodeBlock.of(cp);
      if (cub == Character.UnicodeBlock.PRIVATE_USE_AREA ||
              cub == Character.UnicodeBlock.SUPPLEMENTARY_PRIVATE_USE_AREA_A ||
              cub == Character.UnicodeBlock.SUPPLEMENTARY_PRIVATE_USE_AREA_B) {
        EncodingPrintWriter.err.println("ChineseUtils.normalize warning: private use area codepoint U+" + Integer.toHexString(cp) + " in " + in);
      }
      boolean delete = false;
      switch (ascii) {
        case LEAVE:
          break;
        case ASCII:
          if (cp >= '\uFF01' && cp <= '\uFF5E') {
            cp -= (0xFF00 - 0x0020);
          }
          break;

View on GitHub (pinned to 1b7edd19c4)