{"record":{"id":"f4ab0df14cf3c503","repo":"stanfordnlp/CoreNLP","slug":"adjustfinaltoken-unexpected-final-char-last","errorCode":null,"errorMessage":"adjustFinalToken: Unexpected final char: |${last}| (${(int) last})","messagePattern":"adjustFinalToken: Unexpected final char: \\|(.+?)\\| \\((.+?)\\)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java","lineNumber":383,"sourceCode":"    // runs afoul of two character Windows newlines...\n    for (CoreLabel token : tokensList) {\n      if (token.word().equals(AbstractTokenizer.NEWLINE_TOKEN))\n        token.set(CoreAnnotations.IsNewlineAnnotation.class, true);\n      else\n        token.set(CoreAnnotations.IsNewlineAnnotation.class, false);\n    }\n  }\n\n  public static void adjustFinalToken(List<CoreLabel> tokens) {\n    if (tokens == null || tokens.size() == 0) {\n      return;\n    }\n    CoreLabel finalToken = tokens.get(tokens.size() - 1);\n    String finalTokenAfter = finalToken.get(CoreAnnotations.AfterAnnotation.class);\n    if (finalTokenAfter != null && finalTokenAfter.length() > 0) {\n      char last = finalTokenAfter.charAt(finalTokenAfter.length() - 1);\n      if (last != ' ') {\n        throw new IllegalArgumentException(\"adjustFinalToken: Unexpected final char: |\" + last + \"| (\" + (int) last + ')');\n      }\n      finalTokenAfter = finalTokenAfter.substring(0, finalTokenAfter.length() - 1);\n      finalToken.set(CoreAnnotations.AfterAnnotation.class, finalTokenAfter);\n    }\n  }\n\n  /**\n   * Does the actual work of splitting TextAnnotation into CoreLabels,\n   * which are then attached to the TokensAnnotation.\n   */\n  @Override\n  public void annotate(Annotation annotation) {\n    if (VERBOSE) {\n      log.info(\"Beginning tokenization\");\n    }\n\n    if (cdcAnnotator != null) {\n      cdcAnnotator.annotate(annotation);","sourceCodeStart":365,"sourceCodeEnd":401,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java#L365-L401","documentation":"adjustFinalToken assumes the last token's AfterAnnotation ends with a single space (as produced by the tokenizers it fixes up). If the trailing after-text ends in any other character, it throws IllegalArgumentException including the char and its code point, since it cannot cleanly strip the expected trailing space.","triggerScenarios":"Calling annotate with a tokenizer whose final token's after-text does not end in a space — e.g. text ending with a newline, tab, or non-whitespace character, or a custom tokenizer/factory producing non-standard after text.","commonSituations":"Feeding text that ends in '\\n' or EOF without trailing space into tokenizers like the Spanish/other analytic tokenizers that route through adjustFinalToken; custom CoreLabelTokenFactory altering AfterAnnotation.","solutions":["Ensure input text ends with a regular space before annotation, or normalize trailing whitespace.","Pre-trim and re-append a single space: text = text.trim() + \" \".","Use a tokenizer type that does not invoke adjustFinalToken if your text ends unusually.","Patch AfterAnnotation via a postprocessor instead of altering tokenizer internals."],"exampleFix":"// before\nString text = \"Hello world\\n\";\n// after\nString text = (\"Hello world\\n\").trim() + \" \";","handlingStrategy":"validation","validationCode":"String text = ann.get(CoreAnnotations.TextAnnotation.class);\nif (text != null && !text.isEmpty() && !text.endsWith(\" \")) {\n  ann.set(CoreAnnotations.TextAnnotation.class, text.trim() + \" \");\n}","typeGuard":"boolean endsWithPlainSpace(String s) { return s != null && !s.isEmpty() && s.charAt(s.length()-1) == ' '; }","tryCatchPattern":"try { annotator.annotate(ann); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith(\"adjustFinalToken:\")) { fixTrailingSpace(ann); annotator.annotate(ann); } else throw e; }","preventionTips":["Normalize trailing whitespace to a single space before annotation","Avoid custom token factories that rewrite AfterAnnotation","Add a pre-annotate sanitizer that enforces the trailing-space invariant"],"tags":["java","tokenizer","whitespace","invariant","corenlp"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}