What causes "adjustFinalToken: Unexpected final char: |${last}| (${(int) last})" in stanfordnlp/CoreNLP?

Ensure input text ends with a regular space before annotation, or normalize trailing whitespace.

stanfordnlp/CoreNLP · error · IllegalArgumentException

adjustFinalToken: Unexpected final char: |

Error message

adjustFinalToken: Unexpected final char: |${last}| (${(int) last})

What it means

adjustFinalToken assumes the last token's AfterAnnotation ends with a single space (as produced by the tokenizers it fixes up). If the trailing after-text ends in any other character, it throws IllegalArgumentException including the char and its code point, since it cannot cleanly strip the expected trailing space.

Solutions

Ensure input text ends with a regular space before annotation, or normalize trailing whitespace.
Pre-trim and re-append a single space: text = text.trim() + " ".
Use a tokenizer type that does not invoke adjustFinalToken if your text ends unusually.
Patch AfterAnnotation via a postprocessor instead of altering tokenizer internals.

Example fix

// before
String text = "Hello world\n";
// after
String text = ("Hello world\n").trim() + " ";

Defensive patterns

Strategy: validation

Validate before calling

String text = ann.get(CoreAnnotations.TextAnnotation.class);
if (text != null && !text.isEmpty() && !text.endsWith(" ")) {
  ann.set(CoreAnnotations.TextAnnotation.class, text.trim() + " ");
}

Type guard

boolean endsWithPlainSpace(String s) { return s != null && !s.isEmpty() && s.charAt(s.length()-1) == ' '; }

Try / catch

try { annotator.annotate(ann); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith("adjustFinalToken:")) { fixTrailingSpace(ann); annotator.annotate(ann); } else throw e; }

Prevention

Normalize trailing whitespace to a single space before annotation
Avoid custom token factories that rewrite AfterAnnotation
Add a pre-annotate sanitizer that enforces the trailing-space invariant

When it happens

Trigger: Calling annotate with a tokenizer whose final token's after-text does not end in a space — e.g. text ending with a newline, tab, or non-whitespace character, or a custom tokenizer/factory producing non-standard after text.

Common situations: Feeding text that ends in '\n' or EOF without trailing space into tokenizers like the Spanish/other analytic tokenizers that route through adjustFinalToken; custom CoreLabelTokenFactory altering AfterAnnotation.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/f4ab0df14cf3c503. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java:383

    // runs afoul of two character Windows newlines...
    for (CoreLabel token : tokensList) {
      if (token.word().equals(AbstractTokenizer.NEWLINE_TOKEN))
        token.set(CoreAnnotations.IsNewlineAnnotation.class, true);
      else
        token.set(CoreAnnotations.IsNewlineAnnotation.class, false);
    }
  }

  public static void adjustFinalToken(List<CoreLabel> tokens) {
    if (tokens == null || tokens.size() == 0) {
      return;
    }
    CoreLabel finalToken = tokens.get(tokens.size() - 1);
    String finalTokenAfter = finalToken.get(CoreAnnotations.AfterAnnotation.class);
    if (finalTokenAfter != null && finalTokenAfter.length() > 0) {
      char last = finalTokenAfter.charAt(finalTokenAfter.length() - 1);
      if (last != ' ') {
        throw new IllegalArgumentException("adjustFinalToken: Unexpected final char: |" + last + "| (" + (int) last + ')');
      }
      finalTokenAfter = finalTokenAfter.substring(0, finalTokenAfter.length() - 1);
      finalToken.set(CoreAnnotations.AfterAnnotation.class, finalTokenAfter);
    }
  }

  /**
   * Does the actual work of splitting TextAnnotation into CoreLabels,
   * which are then attached to the TokensAnnotation.
   */
  @Override
  public void annotate(Annotation annotation) {
    if (VERBOSE) {
      log.info("Beginning tokenization");
    }

    if (cdcAnnotator != null) {
      cdcAnnotator.annotate(annotation);

View on GitHub (pinned to 1b7edd19c4)