What causes "Expected HasOffsets from the DocumentPreprocessor" in stanfordnlp/CoreNLP?

Use the default tokenizer factory from the panel's treebank language pack (tlp.getTokenizerFactory()) rather than a custom one

stanfordnlp/CoreNLP · error · ClassCastException

Expected HasOffsets from the DocumentPreprocessor

Error message

Expected HasOffsets from the DocumentPreprocessor

What it means

nearestDelimiter iterates sentences from a DocumentPreprocessor and requires tokens implementing HasOffset so byte/char offsets of sentence boundaries can be tracked. If the first token of a sentence is not a HasOffset, it throws ClassCastException, meaning the configured tokenizer produced plain HasWord tokens without position info.

Solutions

Use the default tokenizer factory from the panel's treebank language pack (tlp.getTokenizerFactory()) rather than a custom one
Use a tokenizer that produces CoreLabel or other HasOffset-implementing tokens
If a custom factory is required, wrap/convert tokens to CoreLabel carrying beginPosition/endPosition

Example fix

// before
TokenizerFactory<? extends HasWord> tf = MyPlainTokenizer.factory();
processor.setTokenizerFactory(tf);
// after
TokenizerFactory<? extends HasWord> tf =
    new PTBTokenizerFactory<>(true, false); // emits CoreLabel with offsets
processor.setTokenizerFactory(tf);

Defensive patterns

Strategy: type-guard

Validate before calling

TokenizerFactory<? extends HasWord> tf = tlp.getTokenizerFactory();
List<HasWord> toks = tf.getTokenizer(new StringReader(text)).tokenize();
if (!(toks.get(0) instanceof HasOffset))
  throw new IllegalStateException("Tokenizer lacks offsets");

Type guard

boolean hasOffsets(List<? extends HasWord> sentence) {
  return !sentence.isEmpty() && sentence.get(0) instanceof HasOffset;
}

Try / catch

try {
  iterateProcessor(processor);
} catch (ClassCastException e) {
  if (e.getMessage().contains("HasOffsets")) {
    log.warn("Falling back to default tokenizer");
  } else throw e;
}

Prevention

Use CoreLabel-producing tokenizers (PTBTokenizer)
Avoid custom TokenizerFactory without offsets
Keep the default TLP tokenizer

When it happens

Trigger: Calling highlightSentence/nearestDelimiter on ParserPanel when the configured tokenizer factory (from the loaded treebank language pack) returns tokens that do not implement HasOffset.

Common situations: Using a custom TokenizerFactory or a TLP whose tokenizer lacks offset propagation; swapping in a custom tokenizer for the parsing GUI.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/c3e6f4591a400469. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/parser/ui/ParserPanel.java:237

   * Finds the nearest delimiter starting from index start. If <tt>seekDir</tt>
   * is SEEK_FORWARD, finds the nearest delimiter after start.  Else, if it is
   * SEEK_BACK, finds the nearest delimiter before start.
   */
  private int nearestDelimiter(String text, int start, int seekDir) {
    if (seekDir != SEEK_BACK && seekDir != SEEK_FORWARD) {
      throw new IllegalArgumentException("Unknown seek direction " +
                                         seekDir);
    }
    StringReader reader = new StringReader(text);
    DocumentPreprocessor processor = new DocumentPreprocessor(reader);
    TokenizerFactory<? extends HasWord> tf = tlp.getTokenizerFactory();
    processor.setTokenizerFactory(tf);
    List<Integer> boundaries = new ArrayList<>();
    for (List<HasWord> sentence : processor) {
      if (sentence.size() == 0)
        continue;
      if (!(sentence.get(0) instanceof HasOffset)) {
        throw new ClassCastException("Expected HasOffsets from the " +
                                     "DocumentPreprocessor");
      }
      if (boundaries.size() == 0) {
        boundaries.add(0);
      } else {
        HasOffset first = (HasOffset) sentence.get(0);
        boundaries.add(first.beginPosition());
      }
    }
    boundaries.add(text.length());
    for (int i = 0; i < boundaries.size() - 1; ++i) {
      if (boundaries.get(i) <= start && start < boundaries.get(i + 1)) {
        if (seekDir == SEEK_BACK) {
          return boundaries.get(i) - 1;
        } else if (seekDir == SEEK_FORWARD) {
          return boundaries.get(i + 1) - 1;
        }
      }

View on GitHub (pinned to 1b7edd19c4)