{"record":{"id":"c3e6f4591a400469","repo":"stanfordnlp/CoreNLP","slug":"expected-hasoffsets-from-the-documentpreprocessor","errorCode":null,"errorMessage":"Expected HasOffsets from the DocumentPreprocessor","messagePattern":"Expected HasOffsets from the DocumentPreprocessor","errorType":"exception","errorClass":"ClassCastException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/parser/ui/ParserPanel.java","lineNumber":237,"sourceCode":"   * Finds the nearest delimiter starting from index start. If <tt>seekDir</tt>\n   * is SEEK_FORWARD, finds the nearest delimiter after start.  Else, if it is\n   * SEEK_BACK, finds the nearest delimiter before start.\n   */\n  private int nearestDelimiter(String text, int start, int seekDir) {\n    if (seekDir != SEEK_BACK && seekDir != SEEK_FORWARD) {\n      throw new IllegalArgumentException(\"Unknown seek direction \" +\n                                         seekDir);\n    }\n    StringReader reader = new StringReader(text);\n    DocumentPreprocessor processor = new DocumentPreprocessor(reader);\n    TokenizerFactory<? extends HasWord> tf = tlp.getTokenizerFactory();\n    processor.setTokenizerFactory(tf);\n    List<Integer> boundaries = new ArrayList<>();\n    for (List<HasWord> sentence : processor) {\n      if (sentence.size() == 0)\n        continue;\n      if (!(sentence.get(0) instanceof HasOffset)) {\n        throw new ClassCastException(\"Expected HasOffsets from the \" +\n                                     \"DocumentPreprocessor\");\n      }\n      if (boundaries.size() == 0) {\n        boundaries.add(0);\n      } else {\n        HasOffset first = (HasOffset) sentence.get(0);\n        boundaries.add(first.beginPosition());\n      }\n    }\n    boundaries.add(text.length());\n    for (int i = 0; i < boundaries.size() - 1; ++i) {\n      if (boundaries.get(i) <= start && start < boundaries.get(i + 1)) {\n        if (seekDir == SEEK_BACK) {\n          return boundaries.get(i) - 1;\n        } else if (seekDir == SEEK_FORWARD) {\n          return boundaries.get(i + 1) - 1;\n        }\n      }","sourceCodeStart":219,"sourceCodeEnd":255,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/parser/ui/ParserPanel.java#L219-L255","documentation":"nearestDelimiter iterates sentences from a DocumentPreprocessor and requires tokens implementing HasOffset so byte/char offsets of sentence boundaries can be tracked. If the first token of a sentence is not a HasOffset, it throws ClassCastException, meaning the configured tokenizer produced plain HasWord tokens without position info.","triggerScenarios":"Calling highlightSentence/nearestDelimiter on ParserPanel when the configured tokenizer factory (from the loaded treebank language pack) returns tokens that do not implement HasOffset.","commonSituations":"Using a custom TokenizerFactory or a TLP whose tokenizer lacks offset propagation; swapping in a custom tokenizer for the parsing GUI.","solutions":["Use the default tokenizer factory from the panel's treebank language pack (tlp.getTokenizerFactory()) rather than a custom one","Use a tokenizer that produces CoreLabel or other HasOffset-implementing tokens","If a custom factory is required, wrap/convert tokens to CoreLabel carrying beginPosition/endPosition"],"exampleFix":"// before\nTokenizerFactory<? extends HasWord> tf = MyPlainTokenizer.factory();\nprocessor.setTokenizerFactory(tf);\n// after\nTokenizerFactory<? extends HasWord> tf =\n    new PTBTokenizerFactory<>(true, false); // emits CoreLabel with offsets\nprocessor.setTokenizerFactory(tf);","handlingStrategy":"type-guard","validationCode":"TokenizerFactory<? extends HasWord> tf = tlp.getTokenizerFactory();\nList<HasWord> toks = tf.getTokenizer(new StringReader(text)).tokenize();\nif (!(toks.get(0) instanceof HasOffset))\n  throw new IllegalStateException(\"Tokenizer lacks offsets\");","typeGuard":"boolean hasOffsets(List<? extends HasWord> sentence) {\n  return !sentence.isEmpty() && sentence.get(0) instanceof HasOffset;\n}","tryCatchPattern":"try {\n  iterateProcessor(processor);\n} catch (ClassCastException e) {\n  if (e.getMessage().contains(\"HasOffsets\")) {\n    log.warn(\"Falling back to default tokenizer\");\n  } else throw e;\n}","preventionTips":["Use CoreLabel-producing tokenizers (PTBTokenizer)","Avoid custom TokenizerFactory without offsets","Keep the default TLP tokenizer"],"tags":["tokenizer","offsets","type-mismatch","java"],"backgroundTag":"type-mismatch","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}