{"record":{"id":"df9a6332318e495d","repo":"stanfordnlp/CoreNLP","slug":"unable-to-process-annotators-annotators","errorCode":null,"errorMessage":"Unable to process annotators ${annotators}","messagePattern":"Unable to process annotators (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/StanfordCoreNLP.java","lineNumber":356,"sourceCode":"   * In such a case, we remove the cleanxml from the annotators and set\n   * the tokenize.cleanxml option instead\n   */\n  static void unifyTokenizeProperty(Properties properties, String property, String option) {\n    String annotators = properties.getProperty(\"annotators\", \"\");\n    int tokenize = annotators.indexOf(STANFORD_TOKENIZE);\n    int unwanted = annotators.indexOf(property);\n\n    if (unwanted >= 0 && tokenize >= 0) {\n      if (option != null) {\n        properties.setProperty(option, \"true\");\n      }\n      int comma = annotators.indexOf(\",\", unwanted);\n      if (comma >= 0) {\n        annotators = annotators.substring(0, unwanted) + annotators.substring(comma+1);\n      } else {\n        comma = annotators.lastIndexOf(\",\");\n        if (comma < 0) {\n          throw new IllegalArgumentException(\"Unable to process annotators \" + annotators);\n        }\n        annotators = annotators.substring(0, comma);\n      }\n      if (option != null) {\n        logger.debug(property + \" can now be triggered as an option to tokenize rather than a separate annotator via \" + option + \"=true\");\n      } else {\n        logger.debug(property + \" is now included as part of the tokenize annotator by default\");\n      }\n      logger.debug(\"Updating annotators from \" + properties.getProperty(\"annotators\") + \" to \" + annotators);\n      properties.setProperty(\"annotators\", annotators);\n    }\n  }\n\n  //\n  // @Override-able methods to change pipeline behavior\n  //\n\n  /**","sourceCodeStart":338,"sourceCodeEnd":374,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/StanfordCoreNLP.java#L338-L374","documentation":"In unifyTokenizeProperty (used by normalizeAnnotators), CoreNLP folds legacy 'whitespace'/'ssplit'-style options into the 'tokenize' annotator options. When it strips an unwanted comma-separated option and cannot find a remaining comma to splice around, it cannot rewrite the annotator list and throws IllegalArgumentException('Unable to process annotators ...').","triggerScenarios":"Setting properties like 'tokenize.whitespace=true' alongside an annotators list where the corresponding option annotator appears as the only comma-separated element (no surrounding commas), so the rewrite logic runs out of commas and fails.","commonSituations":"Migrating old pipelines (annotators=tokenize,whitespace,ssplit patterns or a lone 'whitespace' entry) to modern CoreNLP; programmatic construction of the annotators string with trailing/malformed commas.","solutions":["Rewrite the annotators list to use modern tokenize options instead of the separate option annotator: annotators=tokenize,ssplit with tokenize.whitespace=true.","Remove the deprecated option annotator name from the annotators property entirely.","Ensure the annotators string is well-formed (no stray/duplicate commas) before pipeline construction."],"exampleFix":"// before\nprops.setProperty(\"annotators\", \"tokenize,whitespace,ssplit\");\nprops.setProperty(\"tokenize.whitespace\", \"true\");\n// after\nprops.setProperty(\"annotators\", \"tokenize,ssplit\");\nprops.setProperty(\"tokenize.whitespace\", \"true\");","handlingStrategy":"validation","validationCode":"String annotators = props.getProperty(\"annotators\", \"\");\nif (annotators.contains(\"whitespace\") && props.containsKey(\"tokenize.whitespace\")) {\n  props.setProperty(\"annotators\", annotators.replace(\",whitespace\", \"\").replace(\"whitespace,\", \"\").replace(\"whitespace\", \"\"));\n}","typeGuard":null,"tryCatchPattern":"try {\n  pipeline = new StanfordCoreNLP(props);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().startsWith(\"Unable to process annotators\")) {\n    props.setProperty(\"annotators\", \"tokenize,ssplit\"); // drop legacy option annotators\n    pipeline = new StanfordCoreNLP(props);\n  } else throw e;\n}","preventionTips":["Drop legacy option annotators (e.g. 'whitespace') and use tokenize.* properties instead.","Keep the annotators string a clean comma-separated list with no stray commas.","When upgrading CoreNLP, migrate old pipelines per the tokenize option migration notes."],"tags":["corenlp","pipeline","configuration","tokenization"],"backgroundTag":"invalid-config-value","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}