{"record":{"id":"cffec5e2ad95bdbb","repo":"stanfordnlp/CoreNLP","slug":"pretokenized-option-set-non-standard-annotators-l","errorCode":null,"errorMessage":"preTokenized option set: Non-standard annotators list, preTokenized may not work in this case.","messagePattern":"preTokenized option set: Non-standard annotators list, preTokenized may not work in this case\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/edu/stanford/nlp/pipeline/StanfordCoreNLP.java","lineNumber":253,"sourceCode":"    if (PropertiesUtils.getBool(this.properties, (\"preTokenized\"))) {\n      this.properties.setProperty(\"tokenize.whitespace\", \"true\");\n      this.properties.setProperty(\"ssplit.eolonly\", \"true\");\n      String oldAnnotators = this.properties.getProperty(\"annotators\").replaceAll(\"\\\\s+\", \"\");\n      String newAnnotators = oldAnnotators;\n      if (oldAnnotators != null && oldAnnotators.startsWith(\"cdc_tokenize\")) {\n        newAnnotators = \"tokenize,ssplit\" + oldAnnotators.substring(12,oldAnnotators.length());\n        logger.info(\"preTokenized option set: Changing annotators cdc_tokenize to tokenize,ssplit\");\n      } else if (oldAnnotators != null && oldAnnotators.startsWith(\"tokenize,ssplit,mwt\")) {\n        newAnnotators = \"tokenize,ssplit\" + oldAnnotators.substring(19,oldAnnotators.length());\n        logger.info(\"preTokenized option set: Changing annotators tokenize,ssplit,mwt to tokenize,ssplit\");\n      } else if (oldAnnotators != null && oldAnnotators.startsWith(\"tokenize,ssplit\")) {\n        logger.info(\"preTokenized option set: Annotators list starts with tokenize,ssplit, no change needed.\");\n      } else if (oldAnnotators != null && !oldAnnotators.contains(\"tokenize\") && !oldAnnotators.contains(\"mwt\")\n                 && !oldAnnotators.contains(\"ssplit\") && !oldAnnotators.contains(\"cdc_tokenize\")) {\n        logger.info(\"preTokenized option set: Adding tokenize,ssplit to beginning.\");\n        newAnnotators = \"tokenize,ssplit,\" + oldAnnotators;\n      } else {\n        logger.warn(\"preTokenized option set: Non-standard annotators list, preTokenized may not work in this case.\"); \n      }\n      this.properties.setProperty(\"annotators\", newAnnotators);\n    }\n\n    normalizeAnnotators(this.properties);\n\n    // cdm [2017]: constructAnnotatorPool (PropertiesUtils.getSignature) requires non-null Properties, so after properties setup\n    this.pool = annotatorPool != null ? annotatorPool : constructAnnotatorPool(props, getAnnotatorImplementations());\n\n    // Set threading\n    if (this.properties.containsKey(\"threads\")) {\n      ArgumentParser.threads = PropertiesUtils.getInt(this.properties, \"threads\");\n      this.availableProcessors = new Semaphore(ArgumentParser.threads);\n    } else {\n      this.availableProcessors = new Semaphore(1);\n    }\n\n    // now construct the annotators from the given properties in the given order","sourceCodeStart":235,"sourceCodeEnd":271,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/StanfordCoreNLP.java#L235-L271","documentation":"StanfordCoreNLP's constructor adjusts the annotators list when the preTokenized option is set. If the existing annotators string is non-standard (contains tokenize or mwt/ssplit/cdc_tokenize but not in the expected position), the pipeline cannot safely prepend tokenize,ssplit and warns that preTokenized may not work. Annotation may then re-tokenize or fail to respect your pre-tokenization.","triggerScenarios":"Passing -preTokenized true (or preTokenized property) while specifying a custom annotators string such as 'parse' alone or one that already includes tokenize but in an unusual combination, so the auto-fix branch does not apply.","commonSituations":"Users supplying their own annotators list plus preTokenized=true expecting tokens to be honored; pipelines built from properties files where annotators and preTokenized conflict.","solutions":["Remove the preTokenized option and include tokenize,ssplit explicitly in your annotators list","Or set annotators to start with tokenize,ssplit (e.g. 'tokenize,ssplit,parse') so the standard branch applies","If tokens come from elsewhere, use the whitespace/pretokenized tokenizers (tokenize.whitespace=true) instead of the preTokenized flag","Check the logged oldAnnotators value and align it with the supported combinations"],"exampleFix":"// before\nprops.setProperty(\"annotators\", \"parse\");\nprops.setProperty(\"preTokenized\", \"true\");\n// after\nprops.setProperty(\"annotators\", \"tokenize,ssplit,parse\");\nprops.setProperty(\"tokenize.whitespace\", \"true\");","handlingStrategy":"validation","validationCode":"String annotators = props.getProperty(\"annotators\", \"\");\nif (Boolean.parseBoolean(props.getProperty(\"preTokenized\", \"false\"))\n    && !annotators.startsWith(\"tokenize,ssplit\") && !annotators.isEmpty()) {\n  props.setProperty(\"annotators\", \"tokenize,ssplit,\" + annotators);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Do not combine preTokenized=true with custom annotators lists","Always list tokenize,ssplit explicitly when in doubt","Prefer tokenize.whitespace=true for pre-tokenized input","Review pipeline properties once at startup and log the effective annotators"],"tags":["nlp","configuration","java","pipeline"],"backgroundTag":"conflicting-config-options","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}