preTokenized option set: Non-standard annotators list…

Error message

preTokenized option set: Non-standard annotators list, preTokenized may not work in this case.

What it means

StanfordCoreNLP's constructor adjusts the annotators list when the preTokenized option is set. If the existing annotators string is non-standard (contains tokenize or mwt/ssplit/cdc_tokenize but not in the expected position), the pipeline cannot safely prepend tokenize,ssplit and warns that preTokenized may not work. Annotation may then re-tokenize or fail to respect your pre-tokenization.

Solutions

Remove the preTokenized option and include tokenize,ssplit explicitly in your annotators list
Or set annotators to start with tokenize,ssplit (e.g. 'tokenize,ssplit,parse') so the standard branch applies
If tokens come from elsewhere, use the whitespace/pretokenized tokenizers (tokenize.whitespace=true) instead of the preTokenized flag
Check the logged oldAnnotators value and align it with the supported combinations

Example fix

// before
props.setProperty("annotators", "parse");
props.setProperty("preTokenized", "true");
// after
props.setProperty("annotators", "tokenize,ssplit,parse");
props.setProperty("tokenize.whitespace", "true");

Defensive patterns

Strategy: validation

Validate before calling

String annotators = props.getProperty("annotators", "");
if (Boolean.parseBoolean(props.getProperty("preTokenized", "false"))
    && !annotators.startsWith("tokenize,ssplit") && !annotators.isEmpty()) {
  props.setProperty("annotators", "tokenize,ssplit," + annotators);
}

Prevention

Do not combine preTokenized=true with custom annotators lists
Always list tokenize,ssplit explicitly when in doubt
Prefer tokenize.whitespace=true for pre-tokenized input
Review pipeline properties once at startup and log the effective annotators

When it happens

Trigger: Passing -preTokenized true (or preTokenized property) while specifying a custom annotators string such as 'parse' alone or one that already includes tokenize but in an unusual combination, so the auto-fix branch does not apply.

Common situations: Users supplying their own annotators list plus preTokenized=true expecting tokens to be honored; pipelines built from properties files where annotators and preTokenized conflict.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/cffec5e2ad95bdbb. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/StanfordCoreNLP.java:253

    if (PropertiesUtils.getBool(this.properties, ("preTokenized"))) {
      this.properties.setProperty("tokenize.whitespace", "true");
      this.properties.setProperty("ssplit.eolonly", "true");
      String oldAnnotators = this.properties.getProperty("annotators").replaceAll("\\s+", "");
      String newAnnotators = oldAnnotators;
      if (oldAnnotators != null && oldAnnotators.startsWith("cdc_tokenize")) {
        newAnnotators = "tokenize,ssplit" + oldAnnotators.substring(12,oldAnnotators.length());
        logger.info("preTokenized option set: Changing annotators cdc_tokenize to tokenize,ssplit");
      } else if (oldAnnotators != null && oldAnnotators.startsWith("tokenize,ssplit,mwt")) {
        newAnnotators = "tokenize,ssplit" + oldAnnotators.substring(19,oldAnnotators.length());
        logger.info("preTokenized option set: Changing annotators tokenize,ssplit,mwt to tokenize,ssplit");
      } else if (oldAnnotators != null && oldAnnotators.startsWith("tokenize,ssplit")) {
        logger.info("preTokenized option set: Annotators list starts with tokenize,ssplit, no change needed.");
      } else if (oldAnnotators != null && !oldAnnotators.contains("tokenize") && !oldAnnotators.contains("mwt")
                 && !oldAnnotators.contains("ssplit") && !oldAnnotators.contains("cdc_tokenize")) {
        logger.info("preTokenized option set: Adding tokenize,ssplit to beginning.");
        newAnnotators = "tokenize,ssplit," + oldAnnotators;
      } else {
        logger.warn("preTokenized option set: Non-standard annotators list, preTokenized may not work in this case."); 
      }
      this.properties.setProperty("annotators", newAnnotators);
    }

    normalizeAnnotators(this.properties);

    // cdm [2017]: constructAnnotatorPool (PropertiesUtils.getSignature) requires non-null Properties, so after properties setup
    this.pool = annotatorPool != null ? annotatorPool : constructAnnotatorPool(props, getAnnotatorImplementations());

    // Set threading
    if (this.properties.containsKey("threads")) {
      ArgumentParser.threads = PropertiesUtils.getInt(this.properties, "threads");
      this.availableProcessors = new Semaphore(ArgumentParser.threads);
    } else {
      this.availableProcessors = new Semaphore(1);
    }

    // now construct the annotators from the given properties in the given order

View on GitHub (pinned to 1b7edd19c4)