stanfordnlp/CoreNLP · error · RuntimeIOException

Unexpected input (many fields):

Error message

Unexpected input (many fields): 

What it means

CoNLLDocumentReaderAndWriter.makeCoreLabel parses each CoNLL line into a fixed number of whitespace-separated fields via a switch on the expected field count. A line with more fields than any configured format handles falls to default and throws RuntimeIOException('Unexpected input (many fields): ' + line).

Solutions

1. Trim or preprocess the file so each line has exactly the number of fields the configured format expects.
2. Pass the appropriate reader options/flags (e.g. InputRewriteRules / format flags) so makeCoreLabel handles the file's field count.
3. Verify columns are not accidentally split on internal spaces (quoted fields) inflating the field count.
4. Check you are using the correct DocumentReaderAndWriter for the dataset (CoNLL-2003 vs CoNLL-2012).

Example fix

// before (6 fields given, 5 expected)
Nike   nike   NNP   B-NP   ORG   extra
// after
Nike   nike   NNP   B-NP   ORG

// check field count of first data line against expected format
String[] first = Files.lines(Path.of(conllFile)).filter(l -> !l.isEmpty()).findFirst().orElse("").split("\\s+");
if (first.length > 5) throw new IllegalArgumentException("file has " + first.length + " fields; reader expects 5");

When it happens

Trigger: Reading a CoNLL-formatted file whose rows contain more tab/space-separated columns than the reader is configured for (e.g. a 6+ column CoNLL file when reader expects 5), at CoNLLDocumentReaderAndWriter.java:170 in processDocument.

Common situations: Feeding CoNLL-2012 or richer annotation files into a reader configured for basic CoNLL-2003 format; files with extra trailing columns; using the wrong flags (e.g. missing -readerOptions for format).

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Cannot find matching labelled span for
• CoreLabels required!
• entitySubclassify: unknown style:
• Error extracting labelled spans for column :
• format error in embeddings

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/25719ec9f69a2f6e. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/sequences/CoNLLDocumentReaderAndWriter.java:170

    case 4:
      wi.setWord(bits[0]);
      wi.setTag(bits[1]);
      wi.set(CoreAnnotations.ChunkAnnotation.class, bits[2]);
      wi.set(CoreAnnotations.AnswerAnnotation.class, bits[3]);
      break;
    case 5:
      if (flags.useLemmaAsWord) {
        wi.setWord(bits[1]);
      } else {
        wi.setWord(bits[0]);
      }
      wi.set(CoreAnnotations.LemmaAnnotation.class, bits[1]);
      wi.setTag(bits[2]);
      wi.set(CoreAnnotations.ChunkAnnotation.class, bits[3]);
      wi.set(CoreAnnotations.AnswerAnnotation.class, bits[4]);
      break;
    default:
      throw new RuntimeIOException("Unexpected input (many fields): " + line);
    }

    //Value annotation is used in a lot of place in corenlp so setting here as the word itself
    wi.set(CoreAnnotations.ValueAnnotation.class, wi.word());

    // The copy to GoldAnswerAnnotation is done before the recoding is done, and so it preserves the original coding.
    // This is important if the original coding is true, but the recoding is defective (like IOB2 to IO), since
    // it will allow correct evaluation later.
    wi.set(CoreAnnotations.GoldAnswerAnnotation.class, wi.get(CoreAnnotations.AnswerAnnotation.class));
    return wi;
  }

  /** Return the coding scheme to IOB1 coding, regardless of what was used
   *  internally (unless retainEntitySubclassification is set).
   *  This is useful for scoring against CoNLL test output.
   *
   *  @param tokens List of tokens in some NER encoding
   */

View on GitHub (pinned to 1b7edd19c4)