apache/beam · error · IOException

Cannot create reader as source is not split yet.

Error message

Cannot create reader as source is not split yet.

What it means

For a HadoopInputFormatBoundedSource created by split(), createReader() requires the inputSplit field to be set. Reading from the un-split root source is not supported, so it throws this IOException when inputSplit is null.

Solutions

1. Always let Beam's runner split the source; do not call createReader() on the root source yourself
2. If testing manually, call source.split(desiredBundleSizeBytes, options) first and create a reader from a sub-source
3. Call computeSplitsIfNecessary() via split() before reader creation

Example fix

// before
BoundedReader<KV<K,V>> r = source.createReader(options);
// after
for (BoundedSource<KV<K,V>> s : source.split(64L<<20, options)) {
  BoundedReader<KV<K,V>> r = s.createReader(options);
}

static <T> boolean isSplitted(BoundedSource<T> src) {
  return src instanceof HadoopFormatIO.HadoopInputFormatBoundedSource<?, ?>
      && ((HadoopFormatIO.HadoopInputFormatBoundedSource<?, ?>) src).getClass();
}
// Prefer: only create readers from sources returned by split().

try {
  BoundedReader<KV<K,V>> r = subSource.createReader(options);
} catch (IOException e) {
  if (e.getMessage().contains("source is not split yet")) {
    throw new IllegalStateException("Must call split() before createReader()");
  }
  throw e;
}

When it happens

Trigger: Calling createReader() directly on the source produced by HadoopFormatIO.read() without first calling split() (or using a source that skipped computeSplitsIfNecessary).

Common situations: Custom runner code or tests that construct a BoundedSource and call createReader() without the split step; pipelines executed on runners that bypass Beam's split protocol.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Cannot provide because does not implement the interface
• Could not read because the thread got interrupted while…
• Error in computing the fractions consumed as…
• Support for move options is not yet implemented.
• Unable to create target directory

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/971b09d9272c5500. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java:867

    InputFormat<?, ?> getInputFormat() {
      return inputFormatObj;
    }

    @VisibleForTesting
    void setInputFormatObj(InputFormat<?, ?> inputFormatObj) {
      this.inputFormatObj = inputFormatObj;
    }

    @Override
    public Coder<KV<K, V>> getOutputCoder() {
      return KvCoder.of(keyCoder, valueCoder);
    }

    @Override
    public BoundedReader<KV<K, V>> createReader(PipelineOptions options) throws IOException {
      this.validate();
      if (inputSplit == null) {
        throw new IOException("Cannot create reader as source is not split yet.");
      } else {
        createInputFormatInstance();
        return new HadoopInputFormatReader<>(
            this,
            keyTranslationFunction,
            valueTranslationFunction,
            inputSplit,
            inputFormatObj,
            taskAttemptContext);
      }
    }

    /**
     * BoundedReader for Hadoop InputFormat source.
     *
     * @param <T1> Type of keys RecordReader emits.
     * @param <T2> Type of values RecordReader emits.
     */

View on GitHub (pinned to 12126d8942)