apache/beam · error · RuntimeException

Error determining if %s allows dynamic splitting

Error message

Error determining if %s allows dynamic splitting

What it means

FileBasedReader.allowsDynamicSplitting() wraps any exception from getCurrentSource().isSplittable() into a RuntimeException 'Error determining if %s allows dynamic splitting'. It indicates the reader could not query its source's splittability — usually a filesystem/IO failure while resolving the current source.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java:562

    /**
     * Closes any {@link ReadableByteChannel} created for the current reader. This implementation is
     * idempotent. Any {@code close()} method introduced by a subclass must be idempotent and must
     * call the {@code close()} method in the {@code FileBasedReader}.
     */
    @Override
    public void close() throws IOException {
      if (channel != null) {
        channel.close();
      }
    }

    @Override
    public boolean allowsDynamicSplitting() {
      try {
        return getCurrentSource().isSplittable();
      } catch (Exception e) {
        throw new RuntimeException(
            String.format("Error determining if %s allows dynamic splitting", this), e);
      }
    }

    /**
     * Performs any initialization of the subclass of {@code FileBasedReader} that involves IO
     * operations. Will only be invoked once and before that invocation the base class will seek the
     * channel to the source's starting offset.
     *
     * <p>Provided {@link ReadableByteChannel} is for the file represented by the source of this
     * reader. Subclass may use the {@code channel} to build a higher level IO abstraction, e.g., a
     * BufferedReader or an XML parser.
     *
     * <p>If the corresponding source is for a subrange of a file, {@code channel} is guaranteed to
     * be an instance of the type {@link SeekableByteChannel}.
     *
     * <p>After this method is invoked the base class will not be reading data from the channel or
     * adjusting the position of the channel. But the base class is responsible for properly closing

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the chained cause for the underlying IO error and fix storage connectivity.
Retry the pipeline; transient storage failures often resolve.
If you implemented a custom FileBasedSource, make isSplittable() not throw.
Disable/avoid dynamic splitting scenarios by reading bounded, well-formed file patterns.

Defensive patterns

Strategy: try-catch

Try / catch

try { reader.allowsDynamicSplitting(); } catch (RuntimeException e) { log.warn("Splittability check failed for {}", e.getCause(), e); }

Prevention

Verify file patterns resolve before pipeline launch.
For custom sources, make isSplittable() side-effect free and non-throwing.
Watch storage error rates during jobs with heavy dynamic splitting.

When it happens

Trigger: isSplittable() throwing during dynamic work rebalancing — underlying filesystem errors (metadata lookup failing), source in a bad state after read errors, or runner-driven split attempts during IO instability.

Common situations: Runner attempts dynamic splitting while the backing file/pattern is temporarily unavailable (GCS 5xx, HDFS namenode issues), or custom FileBasedSource.isSplittable() implementations that throw.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0fa9667754a6df03. Report an issue: GitHub.