What causes "Ignore error at closing ResidualSource" in apache/beam?

Inspect the full stack trace in the log to find the underlying IOException thrown by the wrapped reader's close() and fix that root cause.

apache/beam · warning

Ignore error at closing ResidualSource

Error message

Ignore error at closing ResidualSource

What it means

When a BoundedSource reader in UnboundedReadFromBoundedSource transitions to a new residual source, the currently held residualSource is explicitly closed to avoid leaking the underlying reader. An IOException raised during that close is only logged at WARN level and swallowed, so pipeline execution continues. This is intentional best-effort cleanup, not a failure of the read itself.

Solutions

Inspect the full stack trace in the log to find the underlying IOException thrown by the wrapped reader's close() and fix that root cause.
Upgrade the Beam version; the residual-source close handling has been hardened over time.
If the warning is noisy, confirm the pipeline result is still correct — the warning is deliberately non-fatal.
Ensure the wrapped BoundedSource reader tolerates being closed after the split (idempotent close).

Example fix

// before
try {
  this.residualSource.close();
} catch (IOException e) {
  LOG.warn("Ignore error at closing ResidualSource", e);
}
// after
try {
  this.residualSource.close();
} catch (IOException e) {
  LOG.warn("Ignore error at closing ResidualSource", e);
  // optionally count metrics / alert:
  closeFailureCount.inc();
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Before enabling BoundedSource->Unbounded conversion, verify the wrapped source closes cleanly:
try (BoundedSource.BoundedReader<?> r = boundedSource.createReader(options)) {
  // advance/start once to allocate resources, then close in try-with-resources
} // IOException on close here predicts the warning

Try / catch

try {
  reader.close();
} catch (IOException e) {
  LOG.warn("Ignore error at closing ResidualSource", e); // non-fatal; continue
}

Prevention

Wrap bounded sources whose readers implement idempotent close()
Watch logs for repeated close failures as a sign of underlying source bugs
Run the source in a test pipeline to catch close-time IOExceptions early
Keep Beam updated for improved residual-source lifecycle handling

When it happens

Trigger: Calling setResidualSource/getCheckpointMark on the UnboundedSourceReader when the underlying ResidualSource's reader.close() throws IOException, e.g. because the wrapped bounded source's reader failed while releasing resources (file handles, HTTP connections to a test pipeline service).

Common situations: Checkpointing during a drain or failover of a test-stream/bounded-to-unbounded conversion; backend or filesystem flakiness while closing the bounded reader; shutdown races where the channel behind the reader is already closed.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e0ab6d35b3cc4eda. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/UnboundedReadFromBoundedSource.java:311

        this.options = checkNotNull(options, "options");
        this.done = false;
      }

      private void init(
          @Nullable List<TimestampedValue<T>> residualElementsList,
          @Nullable BoundedSource<T> residualSource,
          PipelineOptions options) {
        this.residualElements =
            residualElementsList == null
                ? new ResidualElements(Collections.emptyList())
                : new ResidualElements(residualElementsList);

        if (this.residualSource != null) {
          // close current residualSource to avoid leak of reader.close() in ResidualSource
          try {
            this.residualSource.close();
          } catch (IOException e) {
            LOG.warn("Ignore error at closing ResidualSource", e);
          }
        }
        this.residualSource =
            residualSource == null ? null : new ResidualSource(residualSource, options);
      }

      @Override
      public boolean start() throws IOException {
        return advance();
      }

      @Override
      public boolean advance() throws IOException {
        if (residualElements.advance()) {
          return true;
        } else if (residualSource != null && residualSource.advance()) {
          return true;
        } else {

View on GitHub (pinned to 12126d8942)