apache/beam · error · IOException

Periodic flushing thread finished unexpectedly.

Error message

Periodic flushing thread finished unexpectedly.

What it means

The aggregator schedules a periodic flush task that drains buffered outbound data. checkFlushThreadException verifies the task is still running; if flushFuture is done but get() returns normally (no result), the thread terminated silently, which would leave data unflushed, so an IOException is thrown.

Solutions

Inspect the executor/scheduler used to schedule the flush task — ensure it is not shut down prematurely
Check logs preceding this error for scheduling or cancellation causes
Ensure the aggregator is only used while its bundle and flush thread are active
Upgrade Beam version; scheduling lifecycle bugs have been patched historically

Example fix

// before
ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
executor.shutdown(); // flush thread finishes unexpectedly
// after
// shut down the executor only after aggregator.close()

Defensive patterns

Strategy: try-catch

Validate before calling

if (flushFuture != null && flushFuture.isDone()) {
  recreateAggregator();
}

Try / catch

try {
  aggregator.registerOutputDataLocation(id, coder);
} catch (IOException e) {
  if (e.getMessage().contains("Periodic flushing")) {
    LOG.error("flush thread died unexpectedly", e);
  }
  throw e;
}

Prevention

Keep the scheduler executor alive for the aggregator's lifetime
Monitor flush task failures promptly via logs
Close the aggregator explicitly when done to stop the flush thread cleanly

When it happens

Trigger: The scheduled flush task completed without an exception while the aggregator is still active; detected on the next registerOutputDataLocation or registerOutputTimersLocation call. If the task failed with an exception, ExecutionException is unwrapped instead.

Common situations: Executor shutdown or misconfiguration causing the periodic task to terminate cleanly; a runner closing the aggregator's scheduler early while registration is still happening.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/56b10f8594b99095. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java:334

  void flush() {
    try {
      synchronized (flushLock) {
        flushInternal();
      }
    } catch (OutOfMemoryError oom) {
      throw oom;
    } catch (Throwable t) {
      throw new RuntimeException(t);
    }
  }

  /** Check if the flush thread failed with an exception. */
  private void checkFlushThreadException() throws IOException {
    if (flushFuture != null && flushFuture.isDone()) {
      try {
        flushFuture.get();
        throw new IOException("Periodic flushing thread finished unexpectedly.");
      } catch (ExecutionException ee) {
        unwrapExecutionException(ee);
      } catch (CancellationException ce) {
        throw new IOException(ce);
      } catch (InterruptedException ie) {
        Thread.currentThread().interrupt();
        throw new IOException(ie);
      }
    }
  }

  private void unwrapExecutionException(ExecutionException ee) throws IOException {
    // the cause is always RuntimeException
    RuntimeException re = (RuntimeException) ee.getCause();
    if (re.getCause() instanceof IOException) {
      throw (IOException) re.getCause();
    } else {
      throw new IOException(re.getCause());

View on GitHub (pinned to 12126d8942)