Took over 10 minutes to flush gcs op batches after error and

Error message

Took over 10 minutes to flush gcs op batches after error and interruption.

What it means

After a batch GCS request error, GcsUtilV1 first waits 5 minutes (logging [6031]), interrupts the executor with shutdownNow(), and then waits another 5 minutes. If threads still do not terminate, this warning is logged. GCS worker threads are ignoring interruption, usually because they are blocked in uninterruptible socket I/O. The original batch error is still the failure to investigate.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:972

      futures.add(MoreFutures.runAsync(batch::execute, executor));
    }

    try {
      try {
        MoreFutures.get(MoreFutures.allOf(futures));
      } catch (ExecutionException e) {
        if (e.getCause() instanceof FileNotFoundException) {
          throw (FileNotFoundException) e.getCause();
        }
        throw new IOException("Error executing batch GCS request", e);
      } finally {
        // Give the other batches a chance to complete in error cases.
        executor.shutdown();
        if (!executor.awaitTermination(5, TimeUnit.MINUTES)) {
          LOG.warn("Taking over 5 minutes to flush gcs op batches after error");
          executor.shutdownNow();
          if (!executor.awaitTermination(5, TimeUnit.MINUTES)) {
            LOG.warn("Took over 10 minutes to flush gcs op batches after error and interruption.");
          }
        }
      }
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new IOException("Interrupted while executing batch GCS request", e);
    }
  }

  /**
   * Makes get {@link BatchInterface BatchInterfaces}.
   *
   * @param paths {@link GcsPath GcsPaths}.
   * @param results mutable {@link List} for return values.
   * @return {@link BatchInterface BatchInterfaces} to execute.
   * @throws IOException
   */
  @VisibleForTesting

View on GitHub (pinned to 12126d8942)

Solutions

Treat the pre-existing batch error as root cause; examine the first stack trace in the log.
Verify socket/HTTP timeouts are configured for the GCS transport so stuck threads eventually fail.
Ensure the runner terminates the worker process/VM if graceful flush fails — data integrity is unaffected since ops after the error are abandoned.
Reduce per-batch concurrency and retry the job.

Defensive patterns

Strategy: retry

Try / catch

try {
  gcsUtil.copy(srcs, dests);
} catch (IOException e) {
  // if logs show 10-minute flush warning, threads were stuck; fail worker and retry job
}

Prevention

Configure socket timeouts so stuck GCS threads eventually fail
Reduce per-batch concurrency
Rely on runner-level worker restart when graceful flush times out
Treat the first batch error in the log as the true root cause

When it happens

Trigger: executeBatch fails, the executor's worker threads survive shutdownNow() interruption for more than 10 minutes total — blocked in non-interruptible HTTP/socket reads to GCS.

Common situations: Stalled TCP connections without proper socket timeouts; pathological network conditions in worker VMs; extremely large batch jobs where ops are hung on individual objects.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8d912fad51db23a3. Report an issue: GitHub.