Taking over 5 minutes to flush gcs op batches after error

Error message

Taking over 5 minutes to flush gcs op batches after error

What it means

GcsUtilV1 executes GCS operations in batches on an executor. When one batch fails, the finally block shuts the executor down and waits up to 5 minutes for in-flight operations to finish before propagating the original error. If threads don't terminate within 5 minutes, this warning is logged and shutdownNow() is called to interrupt them. It is a symptom of hung GCS requests, not a distinct failure.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:969

    List<CompletionStage<Void>> futures = new ArrayList<>();
    for (final BatchInterface batch : batches) {
      futures.add(MoreFutures.runAsync(batch::execute, executor));
    }

    try {
      try {
        MoreFutures.get(MoreFutures.allOf(futures));
      } catch (ExecutionException e) {
        if (e.getCause() instanceof FileNotFoundException) {
          throw (FileNotFoundException) e.getCause();
        }
        throw new IOException("Error executing batch GCS request", e);
      } finally {
        // Give the other batches a chance to complete in error cases.
        executor.shutdown();
        if (!executor.awaitTermination(5, TimeUnit.MINUTES)) {
          LOG.warn("Taking over 5 minutes to flush gcs op batches after error");
          executor.shutdownNow();
          if (!executor.awaitTermination(5, TimeUnit.MINUTES)) {
            LOG.warn("Took over 10 minutes to flush gcs op batches after error and interruption.");
          }
        }
      }
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new IOException("Interrupted while executing batch GCS request", e);
    }
  }

  /**
   * Makes get {@link BatchInterface BatchInterfaces}.
   *
   * @param paths {@link GcsPath GcsPaths}.
   * @param results mutable {@link List} for return values.
   * @return {@link BatchInterface BatchInterfaces} to execute.

View on GitHub (pinned to 12126d8942)

Solutions

Fix the underlying batch error logged just before this warning ("Error executing batch GCS request").
Check network connectivity/proxy settings between the worker and GCS endpoints.
Retry the pipeline; transient GCS slowness often resolves.
Reduce batch size/concurrency in GCS util configuration so the executor drains faster.

Example fix

// before
executor.shutdown();
if (!executor.awaitTermination(5, TimeUnit.MINUTES)) {
  LOG.warn("Taking over 5 minutes to flush gcs op batches after error");
  executor.shutdownNow();
}
// after
// No code fix available — this is a diagnostic. Reduce batch concurrency so
// the executor terminates quickly, e.g. smaller batches per executeBatch call.

Defensive patterns

Strategy: retry

Validate before calling

// Pre-check GCS reachability before large batch jobs:
gsutil -m stat gs://bucket/path-prefix | head -n 1 // verifies access & network path

Try / catch

try {
  gcsUtil.copy(srcs, dests);
} catch (IOException e) {
  // inspect log for 'Error executing batch GCS request' and the 5-min flush warnings
  retryWithBackoff();
}

Prevention

Keep batch sizes small so executors drain quickly after errors
Ensure workers have reliable network paths to GCS endpoints
Set explicit HTTP timeouts on the GCS transport
Monitor for repeated 'Error executing batch GCS request' warnings

When it happens

Trigger: An IOException in a batched GCS request (executeBatch) triggers the finally path, and worker threads handling other GCS ops (copy, rewrite, delete) do not finish within the 5-minute awaitTermination window, typically due to stalled network I/O or unresolved HTTP calls.

Common situations: Large bulk copy/rewrite jobs against GCS with thousands of files; network partitions or proxy timeouts leaving GCS calls hanging; overloaded GCS backend during heavy batch usage.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2fa3edbf63b184f6. Report an issue: GitHub.