{"record":{"id":"2fa3edbf63b184f6","repo":"apache/beam","slug":"taking-over-5-minutes-to-flush-gcs-op-batches-after-error","errorCode":null,"errorMessage":"Taking over 5 minutes to flush gcs op batches after error","messagePattern":"Taking over 5 minutes to flush gcs op batches after error","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java","lineNumber":969,"sourceCode":"\n    List<CompletionStage<Void>> futures = new ArrayList<>();\n    for (final BatchInterface batch : batches) {\n      futures.add(MoreFutures.runAsync(batch::execute, executor));\n    }\n\n    try {\n      try {\n        MoreFutures.get(MoreFutures.allOf(futures));\n      } catch (ExecutionException e) {\n        if (e.getCause() instanceof FileNotFoundException) {\n          throw (FileNotFoundException) e.getCause();\n        }\n        throw new IOException(\"Error executing batch GCS request\", e);\n      } finally {\n        // Give the other batches a chance to complete in error cases.\n        executor.shutdown();\n        if (!executor.awaitTermination(5, TimeUnit.MINUTES)) {\n          LOG.warn(\"Taking over 5 minutes to flush gcs op batches after error\");\n          executor.shutdownNow();\n          if (!executor.awaitTermination(5, TimeUnit.MINUTES)) {\n            LOG.warn(\"Took over 10 minutes to flush gcs op batches after error and interruption.\");\n          }\n        }\n      }\n    } catch (InterruptedException e) {\n      Thread.currentThread().interrupt();\n      throw new IOException(\"Interrupted while executing batch GCS request\", e);\n    }\n  }\n\n  /**\n   * Makes get {@link BatchInterface BatchInterfaces}.\n   *\n   * @param paths {@link GcsPath GcsPaths}.\n   * @param results mutable {@link List} for return values.\n   * @return {@link BatchInterface BatchInterfaces} to execute.","sourceCodeStart":951,"sourceCodeEnd":987,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java#L951-L987","documentation":"GcsUtilV1 executes GCS operations in batches on an executor. When one batch fails, the finally block shuts the executor down and waits up to 5 minutes for in-flight operations to finish before propagating the original error. If threads don't terminate within 5 minutes, this warning is logged and shutdownNow() is called to interrupt them. It is a symptom of hung GCS requests, not a distinct failure.","triggerScenarios":"An IOException in a batched GCS request (executeBatch) triggers the finally path, and worker threads handling other GCS ops (copy, rewrite, delete) do not finish within the 5-minute awaitTermination window, typically due to stalled network I/O or unresolved HTTP calls.","commonSituations":"Large bulk copy/rewrite jobs against GCS with thousands of files; network partitions or proxy timeouts leaving GCS calls hanging; overloaded GCS backend during heavy batch usage.","solutions":["Fix the underlying batch error logged just before this warning (\"Error executing batch GCS request\").","Check network connectivity/proxy settings between the worker and GCS endpoints.","Retry the pipeline; transient GCS slowness often resolves.","Reduce batch size/concurrency in GCS util configuration so the executor drains faster."],"exampleFix":"// before\nexecutor.shutdown();\nif (!executor.awaitTermination(5, TimeUnit.MINUTES)) {\n  LOG.warn(\"Taking over 5 minutes to flush gcs op batches after error\");\n  executor.shutdownNow();\n}\n// after\n// No code fix available — this is a diagnostic. Reduce batch concurrency so\n// the executor terminates quickly, e.g. smaller batches per executeBatch call.","handlingStrategy":"retry","validationCode":"// Pre-check GCS reachability before large batch jobs:\ngsutil -m stat gs://bucket/path-prefix | head -n 1 // verifies access & network path","typeGuard":null,"tryCatchPattern":"try {\n  gcsUtil.copy(srcs, dests);\n} catch (IOException e) {\n  // inspect log for 'Error executing batch GCS request' and the 5-min flush warnings\n  retryWithBackoff();\n}","preventionTips":["Keep batch sizes small so executors drain quickly after errors","Ensure workers have reliable network paths to GCS endpoints","Set explicit HTTP timeouts on the GCS transport","Monitor for repeated 'Error executing batch GCS request' warnings"],"tags":["gcs","network","timeout","thread-pool"],"backgroundTag":"request-timeout","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-15T02:17:10.978Z"}