{"record":{"id":"8d912fad51db23a3","repo":"apache/beam","slug":"took-over-10-minutes-to-flush-gcs-op-batches-after-error-and","errorCode":null,"errorMessage":"Took over 10 minutes to flush gcs op batches after error and interruption.","messagePattern":"Took over 10 minutes to flush gcs op batches after error and interruption\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java","lineNumber":972,"sourceCode":"      futures.add(MoreFutures.runAsync(batch::execute, executor));\n    }\n\n    try {\n      try {\n        MoreFutures.get(MoreFutures.allOf(futures));\n      } catch (ExecutionException e) {\n        if (e.getCause() instanceof FileNotFoundException) {\n          throw (FileNotFoundException) e.getCause();\n        }\n        throw new IOException(\"Error executing batch GCS request\", e);\n      } finally {\n        // Give the other batches a chance to complete in error cases.\n        executor.shutdown();\n        if (!executor.awaitTermination(5, TimeUnit.MINUTES)) {\n          LOG.warn(\"Taking over 5 minutes to flush gcs op batches after error\");\n          executor.shutdownNow();\n          if (!executor.awaitTermination(5, TimeUnit.MINUTES)) {\n            LOG.warn(\"Took over 10 minutes to flush gcs op batches after error and interruption.\");\n          }\n        }\n      }\n    } catch (InterruptedException e) {\n      Thread.currentThread().interrupt();\n      throw new IOException(\"Interrupted while executing batch GCS request\", e);\n    }\n  }\n\n  /**\n   * Makes get {@link BatchInterface BatchInterfaces}.\n   *\n   * @param paths {@link GcsPath GcsPaths}.\n   * @param results mutable {@link List} for return values.\n   * @return {@link BatchInterface BatchInterfaces} to execute.\n   * @throws IOException\n   */\n  @VisibleForTesting","sourceCodeStart":954,"sourceCodeEnd":990,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java#L954-L990","documentation":"After a batch GCS request error, GcsUtilV1 first waits 5 minutes (logging [6031]), interrupts the executor with shutdownNow(), and then waits another 5 minutes. If threads still do not terminate, this warning is logged. GCS worker threads are ignoring interruption, usually because they are blocked in uninterruptible socket I/O. The original batch error is still the failure to investigate.","triggerScenarios":"executeBatch fails, the executor's worker threads survive shutdownNow() interruption for more than 10 minutes total — blocked in non-interruptible HTTP/socket reads to GCS.","commonSituations":"Stalled TCP connections without proper socket timeouts; pathological network conditions in worker VMs; extremely large batch jobs where ops are hung on individual objects.","solutions":["Treat the pre-existing batch error as root cause; examine the first stack trace in the log.","Verify socket/HTTP timeouts are configured for the GCS transport so stuck threads eventually fail.","Ensure the runner terminates the worker process/VM if graceful flush fails — data integrity is unaffected since ops after the error are abandoned.","Reduce per-batch concurrency and retry the job."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  gcsUtil.copy(srcs, dests);\n} catch (IOException e) {\n  // if logs show 10-minute flush warning, threads were stuck; fail worker and retry job\n}","preventionTips":["Configure socket timeouts so stuck GCS threads eventually fail","Reduce per-batch concurrency","Rely on runner-level worker restart when graceful flush times out","Treat the first batch error in the log as the true root cause"],"tags":["gcs","network","timeout","interruption"],"backgroundTag":"request-timeout","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-15T02:17:10.978Z"}