{"record":{"id":"032cf732f7e86f76","repo":"apache/beam","slug":"executor-service-was-not-completely-terminated-after","errorCode":null,"errorMessage":"Executor service was not completely terminated after {} attempts, trying to forcibly stop it.","messagePattern":"Executor service was not completely terminated after (.+?) attempts, trying to forcibly stop it\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/ShardReadersPool.java","lineNumber":223,"sourceCode":"      shardIteratorsMap.get().get(record.getShardId()).ackRecord(record);\n\n      // numberOfRecordsInAQueueByShard contains the counter for a given shard until the shard is\n      // closed and then it's counter reaches 0. Thus the access here is safe\n      numberOfRecordsInAQueueByShard.get(record.getShardId()).decrementAndGet();\n      return CustomOptional.of(record);\n    } catch (InterruptedException e) {\n      LOG.warn(\"Interrupted while waiting for KinesisRecord from the buffer\");\n      return CustomOptional.absent();\n    }\n  }\n\n  void stop() {\n    LOG.info(\"Closing shard iterators pool\");\n    poolOpened.set(false);\n    executorService.shutdown();\n    awaitTermination();\n    if (!executorService.isTerminated()) {\n      LOG.warn(\n          \"Executor service was not completely terminated after {} attempts, trying to forcibly stop it.\",\n          ATTEMPTS_TO_SHUTDOWN);\n      executorService.shutdownNow();\n      awaitTermination();\n    }\n  }\n\n  private void awaitTermination() {\n    int attemptsLeft = ATTEMPTS_TO_SHUTDOWN;\n    boolean isTerminated = executorService.isTerminated();\n\n    while (!isTerminated && attemptsLeft-- > 0) {\n      try {\n        isTerminated = executorService.awaitTermination(10, TimeUnit.SECONDS);\n      } catch (InterruptedException e) {\n        LOG.error(\"Interrupted while waiting for the executor service to shutdown\");\n        throw new RuntimeException(e);\n      }","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/ShardReadersPool.java#L205-L241","documentation":"During pool shutdown, ShardReadersPool stops its executor and waits for termination. After ATTEMPTS_TO_SHUTDOWN graceful waits the tasks still run, so the pool logs this warning and escalates to shutdownNow(), which interrupts the reader threads.","triggerScenarios":"Executor tasks (kinesis getRecords loops / queue puts) do not respond to graceful shutdown within the await-termination window at pipeline teardown — e.g. threads blocked on blocking queue puts or long HTTP calls that ignore interruption until their current call finishes.","commonSituations":"Teardown of a streaming pipeline with a slow or hung Kinesis endpoint; consumer queue full because the downstream is slower than reads; very long getRecords call durations.","solutions":["No user fix required — shutdownNow() forcibly interrupts the tasks; verify threads actually stop afterwards","Ensure downstream consumers keep up so the reader queue is not persistently full during shutdown","Tune getRecords request timeouts in the AWS client so calls cannot block shutdown for long","If seen frequently, upgrade Beam; shutdown handling in ShardReadersPool has been revised over versions"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// ensure clean teardown in custom pipelines\nexecutorService.shutdown();\nif (!executorService.awaitTermination(30, TimeUnit.SECONDS)) {\n  executorService.shutdownNow();\n  executorService.awaitTermination(10, TimeUnit.SECONDS);\n}","preventionTips":["Keep reader queue consumers healthy so tasks can exit on interrupt","Set AWS client request timeouts below the shutdown wait budget","Avoid swallowing InterruptedException in custom code","Monitor for tasks that survive shutdownNow"],"tags":["kinesis","thread-pool","shutdown","interruption"],"backgroundTag":"thread-interrupted","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}