apache/beam · warning

Executor service was not completely terminated after {} atte

Error message

Executor service was not completely terminated after {} attempts, trying to forcibly stop it.

What it means

During pool shutdown, ShardReadersPool stops its executor and waits for termination. After ATTEMPTS_TO_SHUTDOWN graceful waits the tasks still run, so the pool logs this warning and escalates to shutdownNow(), which interrupts the reader threads.

Source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/ShardReadersPool.java:223

      shardIteratorsMap.get().get(record.getShardId()).ackRecord(record);

      // numberOfRecordsInAQueueByShard contains the counter for a given shard until the shard is
      // closed and then it's counter reaches 0. Thus the access here is safe
      numberOfRecordsInAQueueByShard.get(record.getShardId()).decrementAndGet();
      return CustomOptional.of(record);
    } catch (InterruptedException e) {
      LOG.warn("Interrupted while waiting for KinesisRecord from the buffer");
      return CustomOptional.absent();
    }
  }

  void stop() {
    LOG.info("Closing shard iterators pool");
    poolOpened.set(false);
    executorService.shutdown();
    awaitTermination();
    if (!executorService.isTerminated()) {
      LOG.warn(
          "Executor service was not completely terminated after {} attempts, trying to forcibly stop it.",
          ATTEMPTS_TO_SHUTDOWN);
      executorService.shutdownNow();
      awaitTermination();
    }
  }

  private void awaitTermination() {
    int attemptsLeft = ATTEMPTS_TO_SHUTDOWN;
    boolean isTerminated = executorService.isTerminated();

    while (!isTerminated && attemptsLeft-- > 0) {
      try {
        isTerminated = executorService.awaitTermination(10, TimeUnit.SECONDS);
      } catch (InterruptedException e) {
        LOG.error("Interrupted while waiting for the executor service to shutdown");
        throw new RuntimeException(e);
      }

View on GitHub (pinned to 12126d8942)

Solutions

1. No user fix required — shutdownNow() forcibly interrupts the tasks; verify threads actually stop afterwards
2. Ensure downstream consumers keep up so the reader queue is not persistently full during shutdown
3. Tune getRecords request timeouts in the AWS client so calls cannot block shutdown for long
4. If seen frequently, upgrade Beam; shutdown handling in ShardReadersPool has been revised over versions

// ensure clean teardown in custom pipelines
executorService.shutdown();
if (!executorService.awaitTermination(30, TimeUnit.SECONDS)) {
  executorService.shutdownNow();
  executorService.awaitTermination(10, TimeUnit.SECONDS);
}

When it happens

Trigger: Executor tasks (kinesis getRecords loops / queue puts) do not respond to graceful shutdown within the await-termination window at pipeline teardown — e.g. threads blocked on blocking queue puts or long HTTP calls that ignore interruption until their current call finishes.

Common situations: Teardown of a streaming pipeline with a slow or hung Kinesis endpoint; consumer queue full because the downstream is slower than reads; very long getRecords call durations.

Related errors

• Thread was interrupted, finishing the read loop
• Executor service is taking long time to shutdown, will retry
• Interrupted while waiting for KinesisRecord from the buffer
• Interrupted while waiting for space in buffer
• RecordWriter is null

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/032cf732f7e86f76. Report an issue: GitHub.