Executor service is taking long time to shutdown, will retry

Error message

Executor service is taking long time to shutdown, will retry. {} attempts left

What it means

awaitTermination repeatedly waits 10-second intervals for the executor to finish. When a wait expires but attempts remain, it logs this warning and retries; shutdown proceeds once attempts are exhausted (escalating to shutdownNow in stop()).

Source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/ShardReadersPool.java:243

          ATTEMPTS_TO_SHUTDOWN);
      executorService.shutdownNow();
      awaitTermination();
    }
  }

  private void awaitTermination() {
    int attemptsLeft = ATTEMPTS_TO_SHUTDOWN;
    boolean isTerminated = executorService.isTerminated();

    while (!isTerminated && attemptsLeft-- > 0) {
      try {
        isTerminated = executorService.awaitTermination(10, TimeUnit.SECONDS);
      } catch (InterruptedException e) {
        LOG.error("Interrupted while waiting for the executor service to shutdown");
        throw new RuntimeException(e);
      }
      if (!isTerminated && attemptsLeft > 0) {
        LOG.warn(
            "Executor service is taking long time to shutdown, will retry. {} attempts left",
            attemptsLeft);
      }
    }
  }

  Instant getWatermark() {
    return getMinTimestamp(ShardRecordsIterator::getShardWatermark);
  }

  Instant getLatestRecordTimestamp() {
    return getMinTimestamp(ShardRecordsIterator::getLatestRecordTimestamp);
  }

  private Instant getMinTimestamp(Function<ShardRecordsIterator, Instant> timestampExtractor) {
    return minTimestamp(shardIteratorsMap.get().values().stream().map(timestampExtractor));
  }

View on GitHub (pinned to 12126d8942)

Solutions

Wait it out — shutdown escalates to shutdownNow after remaining attempts elapse
Keep downstream consumption healthy so the reader queue drains and tasks exit promptly
Set aggressive client-level request timeouts so in-flight getRecords calls cannot stall termination
Check for application code holding interrupts (catching InterruptedException without re-interrupting) inside custom DoFns

Defensive patterns

Strategy: try-catch

Try / catch

try {
  executorService.shutdown();
  if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {
    executorService.shutdownNow();
  }
} catch (InterruptedException e) {
  Thread.currentThread().interrupt();
  executorService.shutdownNow();
}

Prevention

Same as 6134: drain queues, bound request timeouts, respect interrupts
Avoid sharing one executor across many slow sinks
Check SQS/Kinesis endpoint latency before scheduled shutdown windows

When it happens

Trigger: Executor threads running shard reading loops don't terminate within 10 seconds per attempt during ShardReadersPool.stop(), typically because they are blocked on a full records queue or an in-flight Kinesis request.

Common situations: Slow consumer pipelines during teardown; long-running getRecords calls; Kinesis endpoint latency; threads that swallow interrupts.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4ba8e6270dd5fbc1. Report an issue: GitHub.