apache/druid · error · IOException

Timed out after [ ] millis during flushing

Error message

Timed out after [%d] millis during flushing

What it means

flush() waits (emittedBatchCounter.awaitCount) until the batch being flushed has actually been sent by the EmittingThread. If the batch is not emitted within HttpEmitterConfig.flushTimeOut milliseconds, a TimeoutException is converted to IOException('Timed out after [%d] millis during flushing'). This indicates the sending pipeline (network, remote endpoint, queue backlog) is stalled.

Solutions

1. Increase HttpEmitterConfig.flushTimeOut to cover worst-case send duration.
2. Verify the recipientBaseUrl endpoint is reachable and responding; check network/proxy issues.
3. Reduce maxBatchSize or batchQueueSize so individual sends complete faster.
4. Catch IOException from flush()/close() and log; consider retrying flush after checking emitter health.

Example fix

// before
HttpEmitterConfig.builder().setFlushTimeOut(1000).build();
// after
HttpEmitterConfig.builder().setFlushTimeOut(60000).build();

// pre-check endpoint reachability
HttpURLConnection c = (HttpURLConnection) new URI(config.getRecipientBaseUrl()).toURL().openConnection();
c.setConnectTimeout(2000);
if (c.getResponseCode() >= 400) log.warn("Recipient unhealthy");

try {
  emitter.flush();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Timed out")) {
    log.error(e, "Flush timed out; recipient slow or batch too large");
  }
}

When it happens

Trigger: flush() or close() while the EmittingThread is stuck posting large batches to a slow/unreachable recipient; flushTimeOut too small for batch size/network latency; event queue backlog larger than the thread can drain in time.

Common situations: Remote telemetry endpoint (e.g. overlord or external HTTP service) hanging or rate-limiting; network partitions; misconfigured flushTimeOut (default a few seconds) with big maxBatchSize; calling close() during shutdown with many queued events.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Exception pulling item from cache
• failed to flush one or more emitters
• Query [ ] timed out!
• Cache for node role [ ] could not be initialized before…
• Cache initialization for node role

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/7249cf14d96aa782. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/java/util/emitter/core/HttpPostEmitter.java:429

      flush((Batch) batchObj);
    }
  }

  private void flush(Batch batch) throws IOException
  {
    if (batch == null) {
      return;
    }
    batch.seal();
    try {
      // This check doesn't always awaits for this exact batch to be emitted, because another batch could be dropped
      // from the queue ahead of this one, in limitBuffersToEmitSize(). But there is no better way currently to wait for
      // the exact batch, and it's not that important.
      emittedBatchCounter.awaitCount(batch.batchNumber, config.getFlushTimeOut(), TimeUnit.MILLISECONDS);
    }
    catch (TimeoutException e) {
      String message = StringUtils.format("Timed out after [%d] millis during flushing", config.getFlushTimeOut());
      throw new IOException(message, e);
    }
    catch (InterruptedException e) {
      log.debug("Thread Interrupted");
      Thread.currentThread().interrupt();
      throw new IOException("Thread Interrupted while flushing", e);
    }
  }

  @Override
  @LifecycleStop
  public void close() throws IOException
  {
    synchronized (startLock) {
      if (running) {
        running = false;
        Object lastBatch = concurrentBatch.getAndSet(null);
        if (lastBatch instanceof Batch) {
          flush((Batch) lastBatch);

View on GitHub (pinned to 9b90983fd2)