OpenLineage emit queue full, discarded [%,d] events total

Error message

OpenLineage emit queue full, discarded [%,d] events total

What it means

OpenLineageRequestLogger uses a bounded worker queue; when the queue is full, the rejected execution handler discards the event, increments discardedCount, and logs this warning on the first discard and then every DISCARD_WARNING_INTERVAL discards. Events are lost, not queued.

Solutions

Fix upstream slowness: check why POSTs to the OpenLineage endpoint are slow (see dropped-event warnings)
Increase the emit thread pool size / queue capacity for the OpenLineage request logger
Reduce event volume (e.g. only emit lineage for certain datasources) or batch events
Monitor the discardedCount metric and alert when it starts incrementing

Defensive patterns

Strategy: fallback

Prevention

Size the emit queue/thread pool above peak query rate
Fix slow collector endpoints promptly
Reduce lineage event volume (filter datasources)
Alert on discardedCount increases

When it happens

Trigger: emit() is called faster than the HTTP POST worker can drain the queue — e.g. burst of many SQL queries while the OpenLineage endpoint is slow or unreachable, or thread pool size/queue capacity too small for the event rate.

Common situations: Slow OpenLineage collector causing POSTs to block; a query storm (heavy BI refresh) generating lineage events faster than they can be sent; under-provisioned executor after tuning changes.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/112d35e180d7a541. Report an issue: GitHub.

Appendix: source

Thrown at extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:570

  /**
   * Rejection handler that discards events when the emit queue is full, but logs a warning
   * on the first drop and every {@link #DISCARD_WARNING_INTERVAL} drops thereafter.
   */
  private static class DiscardWithWarningPolicy implements RejectedExecutionHandler
  {
    private final AtomicLong discardedCount;

    DiscardWithWarningPolicy(AtomicLong discardedCount)
    {
      this.discardedCount = discardedCount;
    }

    @Override
    public void rejectedExecution(Runnable r, ThreadPoolExecutor executor)
    {
      long count = discardedCount.incrementAndGet();
      if (count == 1 || count % DISCARD_WARNING_INTERVAL == 0) {
        log.warn("OpenLineage emit queue full, discarded [%,d] events total", count);
      }
    }
  }

}

View on GitHub (pinned to 9b90983fd2)