apache/hadoop · error · InvalidConfigurationValueException

fs.azure.blob.dir.list.consumer.max.lag

fs.azure.blob.dir.list.consumer.max.lag

Error message

maxConsumptionLag should be lesser than maxSize

What it means

Error "maxConsumptionLag should be lesser than maxSize" thrown in apache/hadoop.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ListBlobQueue.java:86

   * it will wait for 5000 items to be consumed before generating next 5000 items.
   * <p>
   * If this is not used, the producer will keep on producing items as soon as
   * the queue become available with small size. Let say, 5 items got consumed,
   * producer would make a server call for only 5 items and would populate the queue.
   * <p>
   * This mechanism would prevent producer making server calls for very small amount
   * of items.
   */
  private final int maxConsumptionLag;

  ListBlobQueue(int maxSize, int consumeSetSize, int maxConsumptionLag)
      throws InvalidConfigurationValueException {
    this.maxSize = maxSize;
    this.maxConsumptionLag = maxConsumptionLag;
    this.consumeSetSize = consumeSetSize;

    if (maxConsumptionLag >= maxSize) {
      throw new InvalidConfigurationValueException(FS_AZURE_CONSUMER_MAX_LAG,
          "maxConsumptionLag should be lesser than maxSize");
    }
  }

  /** Mark the queue as failed.*/
  void markProducerFailure(AzureBlobFileSystemException failure) {
    failureFromProducer = failure;
  }

  /** Mark the queue as completed.*/
  void complete() {
    isCompleted = true;
  }

  /** Mark the consumption as failed.*/
  synchronized void markConsumptionFailed() {
    isConsumptionFailed = true;
    notify();

View on GitHub (pinned to 2add963021)

Solutions

1. Configure maxConsumptionLag smaller than the listing queue maxSize.
2. Adjust fs.azure.list.blob.queue.* configuration values consistently.

When it happens

Trigger: ListBlobQueue is configured with maxConsumptionLag >= maxSize.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e4df0d11430c4d15. Report an issue: GitHub.