apache/druid · warning · IOException

Recoverable exception

Error message

Recoverable exception

What it means

AzureByteSource.openStream wraps BlobStorageException from Azure storage: if the error is classified as retryable by AZURE_RETRY, it is rethrown as an IOException('Recoverable exception') so callers can retry; non-retryable errors are logged and rethrown as RuntimeException.

Source

Thrown at extensions-core/azure-extensions/src/main/java/org/apache/druid/storage/azure/AzureByteSource.java:66

    this.azureStorage = azureStorage;
    this.containerName = containerName;
    this.blobPath = blobPath;
  }

  @Override
  public InputStream openStream() throws IOException
  {
    return openStream(0L);
  }

  public InputStream openStream(long offset) throws IOException
  {
    try {
      return azureStorage.getBlockBlobInputStream(offset, containerName, blobPath);
    }
    catch (BlobStorageException e) {
      if (AzureUtils.AZURE_RETRY.apply(e)) {
        throw new IOException("Recoverable exception", e);
      }
      log.error("Exception when opening stream to azure resource, containerName: %s, blobPath: %s, Error: %s",
               containerName, blobPath, e.getMessage()
      );
      throw new RuntimeException(e);
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Retry the operation — the error is explicitly marked recoverable; Druid task retry machinery usually handles it
2. Enable/configure retry policies with backoff on the Azure storage client
3. Check Azure Storage metrics/alerts for throttling (429) or availability incidents
4. Scale down concurrent readers or increase storage account capacity if consistently throttled

Example fix

// caller pattern
try {
  InputStream in = byteSource.openStream();
} catch (IOException e) {
  if (e.getMessage().contains("Recoverable exception")) {
    // retry with backoff
    in = retryer.call(() -> byteSource.openStream());
  } else {
    throw e;
  }
}

// preflight blob existence before openStream
CloudBlockBlob blob = container.getBlockBlobReference(blobPath);
if (!blob.exists()) throw new FileNotFoundException(blobPath);

Retryer<InputStream> retryer = RetryerBuilder.<InputStream>newBuilder()
    .retryIfException(e -> e instanceof IOException
        && "Recoverable exception".equals(e.getMessage()))
    .withWaitStrategy(WaitStrategies.exponentialWait(1, TimeUnit.SECONDS))
    .withStopStrategy(StopStrategies.stopAfterAttempt(5))
    .build();
InputStream in = retryer.call(() -> byteSource.openStream());

When it happens

Trigger: openStream() calls azureStorage.getBlockBlobInputStream(offset, containerName, blobPath) and Azure returns a transient BlobStorageException — HTTP 500/503 from storage, throttling (429), server busy, or intermittent connection resets.

Common situations: Azure storage account under throttling during heavy ingest; transient storage-service incidents; network blips between Druid and Azure blob endpoint during deep-storage reads (e.g., segment loading).

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Failed to get blob item from Azure container[%s], prefix[%s
• RuntimeException wrapping underlying cause
• IOException wrapping underlying cause
• Failed to stream logs from: %s
• IOException wrapping underlying cause

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/bb138bdc9b58ddc9. Report an issue: GitHub.