apache/pulsar · error · IOException

Error reading from BlobStore

Error message

Error reading from BlobStore

What it means

BlobStoreBackedInputStreamImpl.read() serves reads from an in-memory buffer that is refilled with ranged GETs from the object store (S3/GCS/Azure). When a refill fails with any exception other than KeyNotFoundException, the original cause is wrapped in an IOException("Error reading from BlobStore", e) and propagated to the reader.

Source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java:128

                }

                // here we can get the metrics
                // because JClouds streams the content
                // and actually the HTTP call finishes when the stream is fully read
                if (this.offloaderStats != null) {
                    this.offloaderStats.recordReadOffloadDataLatency(topicName,
                            System.nanoTime() - startReadTime, TimeUnit.NANOSECONDS);
                    this.offloaderStats.recordReadOffloadBytes(topicName, endRange - startRange + 1);
                }
            } catch (Throwable e) {
                if (null != this.offloaderStats) {
                    this.offloaderStats.recordReadOffloadError(this.topicName);
                }
                // If the blob is not found, the original exception is thrown and handled by the caller.
                if (e instanceof KeyNotFoundException) {
                    throw e;
                }
                throw new IOException("Error reading from BlobStore", e);
            }
        }
        return true;
    }

    void fillBuffer(InputStream is, int bytesToCopy) throws IOException {
        while (bytesToCopy > 0) {
            int writeBytes = buffer.writeBytes(is, bytesToCopy);
            if (writeBytes < 0) {
                break;
            }
            bytesToCopy -= writeBytes;
        }
    }

    ByteBuf getBuffer() {
        return buffer;
    }

View on GitHub (pinned to 820761864e)

Solutions

1. Inspect the wrapped cause (IOException#getCause) — fix the specific storage error (credentials, permissions, throttling)
2. Re-read the offloaded segment; BlobStore reads are retryable for transient storage/network faults
3. Configure client retry/timeout parameters in the offload driver config (e.g. s3ManagedLedgerOffloadServiceEndpoint, connection limits)
4. Restore connectivity/credentials to the bucket; verify bucket name, region, and access policy

Example fix

// handling the wrap
try { stream.read(buf); }
catch (IOException e) {
  Throwable cause = e.getCause();
  if (cause instanceof SdkClientException) retryWithBackoff(); // transient storage fault
  else throw e;
}

// preflight: confirm object reachable before streaming
try (InputStream probe = blobStore.readBlob(objectKey)) {
  probe.read(); // throws early if bucket/creds are broken
}

try {
  int n = stream.read(buf);
} catch (IOException e) {
  if (e.getCause() instanceof KeyNotFoundException) {
    throw e; // blob gone: do not retry
  }
  if (isTransient(e.getCause())) retryWithBackoff(); // throttle/network
  else throw e;
}

When it happens

Trigger: refillBufferIfNeeded performs a BlobStore read via readBuffer and the underlying object-store client throws (throttling, connection reset, timeout, HTTP 5xx); KeyNotFoundException is deliberately rethrown unchanged, everything else gets this wrapped IOException.

Common situations: S3/GCS rate limiting or request throttling under read load; expired or revoked cloud credentials; network partition between broker and object store; bucket policy or firewall blocking the broker; transient 503 from the storage backend.

Related errors

• Cannot obtain authorization metadata from ${metadataUrl}
• Encountered error "%s" when getting %s package from %s
• Fail to read LedgerMetadata for ledgerId ${key}
• Error seeking, new position %d < current position %d
• ManagedLedgerException (wrapped from err in OffloadedLedgerM

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/bd961116fd6bfc40. Report an issue: GitHub.