apache/pulsar · error · IOException

Error reading from BlobStore

Error message

Error reading from BlobStore

What it means

BlobStoreBackedInputStreamImpl.read() serves reads from an in-memory buffer that is refilled with ranged GETs from the object store (S3/GCS/Azure). When a refill fails with any exception other than KeyNotFoundException, the original cause is wrapped in an IOException("Error reading from BlobStore", e) and propagated to the reader.

Solutions

1. Inspect the wrapped cause (IOException#getCause) — fix the specific storage error (credentials, permissions, throttling)
2. Re-read the offloaded segment; BlobStore reads are retryable for transient storage/network faults
3. Configure client retry/timeout parameters in the offload driver config (e.g. s3ManagedLedgerOffloadServiceEndpoint, connection limits)
4. Restore connectivity/credentials to the bucket; verify bucket name, region, and access policy

Example fix

// handling the wrap
try { stream.read(buf); }
catch (IOException e) {
  Throwable cause = e.getCause();
  if (cause instanceof SdkClientException) retryWithBackoff(); // transient storage fault
  else throw e;
}

// preflight: confirm object reachable before streaming
try (InputStream probe = blobStore.readBlob(objectKey)) {
  probe.read(); // throws early if bucket/creds are broken
}

try {
  int n = stream.read(buf);
} catch (IOException e) {
  if (e.getCause() instanceof KeyNotFoundException) {
    throw e; // blob gone: do not retry
  }
  if (isTransient(e.getCause())) retryWithBackoff(); // throttle/network
  else throw e;
}

When it happens

Trigger: refillBufferIfNeeded performs a BlobStore read via readBuffer and the underlying object-store client throws (throttling, connection reset, timeout, HTTP 5xx); KeyNotFoundException is deliberately rethrown unchanged, everything else gets this wrapped IOException.

Common situations: S3/GCS rate limiting or request throttling under read load; expired or revoked cloud credentials; network partition between broker and object store; bucket policy or firewall blocking the broker; transient 503 from the storage backend.

Related errors

• Cannot obtain authorization metadata from
• Data block header magic word not match. read
• Encountered error " " when getting package from
• Error seeking, new position
• Fail to read LedgerMetadata for ledgerId

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/bd961116fd6bfc40. Report an issue: GitHub.

Appendix: source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java:128

                }

                // here we can get the metrics
                // because JClouds streams the content
                // and actually the HTTP call finishes when the stream is fully read
                if (this.offloaderStats != null) {
                    this.offloaderStats.recordReadOffloadDataLatency(topicName,
                            System.nanoTime() - startReadTime, TimeUnit.NANOSECONDS);
                    this.offloaderStats.recordReadOffloadBytes(topicName, endRange - startRange + 1);
                }
            } catch (Throwable e) {
                if (null != this.offloaderStats) {
                    this.offloaderStats.recordReadOffloadError(this.topicName);
                }
                // If the blob is not found, the original exception is thrown and handled by the caller.
                if (e instanceof KeyNotFoundException) {
                    throw e;
                }
                throw new IOException("Error reading from BlobStore", e);
            }
        }
        return true;
    }

    void fillBuffer(InputStream is, int bytesToCopy) throws IOException {
        while (bytesToCopy > 0) {
            int writeBytes = buffer.writeBytes(is, bytesToCopy);
            if (writeBytes < 0) {
                break;
            }
            bytesToCopy -= writeBytes;
        }
    }

    ByteBuf getBuffer() {
        return buffer;
    }

View on GitHub (pinned to 820761864e)