{"record":{"id":"bd961116fd6bfc40","repo":"apache/pulsar","slug":"error-reading-from-blobstore","errorCode":null,"errorMessage":"Error reading from BlobStore","messagePattern":"Error reading from BlobStore","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java","lineNumber":128,"sourceCode":"                }\n\n                // here we can get the metrics\n                // because JClouds streams the content\n                // and actually the HTTP call finishes when the stream is fully read\n                if (this.offloaderStats != null) {\n                    this.offloaderStats.recordReadOffloadDataLatency(topicName,\n                            System.nanoTime() - startReadTime, TimeUnit.NANOSECONDS);\n                    this.offloaderStats.recordReadOffloadBytes(topicName, endRange - startRange + 1);\n                }\n            } catch (Throwable e) {\n                if (null != this.offloaderStats) {\n                    this.offloaderStats.recordReadOffloadError(this.topicName);\n                }\n                // If the blob is not found, the original exception is thrown and handled by the caller.\n                if (e instanceof KeyNotFoundException) {\n                    throw e;\n                }\n                throw new IOException(\"Error reading from BlobStore\", e);\n            }\n        }\n        return true;\n    }\n\n    void fillBuffer(InputStream is, int bytesToCopy) throws IOException {\n        while (bytesToCopy > 0) {\n            int writeBytes = buffer.writeBytes(is, bytesToCopy);\n            if (writeBytes < 0) {\n                break;\n            }\n            bytesToCopy -= writeBytes;\n        }\n    }\n\n    ByteBuf getBuffer() {\n        return buffer;\n    }","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java#L110-L146","documentation":"BlobStoreBackedInputStreamImpl.read() serves reads from an in-memory buffer that is refilled with ranged GETs from the object store (S3/GCS/Azure). When a refill fails with any exception other than KeyNotFoundException, the original cause is wrapped in an IOException(\"Error reading from BlobStore\", e) and propagated to the reader.","triggerScenarios":"refillBufferIfNeeded performs a BlobStore read via readBuffer and the underlying object-store client throws (throttling, connection reset, timeout, HTTP 5xx); KeyNotFoundException is deliberately rethrown unchanged, everything else gets this wrapped IOException.","commonSituations":"S3/GCS rate limiting or request throttling under read load; expired or revoked cloud credentials; network partition between broker and object store; bucket policy or firewall blocking the broker; transient 503 from the storage backend.","solutions":["Inspect the wrapped cause (IOException#getCause) — fix the specific storage error (credentials, permissions, throttling)","Re-read the offloaded segment; BlobStore reads are retryable for transient storage/network faults","Configure client retry/timeout parameters in the offload driver config (e.g. s3ManagedLedgerOffloadServiceEndpoint, connection limits)","Restore connectivity/credentials to the bucket; verify bucket name, region, and access policy"],"exampleFix":"// handling the wrap\ntry { stream.read(buf); }\ncatch (IOException e) {\n  Throwable cause = e.getCause();\n  if (cause instanceof SdkClientException) retryWithBackoff(); // transient storage fault\n  else throw e;\n}","handlingStrategy":"retry","validationCode":"// preflight: confirm object reachable before streaming\ntry (InputStream probe = blobStore.readBlob(objectKey)) {\n  probe.read(); // throws early if bucket/creds are broken\n}","typeGuard":null,"tryCatchPattern":"try {\n  int n = stream.read(buf);\n} catch (IOException e) {\n  if (e.getCause() instanceof KeyNotFoundException) {\n    throw e; // blob gone: do not retry\n  }\n  if (isTransient(e.getCause())) retryWithBackoff(); // throttle/network\n  else throw e;\n}","preventionTips":["Configure SDK-level retries and timeouts for the object-store client","Rotate credentials before expiry; monitor auth failures","Alert on storage-side throttling (S3 503 SlowDown, GCS rate limits)","Keep network paths between broker and bucket healthy/monitored"],"tags":["io","object-storage","network","tiered-storage"],"backgroundTag":"blobstore-read-failure","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}