{"record":{"id":"fc4e4a826c4f5b59","repo":"apache/pulsar","slug":"invalid-object-version-s-for-s-expect-s","errorCode":null,"errorMessage":"Invalid object version %s for %s, expect %s","messagePattern":"Invalid object version (.+?) for (.+?), expect (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/DataBlockUtils.java","lineNumber":75,"sourceCode":"        return String.format(\"%s-ledger-%d-index\", uuid.toString(), ledgerId);\n    }\n\n    public static String indexBlockOffloadKey(UUID uuid) {\n        return String.format(\"%s-index\", uuid.toString());\n    }\n\n    public static void addVersionInfo(BlobBuilder blobBuilder, Map<String, String> userMetadata) {\n        ImmutableMap.Builder<String, String> metadataBuilder = ImmutableMap.builder();\n        metadataBuilder.putAll(userMetadata);\n        metadataBuilder.put(METADATA_FORMAT_VERSION_KEY.toLowerCase(), CURRENT_VERSION);\n        blobBuilder.userMetadata(metadataBuilder.build());\n    }\n\n    public static final VersionCheck VERSION_CHECK = (key, blob) -> {\n        // NOTE all metadata in jclouds comes out as lowercase, in an effort to normalize the providers\n        String version = blob.getMetadata().getUserMetadata().get(METADATA_FORMAT_VERSION_KEY.toLowerCase());\n        if (version == null || !version.equals(CURRENT_VERSION)) {\n            throw new IOException(String.format(\"Invalid object version %s for %s, expect %s\",\n                version, key, CURRENT_VERSION));\n        }\n    };\n\n    public static Long parseLedgerId(String name) {\n        if (name == null || name.isEmpty()) {\n            return null;\n        }\n        if (name.endsWith(\"-index\")) {\n            name = name.substring(0, name.length() - \"-index\".length());\n        }\n        int pos = name.indexOf(\"-ledger-\");\n        if (pos < 0) {\n            return null;\n        }\n        try {\n            return Long.parseLong(name.substring(pos + 8));\n        } catch (NumberFormatException err) {","sourceCodeStart":57,"sourceCodeEnd":93,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/DataBlockUtils.java#L57-L93","documentation":"DataBlockUtils.VERSION_CHECK validates that a jclouds blob carries user metadata 'StringFormatVersion' (lowercased key) equal to the CURRENT_VERSION constant. If the metadata is missing or holds a different version string, the object was not written by (or is not compatible with) this offload implementation, and an IOException is thrown by the version-check lambda used when reading offloaded data.","triggerScenarios":"Reading an offloaded ledger object whose blob user-metadata lacks the format-version key, or whose value differs from DataBlockUtils.CURRENT_VERSION — typically objects written by an older Pulsar/BookKeeper offload format or objects manually created/uploaded without the metadata.","commonSituations":"Upgrading Pulsar and reading ledgers offloaded by a previous format version; manual object-storage migration tools that strip user metadata (some S3-compatible proxies drop user metadata); hand-crafted blobs in the offload bucket.","solutions":["Check the blob's user metadata (key lowercased, e.g. 'stringformatversion') and confirm it matches DataBlockUtils.CURRENT_VERSION","Re-offload the affected ledgers with the current broker version so the correct format-version metadata is written","Use a migration tool or compatible reader for objects written by older offload formats instead of reading them directly","If a proxy/storage layer strips user metadata, configure it to preserve object user metadata"],"exampleFix":"// before: manual upload loses metadata\nbucket.put(key, payloadStream);\n// after: preserve format metadata on upload\nBucketMetadata bm = new BucketMetadata();\nbm.putMetadata(DataBlockUtils.METADATA_FORMAT_VERSION_KEY, DataBlockUtils.CURRENT_VERSION);\nbucket.putBlob(key, new BlobImpl(key, payloadStream, bm));","handlingStrategy":"validation","validationCode":"String ver = blob.getMetadata().getUserMetadata()\n    .get(DataBlockUtils.METADATA_FORMAT_VERSION_KEY.toLowerCase());\nif (!DataBlockUtils.CURRENT_VERSION.equals(ver)) {\n    throw new IOException(\"Unsupported offload format version: \" + ver);\n}","typeGuard":null,"tryCatchPattern":"try {\n    VERSION_CHECK.check(key, blob);\n    readOffloadedData(key, blob);\n} catch (IOException e) {\n    if (e.getMessage().startsWith(\"Invalid object version\")) {\n        // handle old-format object: use legacy reader or re-offload\n    } else { throw e; }\n}","preventionTips":["Never copy offload blobs with tools that drop user metadata","Pin and document the offload format version for the cluster","Test offload reads after every Pulsar upgrade with pre-upgrade data","Alert on blobs in the offload bucket missing the format-version metadata"],"tags":["versioning","metadata","tiered-storage","compatibility"],"backgroundTag":"format-version-mismatch","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}