apache/pulsar · error · IOException

Invalid object version %s for %s, expect %s

Error message

Invalid object version %s for %s, expect %s

What it means

DataBlockUtils.VERSION_CHECK validates that a jclouds blob carries user metadata 'StringFormatVersion' (lowercased key) equal to the CURRENT_VERSION constant. If the metadata is missing or holds a different version string, the object was not written by (or is not compatible with) this offload implementation, and an IOException is thrown by the version-check lambda used when reading offloaded data.

Source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/DataBlockUtils.java:75

        return String.format("%s-ledger-%d-index", uuid.toString(), ledgerId);
    }

    public static String indexBlockOffloadKey(UUID uuid) {
        return String.format("%s-index", uuid.toString());
    }

    public static void addVersionInfo(BlobBuilder blobBuilder, Map<String, String> userMetadata) {
        ImmutableMap.Builder<String, String> metadataBuilder = ImmutableMap.builder();
        metadataBuilder.putAll(userMetadata);
        metadataBuilder.put(METADATA_FORMAT_VERSION_KEY.toLowerCase(), CURRENT_VERSION);
        blobBuilder.userMetadata(metadataBuilder.build());
    }

    public static final VersionCheck VERSION_CHECK = (key, blob) -> {
        // NOTE all metadata in jclouds comes out as lowercase, in an effort to normalize the providers
        String version = blob.getMetadata().getUserMetadata().get(METADATA_FORMAT_VERSION_KEY.toLowerCase());
        if (version == null || !version.equals(CURRENT_VERSION)) {
            throw new IOException(String.format("Invalid object version %s for %s, expect %s",
                version, key, CURRENT_VERSION));
        }
    };

    public static Long parseLedgerId(String name) {
        if (name == null || name.isEmpty()) {
            return null;
        }
        if (name.endsWith("-index")) {
            name = name.substring(0, name.length() - "-index".length());
        }
        int pos = name.indexOf("-ledger-");
        if (pos < 0) {
            return null;
        }
        try {
            return Long.parseLong(name.substring(pos + 8));
        } catch (NumberFormatException err) {

View on GitHub (pinned to 820761864e)

Solutions

1. Check the blob's user metadata (key lowercased, e.g. 'stringformatversion') and confirm it matches DataBlockUtils.CURRENT_VERSION
2. Re-offload the affected ledgers with the current broker version so the correct format-version metadata is written
3. Use a migration tool or compatible reader for objects written by older offload formats instead of reading them directly
4. If a proxy/storage layer strips user metadata, configure it to preserve object user metadata

Example fix

// before: manual upload loses metadata
bucket.put(key, payloadStream);
// after: preserve format metadata on upload
BucketMetadata bm = new BucketMetadata();
bm.putMetadata(DataBlockUtils.METADATA_FORMAT_VERSION_KEY, DataBlockUtils.CURRENT_VERSION);
bucket.putBlob(key, new BlobImpl(key, payloadStream, bm));

String ver = blob.getMetadata().getUserMetadata()
    .get(DataBlockUtils.METADATA_FORMAT_VERSION_KEY.toLowerCase());
if (!DataBlockUtils.CURRENT_VERSION.equals(ver)) {
    throw new IOException("Unsupported offload format version: " + ver);
}

try {
    VERSION_CHECK.check(key, blob);
    readOffloadedData(key, blob);
} catch (IOException e) {
    if (e.getMessage().startsWith("Invalid object version")) {
        // handle old-format object: use legacy reader or re-offload
    } else { throw e; }
}

When it happens

Trigger: Reading an offloaded ledger object whose blob user-metadata lacks the format-version key, or whose value differs from DataBlockUtils.CURRENT_VERSION — typically objects written by an older Pulsar/BookKeeper offload format or objects manually created/uploaded without the metadata.

Common situations: Upgrading Pulsar and reading ledgers offloaded by a previous format version; manual object-storage migration tools that strip user metadata (some S3-compatible proxies drop user metadata); hand-crafted blobs in the offload bucket.

Related errors

• Timeout during update managedLedger's properties
• The '${offloaderName}' offloader does not provide an offload
• Class ${factoryClass} does not implement interface ${interfa
• Failed to initialize BookKeeper metadata
• Error while scanning ledgers for ${namespaceName}

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/fc4e4a826c4f5b59. Report an issue: GitHub.