What causes "Failed to retrieve counter from key '%s'" in apache/pulsar?

Since the cause is swallowed, check instance logs for the async failure logged at the table layer.

apache/pulsar · error · RuntimeException

Failed to retrieve counter from key '%s'

Error message

Failed to retrieve counter from key '%s'

What it means

BKStateStoreImpl.getCounter blocks on getCounterAsync and rethrows any failure as a RuntimeException with this message. Note the cause is intentionally dropped here (no `e` passed), so only the message is available in logs. It means reading the counter value for the key from the BookKeeper-backed state table failed.

Source

Thrown at pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreImpl.java:112

    public void incrCounter(String key, long amount) {
        try {
            result(incrCounterAsync(key, amount));
        } catch (Exception e) {
            throw new RuntimeException("Failed to increment key '" + key + "' by amount '" + amount + "'", e);
        }
    }

    @Override
    public CompletableFuture<Long> getCounterAsync(String key) {
        return table.getNumber(Unpooled.wrappedBuffer(key.getBytes(UTF_8)));
    }

    @Override
    public long getCounter(String key) {
        try {
            return result(getCounterAsync(key));
        } catch (Exception e) {
            throw new RuntimeException("Failed to retrieve counter from key '" + key + "'");
        }
    }

    @Override
    public CompletableFuture<Void> putAsync(String key, ByteBuffer value) {
        if (value != null) {
            // Set position to off the buffer to the beginning.
            // If a user used an operation like ByteBuffer.allocate(4).putInt(count)
            // to create a ByteBuffer to store to the state store
            // the position of the buffer will be at the end and nothing will be written to table service
            value.position(0);
            return table.put(
                    Unpooled.wrappedBuffer(key.getBytes(UTF_8)),
                    Unpooled.wrappedBuffer(value));
        } else {
            return table.put(
                    Unpooled.wrappedBuffer(key.getBytes(UTF_8)),
                    null);

View on GitHub (pinned to 820761864e)

Solutions

Since the cause is swallowed, check instance logs for the async failure logged at the table layer.
Verify the state table exists and the storage cluster is reachable.
Call getCounterAsync instead to get the real completion exception from the future.
Initialize the counter with incrCounter(key, 0) before reading if your workflow requires it to exist.

Example fix

// before
long c = store.getCounter("hits"); // message loses cause
// after
long c = store.getCounterAsync("hits").exceptionally(ex -> {
    log.error("getCounter failed", ex);
    return 0L;
}).join();

Defensive patterns

Strategy: fallback

Validate before calling

// initialize the counter before reading if it may not exist
store.incrCounter("hits", 0); // no-op if the table supports it, ensures key path works

Try / catch

long hits;
try {
    hits = store.getCounter("hits");
} catch (RuntimeException e) {
    // cause is swallowed; log and fall back to async to capture it
    hits = store.getCounterAsync("hits").handle((v, ex) -> {
        if (ex != null) { log.error("counter read failed", ex); return 0L; }
        return v;
    }).join();
}

Prevention

Prefer the Async variants when you need the underlying exception — the sync wrapper drops the cause.
Initialize counters before first read.
Alert on storage-cluster health instead of per-call exceptions.

When it happens

Trigger: Calling getCounter(key) when the async table getNumber fails — storage service error, timeout, connection failure, or the table containing the key is unavailable. Also thrown if result() unwraps a failed future of any kind.

Common situations: State table not initialized; BookKeeper/table-proxy outage; deserialization/number-format issues on the stored value; first read of a never-incremented key in a failing cluster.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/c7cd3862d82e33f2. Report an issue: GitHub.