apache/pulsar · error · RuntimeException

Failed to increment key '%s' by amount '%s'

Error message

Failed to increment key '%s' by amount '%s'

What it means

BKStateStoreImpl.incrCounter blocks on the async increment via result(incrCounterAsync) and wraps any failure in a RuntimeException carrying the key and amount. It indicates the state-table increment against BookKeeper-backed state storage failed, e.g. due to table service errors, timeouts, or invalid input.

Source

Thrown at pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreImpl.java:98

    @Override
    public void close() {
        table.close();
    }

    @Override
    public CompletableFuture<Void> incrCounterAsync(String key, long amount) {
        // TODO: this can be optimized with a batch operation.
        return table.increment(
            Unpooled.wrappedBuffer(key.getBytes(UTF_8)),
            amount);
    }

    @Override
    public void incrCounter(String key, long amount) {
        try {
            result(incrCounterAsync(key, amount));
        } catch (Exception e) {
            throw new RuntimeException("Failed to increment key '" + key + "' by amount '" + amount + "'", e);
        }
    }

    @Override
    public CompletableFuture<Long> getCounterAsync(String key) {
        return table.getNumber(Unpooled.wrappedBuffer(key.getBytes(UTF_8)));
    }

    @Override
    public long getCounter(String key) {
        try {
            return result(getCounterAsync(key));
        } catch (Exception e) {
            throw new RuntimeException("Failed to retrieve counter from key '" + key + "'");
        }
    }

    @Override

View on GitHub (pinned to 820761864e)

Solutions

1. Check the cause exception for the underlying table/BookKeeper error (timeout, NotFound, connection refused).
2. Verify the state table exists for the function (tenant/namespace/name) and was initialized successfully.
3. Check BookKeeper storage cluster health and the table service (proxy) availability.
4. Retry the increment; incrCounter is a read-modify-write at the table layer, ensure retries are idempotent for your use case.

Example fix

// before
store.incrCounter("hits", 1); // throws raw RuntimeException on any table failure
// after
try {
    store.incrCounter("hits", 1);
} catch (RuntimeException e) {
    log.error("counter incr failed, will retry", e);
    retryIncr("hits", 1);
}

// before using state
StateStore store = context.getStateStore(stateName); // ensure initialized
// check table reachable via a cheap read
store.getCounter("__health__"); // throws early if storage is down

try {
    store.incrCounter(key, amount);
} catch (RuntimeException e) {
    // transient table-service failure: retry with backoff
    backoffRetry(() -> store.incrCounter(key, amount), 3);
}

When it happens

Trigger: Calling incrCounter(key, amount) when the underlying table incr fails: state table not yet created, storage service unreachable, timeout, or null/invalid key handling in the table service.

Common situations: BookKeeper storage cluster degraded or table service proxy down; function state configured but state table creation failed; network partition between function instance and storage.

Related errors

• Failed to retrieve counter from key '%s'
• Failed to update the state value for key '%s'
• Failed to delete the state value for key '%s'
• Failed to retrieve the state value for key '%s'
• Failed to retrieve the state value for key '${key}'

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/8578b6be8f8bf901. Report an issue: GitHub.