What causes "Internal Error updating function at the leader" in apache/pulsar?

Inspect the wrapped cause (e.getCause()) — the fix depends on it (stale version vs producer/topic failure).

apache/pulsar · error · IllegalStateException

Internal Error updating function at the leader

Error message

Internal Error updating function at the leader

What it means

updateFunctionOnLeader processes the metadata update (register/update/delete) and writes it into the function metadata topic via processUpdate/processDeregister. Any exception thrown during that processing is wrapped in IllegalStateException("Internal Error updating function at the leader", e), indicating the leader failed to persist the change — the original cause is attached.

Source

Thrown at pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionMetaDataManager.java:246

            serviceRequest.setRequestId(UUID.randomUUID().toString());
            toWrite = serviceRequest.toByteArray();
        }
        try {
            TypedMessageBuilder<byte[]> builder = exclusiveLeaderProducer.newMessage()
                    .value(toWrite)
                    .property(versionTag, Long.toString(functionMetaData.getVersion()));
            if (workerConfig.getUseCompactedMetadataTopic()) {
                builder = builder.key(FunctionCommon.getFullyQualifiedName(functionMetaData.getFunctionDetails()));
            }
            lastMessageSeen = builder.send();
            if (delete) {
                needsScheduling = processDeregister(functionMetaData);
            } else {
                needsScheduling = processUpdate(functionMetaData);
            }
        } catch (Exception e) {
            log.error().exception(e).log("Could not write into Function Metadata topic");
            throw new IllegalStateException("Internal Error updating function at the leader", e);
        }

        if (needsScheduling) {
            this.schedulerManager.schedule();
        }
    }

    private void checkRequestOutDated(FunctionMetaData functionMetaData, boolean delete) {
        FunctionDetails details = functionMetaData.getFunctionDetails();
        if (isRequestOutdated(details.getTenant(), details.getNamespace(),
                details.getName(), functionMetaData.getVersion())) {
            log.debug().attr("tenant", details.getTenant())
                    .attr("namespace", details.getNamespace())
                    .attr("functionName", details.getName())
                    .attr("version", functionMetaData.getVersion())
                    .log("Ignoring outdated request version");
            if (delete) {
                throw new IllegalArgumentException(

View on GitHub (pinned to 820761864e)

Solutions

Inspect the wrapped cause (e.getCause()) — the fix depends on it (stale version vs producer/topic failure).
If the cause is the 'out of date' IllegalArgumentException, re-fetch the current function metadata and retry with the fresh version.
Verify the function metadata topic exists and the broker/bookies backing it are healthy.
Retry the update after the leader's producer reconnects; check leader worker logs for the 'Could not write into Function Metadata topic' error for the root cause.

Example fix

// before
manager.updateFunctionOnLeader(meta, false);
// after
try {
    manager.updateFunctionOnLeader(meta, false);
} catch (IllegalStateException e) {
    Throwable cause = e.getCause();
    if (cause instanceof IllegalArgumentException) {
        FunctionMetaData fresh = fetchCurrentMetaData(tenant, namespace, functionName);
        manager.updateFunctionOnLeader(fresh, false);
    } else {
        throw e;
    }
}

Defensive patterns

Strategy: try-catch

Validate before calling

// verify metadata topic health before writing
TopicName metaTopic = TopicName.get("persistent://public/functions/metadata");
admin.topics().getStats(metaTopic.toString()); // throws if topic/broker unavailable

Try / catch

try {
    manager.updateFunctionOnLeader(meta, delete);
} catch (IllegalStateException e) {
    Throwable cause = e.getCause();
    if (cause instanceof IllegalArgumentException) {
        refreshAndRetry(meta, delete);
    } else {
        throw new FunctionUpdateException(cause); // topic/storage problem
    }
}

Prevention

Always inspect the wrapped cause before deciding on a fix.
Keep the function metadata topic and its backing bookies healthy/monitored.
Use read-modify-write with version increment for all metadata updates.

When it happens

Trigger: Calling updateFunctionOnLeader where processUpdate/processDeregister throws — e.g. the update is outdated (IllegalArgumentException from processUpdate), topic producer failures, serialization errors, or scheduler state corruption while writing to the function metadata topic.

Common situations: Metadata topic producer is disconnected or the topic is unavailable; concurrent updates with stale versions triggering the outdated-request path; worker bookie/storage problems making the topic write fail.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/8b2d3735ab27c353. Report an issue: GitHub.