apache/pulsar · error · WebApplicationException

Externally managed schedulers can't do per instance stop

Error message

Externally managed schedulers can't do per instance stop

What it means

FunctionRuntimeManager.restartFunctionInstance refuses to restart a single function instance when the runtime factory is externally managed (e.g. Kubernetes). With external schedulers, instance placement/restarts belong to the external scheduler, so the manager answers HTTP 505/501-style NOT_IMPLEMENTED wrapped in a WebApplicationException with the message 'Externally managed schedulers can't do per instance stop'.

Source

Thrown at pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java:363

        return assignments;
    }

    /**
     * Removes a collection of assignments.
     *
     * @param assignments assignments to remove
     */
    public synchronized void removeAssignments(Collection<Assignment> assignments) {
        for (Assignment assignment : assignments) {
            this.deleteAssignment(assignment);
        }
    }

    public synchronized void restartFunctionInstance(String tenant, String namespace,
                                                     String functionName, int instanceId, URI uri) throws Exception {
        if (runtimeFactory.externallyManaged()) {
            throw new WebApplicationException(Response.serverError().status(Status.NOT_IMPLEMENTED)
                    .type(MediaType.APPLICATION_JSON)
                    .entity(new ErrorData("Externally managed schedulers can't do per instance stop")).build());
        }
        Assignment assignment = this.findAssignment(tenant, namespace, functionName, instanceId);
        final String fullFunctionName = String.format("%s/%s/%s/%s", tenant, namespace, functionName, instanceId);
        if (assignment == null) {
            throw new WebApplicationException(Response.serverError().status(Status.BAD_REQUEST)
                    .type(MediaType.APPLICATION_JSON)
                    .entity(new ErrorData(fullFunctionName + " doesn't exist")).build());
        }

        final String assignedWorkerId = assignment.getWorkerId();
        final String workerId = this.workerConfig.getWorkerId();

        if (assignedWorkerId.equals(workerId)) {
            stopFunction(FunctionCommon.getFullyQualifiedInstanceId(assignment.getInstance()), true);
            return;
        } else {

View on GitHub (pinned to 820761864e)

Solutions

  1. Restart the whole function instead of a single instance (restartFunctionInstances / restart all instances endpoint).
  2. Delete/restart the individual Kubernetes pod backing the instance (kubectl delete pod <instance-pod>) — the external scheduler owns instance lifecycle.
  3. Switch the worker's function runtime factory to process/thread mode if you truly need per-instance control (not usually desirable in production).
  4. Handle the 501 NOT_IMPLEMENTED response in client code and fall back to full-function restart.

Example fix

// before
admin.functions().restartFunctionInstance(tenant, namespace, functionName, instanceId);
// after
try {
    admin.functions().restartFunctionInstance(tenant, namespace, functionName, instanceId);
} catch (PulsarAdminException e) {
    if (e.getStatusCode() == 501) {
        // externally managed: restart all instances instead
        admin.functions().restartFunction(tenant, namespace, functionName);
    } else {
        throw e;
    }
}
Defensive patterns

Strategy: validation

Validate before calling

FunctionRuntimeInfo rt = admin.functions().getRuntimeInfo(tenant, ns, fn);
// externally managed (K8s) deployments do not support per-instance restart;
// fall back to full-function restart instead of calling restartFunctionInstance

Try / catch

try {
    admin.functions().restartFunctionInstance(tenant, ns, fn, instanceId);
} catch (PulsarAdminException e) {
    if (e.getStatusCode() == 501) {
        admin.functions().restartFunction(tenant, ns, fn);
    } else {
        throw e;
    }
}

Prevention

When it happens

Trigger: Calling the per-instance restart admin API (restartFunctionInstance with a specific instanceId) against a worker whose function runtime factory is externally managed (runtimeFactory.externallyManaged() == true, e.g. kubernetes runtime).

Common situations: Running Pulsar Functions on Kubernetes but calling the standalone-runtime per-instance restart REST endpoint; automation/scripts written for thread/process runtime being reused on a K8s deployment; admin UI invoking per-instance restart buttons on an externally managed cluster.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/690d25a6ed60df0b. Report an issue: GitHub.