apache/pulsar · warning · RestException

Leader not yet ready. Please retry again

Error message

Leader not yet ready. Please retry again

What it means

HTTP 503 thrown by updateFunctionOnWorkerLeader when this worker is not the leader and either no leader is currently known (workerInfo == null) or the 'leader' recorded is this very worker (membership state briefly inconsistent). The worker cannot serve the update and cannot redirect, so it asks the client to retry.

Source

Thrown at pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/FunctionsImpl.java:734

            throw new RestException(Response.Status.BAD_REQUEST, "Namespace is not provided");
        }
        if (functionName == null) {
            throw new RestException(Response.Status.BAD_REQUEST, "Function name is not provided");
        }
        FunctionMetaData functionMetaData;
        try {
            byte[] data = uploadedInputStream.readAllBytes();
            functionMetaData = new FunctionMetaData();
            functionMetaData.parseFrom(data);
        } catch (IOException e) {
            throw new RestException(Response.Status.BAD_REQUEST, "Corrupt Function MetaData");
        }

        // Redirect if we are not the leader
        if (!worker().getLeaderService().isLeader()) {
            WorkerInfo workerInfo = worker().getMembershipManager().getLeader();
            if (workerInfo == null || workerInfo.getWorkerId().equals(worker().getWorkerConfig().getWorkerId())) {
                throw new RestException(Response.Status.SERVICE_UNAVAILABLE,
                        "Leader not yet ready. Please retry again");
            }
            URI redirect = UriBuilder.fromUri(uri).host(workerInfo.getWorkerHostname())
                    .port(workerInfo.getPort()).build();
            throw new WebApplicationException(Response.temporaryRedirect(redirect).build());
        }

        // Its possible that we are not the leader anymore. That will be taken care of by FunctionMetaDataManager
        FunctionMetaDataManager functionMetaDataManager = worker().getFunctionMetaDataManager();
        try {
            functionMetaDataManager.updateFunctionOnLeader(functionMetaData, delete);
        } catch (IllegalStateException e) {
            throw new RestException(Response.Status.INTERNAL_SERVER_ERROR, e.getMessage());
        } catch (IllegalArgumentException e) {
            throw new RestException(Response.Status.BAD_REQUEST, e.getMessage());
        }
    }

View on GitHub (pinned to 820761864e)

Solutions

  1. Retry the request with backoff until a leader is elected (the message explicitly says 'Please retry again').
  2. Query the cluster's current leader via the /leaderBroker or worker membership endpoints and route the update there.
  3. Check worker logs and metadata-store connectivity if the condition persists beyond a few seconds.

Example fix

// before
Response resp = client.post(updateUrl, body); // fails with 503 during election
// after
Response resp;
int attempts = 0;
do {
    resp = client.post(updateUrl, body);
    if (resp.status() == 503) {
        Thread.sleep(1000L * ++attempts);
    }
} while (resp.status() == 503 && attempts < 5);
Defensive patterns

Strategy: retry

Try / catch

try {
    client.post(updateUrl, body);
} catch (ApiException e) {
    if (e.code() == 503 && e.body().contains("Leader not yet ready")) {
        sleepWithBackoff();
        retryUpTo(5);
    }
}

Prevention

When it happens

Trigger: POSTing an updateOnLeader request to a follower worker during leader election, right after the previous leader crashed, or during a cluster split where the membership manager has no fresh leader entry.

Common situations: Hitting a non-leader worker directly because a load balancer routed the request there; submitting function updates immediately after a worker restart or rolling upgrade; ZooKeeper/metadata-store hiccup delaying leader publication; small clusters where leadership is still being established at startup.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/d7c1c23ceb311466. Report an issue: GitHub.