{"record":{"id":"3b865ab75fc30531","repo":"mudler/LocalAI","slug":"w-d-d-is-fully-consumed-by-d-running-backend","errorCode":null,"errorMessage":"%w: %d-%d is fully consumed by %d running backend(s) and %d port(s) still in quarantine; raise LOCALAI_GRPC_MAX_PORT to widen the range","messagePattern":"%w: (.+?)-(.+?) is fully consumed by (.+?) running backend\\(s\\) and (.+?) port\\(s\\) still in quarantine; raise LOCALAI_GRPC_MAX_PORT to widen the range","errorType":"exception","errorClass":"ErrNoFreePort","httpStatus":null,"severity":"error","filePath":"core/services/worker/supervisor.go","lineNumber":305,"sourceCode":"\t}\n\n\t// 3. Grow into ports never handed out, staying inside the range.\n\tif s.nextPort >= minPort && s.nextPort <= maxPort {\n\t\tport := s.nextPort\n\t\ts.nextPort++\n\t\treturn s.claimPort(key, port), nil\n\t}\n\n\t// 4. Steal another key's port rather than refuse to start a backend.\n\tif len(s.freePorts) > 0 {\n\t\tport := s.freePorts[len(s.freePorts)-1]\n\t\ts.freePorts = s.freePorts[:len(s.freePorts)-1]\n\t\txlog.Warn(\"gRPC port range is exhausted; reusing a port that belonged to another backend. A stale controller row for the previous owner could briefly misroute to this backend — raise LOCALAI_GRPC_MAX_PORT to restore headroom\",\n\t\t\t\"backend\", key, \"port\", port, \"previousOwner\", owners[port], \"min\", minPort, \"max\", maxPort)\n\t\treturn s.claimPort(key, port), nil\n\t}\n\n\treturn 0, fmt.Errorf(\"%w: %d-%d is fully consumed by %d running backend(s) and %d port(s) still in quarantine; raise LOCALAI_GRPC_MAX_PORT to widen the range\",\n\t\tErrNoFreePort, minPort, maxPort, len(s.processes), len(s.quarantinedPorts))\n}\n\n// sweepAffinity drops claims whose window has lapsed, so their ports become\n// ordinary free ports again. Swept lazily on allocation for the same reason as\n// sweepQuarantine: the only observer is allocation itself, so a timer goroutine\n// per released port would buy nothing. Callers must hold s.mu.\nfunc (s *backendSupervisor) sweepAffinity() {\n\tif len(s.portAffinity) == 0 {\n\t\treturn\n\t}\n\tnow := time.Now()\n\tfor key, own := range s.portAffinity {\n\t\tif own.expired(now) {\n\t\t\tdelete(s.portAffinity, key)\n\t\t}\n\t}\n}","sourceCodeStart":287,"sourceCodeEnd":323,"githubUrl":"https://github.com/mudler/LocalAI/blob/44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26/core/services/worker/supervisor.go#L287-L323","documentation":"Hard allocation failure from the same supervisor: the gRPC port range is fully consumed — every port is either held by one of len(s.processes) running backends or sitting in len(s.quarantinedPorts) quarantine (ports held back after a backend died, pending a sweep). The returned error wraps ErrNoFreePort with the exact counts, so Go callers can match it with errors.Is. Backend startup is refused.","triggerScenarios":"Requesting a new backend when s.nextPort is outside [minPort,maxPort] AND freePorts is empty — i.e. running backend count + quarantined ports covers the whole range. Typical with a tight default range, several loaded models, and recent crashes whose ports are still quarantined.","commonSituations":"Crash-looping backends filling quarantine; a small LOCALAI_GRPC port range on a busy box; loading a new model when the range was sized for fewer replicas; stale supervisor state after many restarts within a short window.","solutions":["Raise LOCALAI_GRPC_MAX_PORT (the message's own remedy) to widen the range, then retry loading the model.","Free ports by unloading/stopping backends you do not need right now.","If quarantine is the culprit (the error prints the count), wait for the quarantine window to lapse and be swept on the next allocation, or restart LocalAI to reset allocator state.","Set the range to at least (max concurrent backends + expected quarantine burst) in your service unit/env before starting LocalAI."],"exampleFix":"# before\nLOCALAI_GRPC_MAX_PORT=50016  # fully consumed by 14 running backend(s) and 2 port(s) still in quarantine\n\n# after\nLOCALAI_GRPC_MAX_PORT=50100  # wide enough for peak backends plus quarantine headroom","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"port, err := supervisor.StartBackend(key)\nif err != nil {\n    if errors.Is(err, worker.ErrNoFreePort) {\n        // free backends or wait out quarantine, then retry once headroom exists\n        time.Sleep(quarantineWindow)\n        port, err = supervisor.StartBackend(key)\n    }\n    if err != nil {\n        return fmt.Errorf(\"cannot start backend %s: %w\", key, err)\n    }\n}","preventionTips":["Set LOCALAI_GRPC_MAX_PORT from your max-concurrent-backend SLO, not the default.","Parse the running/quarantined counts out of the error to decide between waiting (quarantine-heavy) and widening (capacity-heavy).","Restart LocalAI to reset quarantine state after a crash loop, then re-check the range size."],"tags":["go","grpc","ports","resource-exhaustion","supervisor","localai"],"backgroundTag":null,"analyzedSha":"44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26","analyzedAt":"2026-08-15T10:13:50.291Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}