{"record":{"id":"ffd97e870a46b1b0","repo":"weaviate/weaviate","slug":"startup-is-not-complete","errorCode":null,"errorMessage":"startup is not complete","messagePattern":"startup is not complete","errorType":"http","errorClass":null,"httpStatus":503,"severity":"error","filePath":"adapters/handlers/rest/clusterapi/indices.go","lineNumber":544,"sourceCode":"\t\t\t\thttp.StatusBadRequest)\n\t\t\treturn\n\t\t}\n\n\t\tvar additional additional.Properties\n\t\tif err := json.Unmarshal(additionalBytes, &additional); err != nil {\n\t\t\thttp.Error(w, \"unmarshal 'additional' param from json: \"+err.Error(),\n\t\t\t\thttp.StatusBadRequest)\n\t\t\treturn\n\t\t}\n\n\t\tvar selectProperties search.SelectProperties\n\t\tif err := json.Unmarshal(selectPropertiesBytes, &selectProperties); err != nil {\n\t\t\thttp.Error(w, \"unmarshal 'selectProperties' param from json: \"+err.Error(),\n\t\t\t\thttp.StatusBadRequest)\n\t\t\treturn\n\t\t}\n\t\tif !i.db.StartupComplete() {\n\t\t\thttp.Error(w, \"startup is not complete\", http.StatusServiceUnavailable)\n\t\t\treturn\n\t\t}\n\n\t\ti.logger.WithFields(logrus.Fields{\n\t\t\t\"shard\":  shard,\n\t\t\t\"action\": \"GetObject\",\n\t\t}).Debug(\"getting object ...\")\n\n\t\tobj, err := i.shards.GetObject(r.Context(), index, shard, strfmt.UUID(id),\n\t\t\tselectProperties, additional)\n\t\tif err != nil {\n\t\t\thttp.Error(w, err.Error(), http.StatusInternalServerError)\n\t\t\treturn\n\t\t}\n\n\t\tif obj == nil {\n\t\t\t// this is a legitimate case - the requested ID doesn't exist, don't try\n\t\t\t// to marshal anything","sourceCodeStart":526,"sourceCodeEnd":562,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/handlers/rest/clusterapi/indices.go#L526-L562","documentation":"Before serving the internal GET object request, the handler checks i.db.StartupComplete(). If the database has not finished startup (recovery, shard loading, schema init), it responds \"startup is not complete\" with HTTP 503 Service Unavailable. This is a deliberate guard, not a bug — the node cannot safely serve reads yet.","triggerScenarios":"Any GET object request on the internal cluster API sent to a node that is still starting up: after a restart with large LSM stores still recovering, during shard migration/replication to a freshly joined node, or immediately after container start before health checks pass.","commonSituations":"Kubernetes rolling updates where replica-set clients hit a not-yet-ready pod, orchestration sending traffic before readiness, or replication tooling racing node boot after a crash.","solutions":["Retry with backoff until the node reports ready (monitor /v1/.well-known/ready or the metrics endpoint).","Gate replication traffic on the node's readiness probe instead of container liveness.","Increase startup resources/timeouts if startup is genuinely slow (large disks, many shards).","Check logs for slow startup phases (recovery, shard init) to size readiness timeouts correctly."],"exampleFix":"// before\nresp := http.Get(internalURL) // fires immediately after pod start\n// after\nwaitForReady(nodeHost, 5*time.Minute) // poll /v1/.well-known/ready with backoff\nresp := http.Get(internalURL)","handlingStrategy":"retry","validationCode":"// Gate requests on the node's readiness before calling the internal API\nfunc nodeReady(host string) bool {\n    resp, err := http.Get(host + \"/v1/.well-known/ready\")\n    return err == nil && resp.StatusCode == http.StatusOK\n}","typeGuard":null,"tryCatchPattern":"if resp.StatusCode == http.StatusServiceUnavailable {\n    // retry with exponential backoff until startup completes\n    return backoff.Retry(func() error {\n        r, err := client.Get(internalURL)\n        if r != nil && r.StatusCode == http.StatusServiceUnavailable {\n            return fmt.Errorf(\"node not ready, retrying\")\n        }\n        return err\n    }, backoff.WithMaxRetries(backoff.NewExponentialBackOff(), 10))\n}","preventionTips":["Wait for /v1/.well-known/ready before sending internal replication traffic","Configure Kubernetes readiness probes so traffic only reaches started nodes","Retry 503s with exponential backoff and a sensible deadline","Size startup timeouts for large recovery (many shards / slow disks)"],"tags":["http-503","startup","availability","readiness"],"backgroundTag":"service-not-ready","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}