weaviate/weaviate · error

startup is not complete

Error message

startup is not complete

What it means

Before serving the internal GET object request, the handler checks i.db.StartupComplete(). If the database has not finished startup (recovery, shard loading, schema init), it responds "startup is not complete" with HTTP 503 Service Unavailable. This is a deliberate guard, not a bug — the node cannot safely serve reads yet.

Source

Thrown at adapters/handlers/rest/clusterapi/indices.go:544

				http.StatusBadRequest)
			return
		}

		var additional additional.Properties
		if err := json.Unmarshal(additionalBytes, &additional); err != nil {
			http.Error(w, "unmarshal 'additional' param from json: "+err.Error(),
				http.StatusBadRequest)
			return
		}

		var selectProperties search.SelectProperties
		if err := json.Unmarshal(selectPropertiesBytes, &selectProperties); err != nil {
			http.Error(w, "unmarshal 'selectProperties' param from json: "+err.Error(),
				http.StatusBadRequest)
			return
		}
		if !i.db.StartupComplete() {
			http.Error(w, "startup is not complete", http.StatusServiceUnavailable)
			return
		}

		i.logger.WithFields(logrus.Fields{
			"shard":  shard,
			"action": "GetObject",
		}).Debug("getting object ...")

		obj, err := i.shards.GetObject(r.Context(), index, shard, strfmt.UUID(id),
			selectProperties, additional)
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		if obj == nil {
			// this is a legitimate case - the requested ID doesn't exist, don't try
			// to marshal anything

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Retry with backoff until the node reports ready (monitor /v1/.well-known/ready or the metrics endpoint).
2. Gate replication traffic on the node's readiness probe instead of container liveness.
3. Increase startup resources/timeouts if startup is genuinely slow (large disks, many shards).
4. Check logs for slow startup phases (recovery, shard init) to size readiness timeouts correctly.

Example fix

// before
resp := http.Get(internalURL) // fires immediately after pod start
// after
waitForReady(nodeHost, 5*time.Minute) // poll /v1/.well-known/ready with backoff
resp := http.Get(internalURL)

// Gate requests on the node's readiness before calling the internal API
func nodeReady(host string) bool {
    resp, err := http.Get(host + "/v1/.well-known/ready")
    return err == nil && resp.StatusCode == http.StatusOK
}

if resp.StatusCode == http.StatusServiceUnavailable {
    // retry with exponential backoff until startup completes
    return backoff.Retry(func() error {
        r, err := client.Get(internalURL)
        if r != nil && r.StatusCode == http.StatusServiceUnavailable {
            return fmt.Errorf("node not ready, retrying")
        }
        return err
    }, backoff.WithMaxRetries(backoff.NewExponentialBackOff(), 10))
}

When it happens

Trigger: Any GET object request on the internal cluster API sent to a node that is still starting up: after a restart with large LSM stores still recovering, during shard migration/replication to a freshly joined node, or immediately after container start before health checks pass.

Common situations: Kubernetes rolling updates where replica-set clients hit a not-yet-ready pod, orchestration sending traffic before readiness, or replication tooling racing node boot after a crash.

Related errors

• <joined startup error messages>
• init context expired before remote was ready
• not ready: status %d
• init context expired before remote was ready
• not ready: status %d

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/ffd97e870a46b1b0. Report an issue: GitHub.