weaviate/weaviate · error

search layer at level %d

Error message

search layer at level %d

What it means

doAtLevel performs the core searchLayerByVectorWithDistancer to find candidate neighbors at the given level. This error wraps any failure of that beam search — context cancellation/deadline, distancer errors from a compressor (PQ/BQ/SQ), or allocation failures for the search's visited list.

Source

Thrown at adapters/repos/db/vector/hnsw/neighbor_connections.go:270

				n.graph.pools.visitedLists.Return(visited)
				return err
			}
		}
		n.graph.pools.visitedLists.Return(visited)
		// use dynamic max connections only during tombstone cleanup
		maxConnections = n.maximumConnections(level)
	} else {
		if err := n.pickEntrypoint(); err != nil {
			return errors.Wrap(err, "pick entrypoint at level beginning")
		}
		eps := priorityqueue.NewMin[any](1)
		eps.Insert(n.entryPointID, n.entryPointDist)
		var err error

		results, err = n.graph.searchLayerByVectorWithDistancer(ctx, n.nodeVec, eps, n.graph.efConstruction,
			level, nil, n.distancer)
		if err != nil {
			return errors.Wrapf(err, "search layer at level %d", level)
		}

		n.graph.insertMetrics.findAndConnectSearch(before)
		before = time.Now()
	}

	if err := n.graph.selectNeighborsHeuristic(results, maxConnections-total, n.denyList); err != nil {
		return errors.Wrap(err, "heuristic")
	}

	n.graph.insertMetrics.findAndConnectHeuristic(before)
	before = time.Now()

	// // for distributed spike
	// neighborsAtLevel[level] = neighbors

	neighbors := make([]uint64, total, total+results.Len())
	copy(neighbors, extraIDs)

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check the wrapped root cause: raise timeouts or lower efConstruction for time-out cases.
2. Reindex/rebuild if the compressor state is corrupt (errors from the distancer on every search).
3. Reduce concurrent inserts per shard to relieve pool/visited-list pressure.
4. Retry the batch after transient conditions resolve.

Example fix

// before: tight timeout around import
timeout := 5 * time.Second

// after: sized to ef and dataset
timeout := 60 * time.Second
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()

if err := bucket.Add(ctx, vector); err != nil {
	if strings.Contains(err.Error(), "search layer at level") {
		if errors.Is(err, context.DeadlineExceeded) {
			return retryWithLongerTimeout(err)
		}
		// distancer/codec errors: schedule reindex of compressed state
		return scheduleReindex(err)
	}
	return err
}

When it happens

Trigger: Insert (findAndConnectNeighbors) or reconnect during cleanup, when layer search fails: context deadline exceeded, corrupted compressed-codec data, or search state (pools/visited lists) errors under heavy concurrency.

Common situations: Very high efConstruction with small request timeouts; PQ-compressed indexes with corrupted codebooks after a crash; extreme concurrent import load exhausting search pools.

Related errors

• index is not hnsw
• drop vector index %q: %w
• delete from vector index of vector %q: %w
• vector index config (%T) is not of type HNSW, but objects ma
• init vector index

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/9143f2ae0cd74c7e. Report an issue: GitHub.