{"record":{"id":"9143f2ae0cd74c7e","repo":"weaviate/weaviate","slug":"search-layer-at-level-d","errorCode":null,"errorMessage":"search layer at level %d","messagePattern":"search layer at level (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/vector/hnsw/neighbor_connections.go","lineNumber":270,"sourceCode":"\t\t\t\tn.graph.pools.visitedLists.Return(visited)\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\t\tn.graph.pools.visitedLists.Return(visited)\n\t\t// use dynamic max connections only during tombstone cleanup\n\t\tmaxConnections = n.maximumConnections(level)\n\t} else {\n\t\tif err := n.pickEntrypoint(); err != nil {\n\t\t\treturn errors.Wrap(err, \"pick entrypoint at level beginning\")\n\t\t}\n\t\teps := priorityqueue.NewMin[any](1)\n\t\teps.Insert(n.entryPointID, n.entryPointDist)\n\t\tvar err error\n\n\t\tresults, err = n.graph.searchLayerByVectorWithDistancer(ctx, n.nodeVec, eps, n.graph.efConstruction,\n\t\t\tlevel, nil, n.distancer)\n\t\tif err != nil {\n\t\t\treturn errors.Wrapf(err, \"search layer at level %d\", level)\n\t\t}\n\n\t\tn.graph.insertMetrics.findAndConnectSearch(before)\n\t\tbefore = time.Now()\n\t}\n\n\tif err := n.graph.selectNeighborsHeuristic(results, maxConnections-total, n.denyList); err != nil {\n\t\treturn errors.Wrap(err, \"heuristic\")\n\t}\n\n\tn.graph.insertMetrics.findAndConnectHeuristic(before)\n\tbefore = time.Now()\n\n\t// // for distributed spike\n\t// neighborsAtLevel[level] = neighbors\n\n\tneighbors := make([]uint64, total, total+results.Len())\n\tcopy(neighbors, extraIDs)","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/vector/hnsw/neighbor_connections.go#L252-L288","documentation":"doAtLevel performs the core searchLayerByVectorWithDistancer to find candidate neighbors at the given level. This error wraps any failure of that beam search — context cancellation/deadline, distancer errors from a compressor (PQ/BQ/SQ), or allocation failures for the search's visited list.","triggerScenarios":"Insert (findAndConnectNeighbors) or reconnect during cleanup, when layer search fails: context deadline exceeded, corrupted compressed-codec data, or search state (pools/visited lists) errors under heavy concurrency.","commonSituations":"Very high efConstruction with small request timeouts; PQ-compressed indexes with corrupted codebooks after a crash; extreme concurrent import load exhausting search pools.","solutions":["Check the wrapped root cause: raise timeouts or lower efConstruction for time-out cases.","Reindex/rebuild if the compressor state is corrupt (errors from the distancer on every search).","Reduce concurrent inserts per shard to relieve pool/visited-list pressure.","Retry the batch after transient conditions resolve."],"exampleFix":"// before: tight timeout around import\ntimeout := 5 * time.Second\n\n// after: sized to ef and dataset\ntimeout := 60 * time.Second\nctx, cancel := context.WithTimeout(ctx, timeout)\ndefer cancel()","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := bucket.Add(ctx, vector); err != nil {\n\tif strings.Contains(err.Error(), \"search layer at level\") {\n\t\tif errors.Is(err, context.DeadlineExceeded) {\n\t\t\treturn retryWithLongerTimeout(err)\n\t\t}\n\t\t// distancer/codec errors: schedule reindex of compressed state\n\t\treturn scheduleReindex(err)\n\t}\n\treturn err\n}","preventionTips":["Size timeouts to efConstruction and dataset scale.","Reduce per-shard insert concurrency to avoid pool exhaustion.","Monitor compressed-index integrity after crashes."],"tags":["hnsw","vector-index","search-layer","go"],"backgroundTag":"hnsw-layer-search-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}