{"record":{"id":"32594dae5491c421","repo":"weaviate/weaviate","slug":"compare-digests","errorCode":null,"errorMessage":"compare digests: ","messagePattern":"compare digests: ","errorType":"http","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/handlers/rest/clusterapi/indices_replicas.go","lineNumber":827,"sourceCode":"\t\tif err != nil {\n\t\t\tvar maxBytesErr *http.MaxBytesError\n\t\t\tif errors.As(err, &maxBytesErr) {\n\t\t\t\thttp.Error(w, \"request body too large\", http.StatusRequestEntityTooLarge)\n\t\t\t\treturn\n\t\t\t}\n\t\t\thttp.Error(w, \"read request body: \"+err.Error(), http.StatusInternalServerError)\n\t\t\treturn\n\t\t}\n\n\t\tsourceDigests, err := replica.RepairDigestsFromBinary(body)\n\t\tif err != nil {\n\t\t\thttp.Error(w, \"decode packed digests: \"+err.Error(), http.StatusBadRequest)\n\t\t\treturn\n\t\t}\n\n\t\tstale, err := i.replicator.CompareDigests(r.Context(), index, shard, sourceDigests)\n\t\tif err != nil {\n\t\t\thttp.Error(w, \"compare digests: \"+err.Error(), asyncCheckpointHTTPStatus(err))\n\t\t\treturn\n\t\t}\n\n\t\tout := replica.RepairDigestsToBinary(stale)\n\n\t\tw.Header().Set(\"Content-Type\", \"application/octet-stream\")\n\t\tw.Header().Set(\"Content-Length\", strconv.Itoa(len(out)))\n\t\tw.Write(out) //nolint:errcheck\n\t})\n}\n\nfunc (i *replicatedIndices) countObjects() http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\targs := regxCountObjects.FindStringSubmatch(r.URL.Path)\n\t\tif len(args) != 3 {\n\t\t\thttp.Error(w, \"invalid URI\", http.StatusBadRequest)\n\t\t\treturn\n\t\t}","sourceCodeStart":809,"sourceCodeEnd":845,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/handlers/rest/clusterapi/indices_replicas.go#L809-L845","documentation":"The replicator's CompareDigests call (which walks the local shard and compares digests against the received source list) returned an error; the handler responds 'compare digests: <cause>' with a status from asyncCheckpointHTTPStatus(err). This is the core shard-level digest comparison failing on the target replica.","triggerScenarios":"Target shard missing, closed, or offloaded (tenant not loaded); context timeout/cancellation while scanning a large shard; underlying store/LSM read errors; async checkpoint state on the shard blocking the comparison.","commonSituations":"Running repairs on an offloaded tenant shard; shard migration/rebalance racing with digest comparison; disk errors or corrupted segments on the target node; shard index/shard name mismatch.","solutions":["Read the wrapped cause: if shard-not-found, verify the shard exists and the tenant/shard is loaded (un-offload it) before comparing","Retry after transient/timeout errors, ideally when the shard is idle and not being migrated","Check node logs and disk health for underlying store read errors"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"loaded, err := client.Tenants().Exist(ctx, index, shard)\nif err != nil || loaded != objects.TenantActivityStatusActive {\n    return fmt.Errorf(\"shard %s/%s not active\", index, shard)\n}","typeGuard":null,"tryCatchPattern":"stale, err := replicator.CompareDigests(ctx, index, shard, sourceDigests)\nif err != nil {\n    if asyncCheckpointHTTPStatus(err) >= 500 || errors.Is(err, context.DeadlineExceeded) {\n        return retryWithBackoff(ctx, 3, func() error { return compareDigests(...) })\n    }\n    return fmt.Errorf(\"compare digests: %w\", err)\n}","preventionTips":["Un-offload/activate the tenant shard before comparing digests","Don't run digest comparisons concurrently with shard migrations","Use generous context timeouts for large shards; watch node disk health"],"tags":["http","clusterapi","digests","replication","shard"],"backgroundTag":"digest-comparison-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}