weaviate/weaviate · error
compare digests:
Error message
compare digests:
What it means
The replicator's CompareDigests call (which walks the local shard and compares digests against the received source list) returned an error; the handler responds 'compare digests: <cause>' with a status from asyncCheckpointHTTPStatus(err). This is the core shard-level digest comparison failing on the target replica.
Source
Thrown at adapters/handlers/rest/clusterapi/indices_replicas.go:827
if err != nil {
var maxBytesErr *http.MaxBytesError
if errors.As(err, &maxBytesErr) {
http.Error(w, "request body too large", http.StatusRequestEntityTooLarge)
return
}
http.Error(w, "read request body: "+err.Error(), http.StatusInternalServerError)
return
}
sourceDigests, err := replica.RepairDigestsFromBinary(body)
if err != nil {
http.Error(w, "decode packed digests: "+err.Error(), http.StatusBadRequest)
return
}
stale, err := i.replicator.CompareDigests(r.Context(), index, shard, sourceDigests)
if err != nil {
http.Error(w, "compare digests: "+err.Error(), asyncCheckpointHTTPStatus(err))
return
}
out := replica.RepairDigestsToBinary(stale)
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Length", strconv.Itoa(len(out)))
w.Write(out) //nolint:errcheck
})
}
func (i *replicatedIndices) countObjects() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
args := regxCountObjects.FindStringSubmatch(r.URL.Path)
if len(args) != 3 {
http.Error(w, "invalid URI", http.StatusBadRequest)
return
}View on GitHub (pinned to 75aa4b6d11)
Solutions
- Read the wrapped cause: if shard-not-found, verify the shard exists and the tenant/shard is loaded (un-offload it) before comparing
- Retry after transient/timeout errors, ideally when the shard is idle and not being migrated
- Check node logs and disk health for underlying store read errors
Defensive patterns
Strategy: retry
Validate before calling
loaded, err := client.Tenants().Exist(ctx, index, shard)
if err != nil || loaded != objects.TenantActivityStatusActive {
return fmt.Errorf("shard %s/%s not active", index, shard)
} Try / catch
stale, err := replicator.CompareDigests(ctx, index, shard, sourceDigests)
if err != nil {
if asyncCheckpointHTTPStatus(err) >= 500 || errors.Is(err, context.DeadlineExceeded) {
return retryWithBackoff(ctx, 3, func() error { return compareDigests(...) })
}
return fmt.Errorf("compare digests: %w", err)
} Prevention
- Un-offload/activate the tenant shard before comparing digests
- Don't run digest comparisons concurrently with shard migrations
- Use generous context timeouts for large shards; watch node disk health
When it happens
Trigger: Target shard missing, closed, or offloaded (tenant not loaded); context timeout/cancellation while scanning a large shard; underlying store/LSM read errors; async checkpoint state on the shard blocking the comparison.
Common situations: Running repairs on an offloaded tenant shard; shard migration/rebalance racing with digest comparison; disk errors or corrupted segments on the target node; shard index/shard name mismatch.
Related errors
AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04).
Data as JSON: /api/errors/32594dae5491c421.
Report an issue: GitHub.