{"record":{"id":"cb8bb176185b6e79","repo":"weaviate/weaviate","slug":"w-w-errcleanupshardfailed","errorCode":null,"errorMessage":"%w: %w (ErrCleanupShardFailed)","messagePattern":"%w: %w \\(ErrCleanupShardFailed\\)","errorType":"exception","errorClass":"ErrCleanupShardFailed","httpStatus":null,"severity":"error","filePath":"adapters/repos/db/reindex_cancel_cleanup.go","lineNumber":239,"sourceCode":"\t\t\t}\n\t\t\tshard = unwrapped\n\t\t}\n\t\t// Charged whether or not the sweep then failed, for the same reason the\n\t\t// gate's reads are: the reads are paid before the outcome is known.\n\t\tshardReads, err := shard.CleanStalePartialReindexState(ctx, propName, indexType)\n\t\tpayloadReads += shardReads\n\t\tif err != nil {\n\t\t\treported := fmt.Errorf(\"shard %q: %w\", name, err)\n\t\t\tif truncated := truncatedByCancellation(reported); truncated != nil {\n\t\t\t\treturn truncated\n\t\t\t}\n\t\t\tshardErrs.Add(reported)\n\t\t}\n\t\treturn nil\n\t})\n\tvar failedShards error\n\tif reported := shardErrs.ToErrorLimited(maxReportedErrors); reported != nil {\n\t\tfailedShards = fmt.Errorf(\"%w: %w\", ErrCleanupShardFailed, reported)\n\t}\n\tsweepErr := errors.Join(failedShards, classifyIncompleteWalk(walkErr))\n\n\toutcome, _ := ClassifyCleanupSweep(sweepErr)\n\tmsg, level := CleanupSweepSummary(sweepPhaseIndexCleanup, outcome)\n\tuncachedListings := dirs.refusedListings() - refusedBefore\n\tif uncachedListings > 0 {\n\t\t// logrus orders its levels descending, so this only ever raises severity:\n\t\t// a bound the cache silently hit has no other signal.\n\t\tlevel = min(level, logrus.WarnLevel)\n\t}\n\ti.logger.WithFields(map[string]any{\n\t\t\"property\":          propName,\n\t\t\"index_type\":        indexType,\n\t\t\"operation\":         \"CleanStalePartialReindexState\",\n\t\t\"skipped_shards\":    skippedShards,\n\t\t\"payload_reads\":     payloadReads,\n\t\t\"uncached_listings\": uncachedListings,","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/reindex_cancel_cleanup.go#L221-L257","documentation":"After the walk, shardErrs.ToErrorLimited(maxReportedErrors) is wrapped as ErrCleanupShardFailed ('partial-reindex cleanup could not sweep every shard it reached'). This sentinel marks sweeps that reached one or more shards and could not clean them — distinct from truncation (unreached shards). A delete landing mid-walk after a shard failed can carry both this and ErrCleanupCollectionDropped; use IsCleanupCollectionDropped, which returns false in that case.","triggerScenarios":"One or more shards' CleanStalePartialReindexState (or lazy unwrap) failed with non-cancellation errors during Index.cleanStalePartialReindexState; the individual causes are joined underneath, capped at maxReportedErrors entries.","commonSituations":"Several shards on a failing disk; a property migration that left buckets in a state the cleaner cannot remove; repeated submit-time pre-cleanup failures blocking reindex resubmission for the tuple.","solutions":["Unwrap the joined error to list the failing shards (capped at maxReportedErrors) and fix each underlying cause.","Distinguish via errors.Is: ErrCleanupShardFailed (real failures) vs ErrCleanupSweepTruncated (unreached/unknown) before alerting.","After fixing, re-run the sweep or resubmit the reindex — a stuck shard is never allowed to wedge the tuple permanently."],"exampleFix":"// before: alerting on truncation and shard failures alike\nif err != nil { alert(err) }\n// after\nvar truncated, failed bool\nif errors.Is(err, ErrCleanupSweepTruncated) { truncated = true }\nif errors.Is(err, ErrCleanupShardFailed) { failed = true }\nif failed { alert(err) } else if truncated { log.Warnf(\"retry later: %v\", err) }","handlingStrategy":"type-guard","validationCode":null,"typeGuard":"func classifySweep(err error) string {\n    switch {\n    case errors.Is(err, ErrCleanupShardFailed):\n        return \"shard-failed\"\n    case errors.Is(err, ErrCleanupSweepTruncated):\n        return \"truncated\"\n    case errors.Is(err, ErrCleanupCollectionDropped):\n        return \"collection-dropped\"\n    default:\n        return \"clean\"\n    }\n}","tryCatchPattern":"if err := sweep(...); err != nil {\n    switch classifySweep(err) {\n    case \"shard-failed\":\n        alertOps(err) // real failures, capped list of causes inside\n    case \"truncated\", \"collection-dropped\":\n        log.Warnf(\"benign: %v\", err)\n    }\n}","preventionTips":["Use the exported sentinels with errors.Is rather than string matching","Remember a mid-walk delete can carry both ErrCleanupShardFailed and ErrCleanupCollectionDropped — use IsCleanupCollectionDropped for the 'gone is the whole story' check","Fix root causes per shard; retries alone will not clear ErrCleanupShardFailed"],"tags":["weaviate","reindex","cleanup","sentinel-error","shard-failure"],"backgroundTag":"shard-cleanup-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}