weaviate/weaviate · error · ErrCleanupShardFailed

: (ErrCleanupShardFailed)

Error message

%w: %w (ErrCleanupShardFailed)

What it means

After the walk, shardErrs.ToErrorLimited(maxReportedErrors) is wrapped as ErrCleanupShardFailed ('partial-reindex cleanup could not sweep every shard it reached'). This sentinel marks sweeps that reached one or more shards and could not clean them — distinct from truncation (unreached shards). A delete landing mid-walk after a shard failed can carry both this and ErrCleanupCollectionDropped; use IsCleanupCollectionDropped, which returns false in that case.

Solutions

  1. Unwrap the joined error to list the failing shards (capped at maxReportedErrors) and fix each underlying cause.
  2. Distinguish via errors.Is: ErrCleanupShardFailed (real failures) vs ErrCleanupSweepTruncated (unreached/unknown) before alerting.
  3. After fixing, re-run the sweep or resubmit the reindex — a stuck shard is never allowed to wedge the tuple permanently.

Example fix

// before: alerting on truncation and shard failures alike
if err != nil { alert(err) }
// after
var truncated, failed bool
if errors.Is(err, ErrCleanupSweepTruncated) { truncated = true }
if errors.Is(err, ErrCleanupShardFailed) { failed = true }
if failed { alert(err) } else if truncated { log.Warnf("retry later: %v", err) }
Defensive patterns

Strategy: type-guard

Type guard

func classifySweep(err error) string {
    switch {
    case errors.Is(err, ErrCleanupShardFailed):
        return "shard-failed"
    case errors.Is(err, ErrCleanupSweepTruncated):
        return "truncated"
    case errors.Is(err, ErrCleanupCollectionDropped):
        return "collection-dropped"
    default:
        return "clean"
    }
}

Try / catch

if err := sweep(...); err != nil {
    switch classifySweep(err) {
    case "shard-failed":
        alertOps(err) // real failures, capped list of causes inside
    case "truncated", "collection-dropped":
        log.Warnf("benign: %v", err)
    }
}

Prevention

When it happens

Trigger: One or more shards' CleanStalePartialReindexState (or lazy unwrap) failed with non-cancellation errors during Index.cleanStalePartialReindexState; the individual causes are joined underneath, capped at maxReportedErrors entries.

Common situations: Several shards on a failing disk; a property migration that left buckets in a state the cleaner cannot remove; repeated submit-time pre-cleanup failures blocking reindex resubmission for the tuple.

Related errors


AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/cb8bb176185b6e79. Report an issue: GitHub.

Appendix: source

Thrown at adapters/repos/db/reindex_cancel_cleanup.go:239

			}
			shard = unwrapped
		}
		// Charged whether or not the sweep then failed, for the same reason the
		// gate's reads are: the reads are paid before the outcome is known.
		shardReads, err := shard.CleanStalePartialReindexState(ctx, propName, indexType)
		payloadReads += shardReads
		if err != nil {
			reported := fmt.Errorf("shard %q: %w", name, err)
			if truncated := truncatedByCancellation(reported); truncated != nil {
				return truncated
			}
			shardErrs.Add(reported)
		}
		return nil
	})
	var failedShards error
	if reported := shardErrs.ToErrorLimited(maxReportedErrors); reported != nil {
		failedShards = fmt.Errorf("%w: %w", ErrCleanupShardFailed, reported)
	}
	sweepErr := errors.Join(failedShards, classifyIncompleteWalk(walkErr))

	outcome, _ := ClassifyCleanupSweep(sweepErr)
	msg, level := CleanupSweepSummary(sweepPhaseIndexCleanup, outcome)
	uncachedListings := dirs.refusedListings() - refusedBefore
	if uncachedListings > 0 {
		// logrus orders its levels descending, so this only ever raises severity:
		// a bound the cache silently hit has no other signal.
		level = min(level, logrus.WarnLevel)
	}
	i.logger.WithFields(map[string]any{
		"property":          propName,
		"index_type":        indexType,
		"operation":         "CleanStalePartialReindexState",
		"skipped_shards":    skippedShards,
		"payload_reads":     payloadReads,
		"uncached_listings": uncachedListings,

View on GitHub (pinned to 75aa4b6d11)