weaviate/weaviate · error · ErrCleanupShardFailed

%w: %w (ErrCleanupShardFailed)

Error message

%w: %w (ErrCleanupShardFailed)

What it means

After the walk, shardErrs.ToErrorLimited(maxReportedErrors) is wrapped as ErrCleanupShardFailed ('partial-reindex cleanup could not sweep every shard it reached'). This sentinel marks sweeps that reached one or more shards and could not clean them — distinct from truncation (unreached shards). A delete landing mid-walk after a shard failed can carry both this and ErrCleanupCollectionDropped; use IsCleanupCollectionDropped, which returns false in that case.

Source

Thrown at adapters/repos/db/reindex_cancel_cleanup.go:239

			}
			shard = unwrapped
		}
		// Charged whether or not the sweep then failed, for the same reason the
		// gate's reads are: the reads are paid before the outcome is known.
		shardReads, err := shard.CleanStalePartialReindexState(ctx, propName, indexType)
		payloadReads += shardReads
		if err != nil {
			reported := fmt.Errorf("shard %q: %w", name, err)
			if truncated := truncatedByCancellation(reported); truncated != nil {
				return truncated
			}
			shardErrs.Add(reported)
		}
		return nil
	})
	var failedShards error
	if reported := shardErrs.ToErrorLimited(maxReportedErrors); reported != nil {
		failedShards = fmt.Errorf("%w: %w", ErrCleanupShardFailed, reported)
	}
	sweepErr := errors.Join(failedShards, classifyIncompleteWalk(walkErr))

	outcome, _ := ClassifyCleanupSweep(sweepErr)
	msg, level := CleanupSweepSummary(sweepPhaseIndexCleanup, outcome)
	uncachedListings := dirs.refusedListings() - refusedBefore
	if uncachedListings > 0 {
		// logrus orders its levels descending, so this only ever raises severity:
		// a bound the cache silently hit has no other signal.
		level = min(level, logrus.WarnLevel)
	}
	i.logger.WithFields(map[string]any{
		"property":          propName,
		"index_type":        indexType,
		"operation":         "CleanStalePartialReindexState",
		"skipped_shards":    skippedShards,
		"payload_reads":     payloadReads,
		"uncached_listings": uncachedListings,

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Unwrap the joined error to list the failing shards (capped at maxReportedErrors) and fix each underlying cause.
2. Distinguish via errors.Is: ErrCleanupShardFailed (real failures) vs ErrCleanupSweepTruncated (unreached/unknown) before alerting.
3. After fixing, re-run the sweep or resubmit the reindex — a stuck shard is never allowed to wedge the tuple permanently.

Example fix

// before: alerting on truncation and shard failures alike
if err != nil { alert(err) }
// after
var truncated, failed bool
if errors.Is(err, ErrCleanupSweepTruncated) { truncated = true }
if errors.Is(err, ErrCleanupShardFailed) { failed = true }
if failed { alert(err) } else if truncated { log.Warnf("retry later: %v", err) }

func classifySweep(err error) string {
    switch {
    case errors.Is(err, ErrCleanupShardFailed):
        return "shard-failed"
    case errors.Is(err, ErrCleanupSweepTruncated):
        return "truncated"
    case errors.Is(err, ErrCleanupCollectionDropped):
        return "collection-dropped"
    default:
        return "clean"
    }
}

if err := sweep(...); err != nil {
    switch classifySweep(err) {
    case "shard-failed":
        alertOps(err) // real failures, capped list of causes inside
    case "truncated", "collection-dropped":
        log.Warnf("benign: %v", err)
    }
}

When it happens

Trigger: One or more shards' CleanStalePartialReindexState (or lazy unwrap) failed with non-cancellation errors during Index.cleanStalePartialReindexState; the individual causes are joined underneath, capped at maxReportedErrors entries.

Common situations: Several shards on a failing disk; a property migration that left buckets in a state the cleaner cannot remove; repeated submit-time pre-cleanup failures blocking reindex resubmission for the tuple.

Related errors

• %w: %w (ErrCleanupCollectionDropped)
• %w: %w (ErrCleanupSweepTruncated)
• shard %q: %w
• could not cast generative result additional prop
• already shut or dropped

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/cb8bb176185b6e79. Report an issue: GitHub.