thanos-io/thanos · error

shard size is larger than number of nodes in AZ ( )

Error message

shard size %d is larger than number of nodes in AZ %s (%d)

What it means

When computing a tenant's shard, the shuffle-sharding algorithm spreads the shard size across availability zones and takes `take` nodes per AZ. If the requested per-AZ take exceeds the number of nodes available in that AZ, it refuses rather than producing an unbalanced shard, returning this error naming the AZ and node count.

Solutions

1. Reduce shuffleSharding.shardSize so each AZ has at least the required per-AZ node count
2. Restore/replace the missing replicas in the deficient AZ so its node count covers the take
3. Remove or fix the wrong AZ label on endpoints so nodes are grouped into the intended AZs
4. As a stopgap, temporarily disable shuffle sharding (unset shardSize) to use the full ketama ring

Example fix

// before
shuffleSharding:
  shardSize: 6   # but AZ us-east-1b has only 2 nodes
// after
shuffleSharding:
  shardSize: 4   # or scale the AZ to >= take nodes

func shardSizeOK(nodes []Endpoint, shardSize int) bool {
    if shardSize == 0 { return true }
    byAZ := map[string]int{}
    for _, n := range nodes { byAZ[n.AZ]++ }
    take := shardSize / len(byAZ)
    for az, c := range byAZ {
        if take > c { return false }
    }
    return true
}

ep, err := ring.Get(tenant)
if err != nil && strings.Contains(err.Error(), "shard size") {
    // fall back to full ring or alert ops to fix AZ capacity
    ep = baseRing.Get(tenant)
}

When it happens

Trigger: shuffleSharding.shardSize (divided across AZs) is greater than the number of endpoints registered in some AZ of the ketama ring; encountered inside getTenantShard when a tenant lookup triggers shard construction on an under-provisioned AZ.

Common situations: One receiver replica in an AZ is down or was removed, shrinking that AZ below the required shard share; shardSize configured larger than the deployment supports; partial rollout where some AZs have fewer nodes.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• shard size is larger than number of nodes in hashring ( )
• shuffle sharding requires ketama hashring as base ring
• hashmod algorithm does not support shuffle sharding. Either…
• configuration file is not parsable
• configuration file is empty

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/c26ed3c8d5528403. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/hashring.go:642

	}

	for az := range sectionsByAZ {
		sort.Sort(sectionsByAZ[az])
	}

	ss := s.getShardSize(tenant)
	var take int
	if s.shuffleShardingConfig.ZoneAwarenessDisabled {
		take = ss
	} else {
		take = ShuffleShardExpectedInstancesPerZone(ss, len(nodesByAZ))
	}

	var finalNodes = make([]Endpoint, 0, take*len(nodesByAZ))

	for az, azNodes := range nodesByAZ {
		if take > len(azNodes) {
			return nil, fmt.Errorf("shard size %d is larger than number of nodes in AZ %s (%d)", ss, az, len(azNodes))
		}

		azSections := sectionsByAZ[az]
		if len(azSections) == 0 {
			continue
		}

		seed := ShuffleShardSeed(tenant, az)
		r := rand.New(rand.NewSource(seed))

		selected := make(map[uint64]struct{})

		for i := 0; i < take; i++ {
			randomPos := r.Uint64()
			startIdx := sort.Search(len(azSections), func(idx int) bool {
				return azSections[idx].hash >= randomPos
			})
			if startIdx == len(azSections) {

View on GitHub (pinned to 35b8b99117)