{"record":{"id":"c26ed3c8d5528403","repo":"thanos-io/thanos","slug":"shard-size-d-is-larger-than-number-of-nodes-in-az","errorCode":null,"errorMessage":"shard size %d is larger than number of nodes in AZ %s (%d)","messagePattern":"shard size (.+?) is larger than number of nodes in AZ (.+?) \\((.+?)\\)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/hashring.go","lineNumber":642,"sourceCode":"\t}\n\n\tfor az := range sectionsByAZ {\n\t\tsort.Sort(sectionsByAZ[az])\n\t}\n\n\tss := s.getShardSize(tenant)\n\tvar take int\n\tif s.shuffleShardingConfig.ZoneAwarenessDisabled {\n\t\ttake = ss\n\t} else {\n\t\ttake = ShuffleShardExpectedInstancesPerZone(ss, len(nodesByAZ))\n\t}\n\n\tvar finalNodes = make([]Endpoint, 0, take*len(nodesByAZ))\n\n\tfor az, azNodes := range nodesByAZ {\n\t\tif take > len(azNodes) {\n\t\t\treturn nil, fmt.Errorf(\"shard size %d is larger than number of nodes in AZ %s (%d)\", ss, az, len(azNodes))\n\t\t}\n\n\t\tazSections := sectionsByAZ[az]\n\t\tif len(azSections) == 0 {\n\t\t\tcontinue\n\t\t}\n\n\t\tseed := ShuffleShardSeed(tenant, az)\n\t\tr := rand.New(rand.NewSource(seed))\n\n\t\tselected := make(map[uint64]struct{})\n\n\t\tfor i := 0; i < take; i++ {\n\t\t\trandomPos := r.Uint64()\n\t\t\tstartIdx := sort.Search(len(azSections), func(idx int) bool {\n\t\t\t\treturn azSections[idx].hash >= randomPos\n\t\t\t})\n\t\t\tif startIdx == len(azSections) {","sourceCodeStart":624,"sourceCodeEnd":660,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/hashring.go#L624-L660","documentation":"When computing a tenant's shard, the shuffle-sharding algorithm spreads the shard size across availability zones and takes `take` nodes per AZ. If the requested per-AZ take exceeds the number of nodes available in that AZ, it refuses rather than producing an unbalanced shard, returning this error naming the AZ and node count.","triggerScenarios":"shuffleSharding.shardSize (divided across AZs) is greater than the number of endpoints registered in some AZ of the ketama ring; encountered inside getTenantShard when a tenant lookup triggers shard construction on an under-provisioned AZ.","commonSituations":"One receiver replica in an AZ is down or was removed, shrinking that AZ below the required shard share; shardSize configured larger than the deployment supports; partial rollout where some AZs have fewer nodes.","solutions":["Reduce shuffleSharding.shardSize so each AZ has at least the required per-AZ node count","Restore/replace the missing replicas in the deficient AZ so its node count covers the take","Remove or fix the wrong AZ label on endpoints so nodes are grouped into the intended AZs","As a stopgap, temporarily disable shuffle sharding (unset shardSize) to use the full ketama ring"],"exampleFix":"// before\nshuffleSharding:\n  shardSize: 6   # but AZ us-east-1b has only 2 nodes\n// after\nshuffleSharding:\n  shardSize: 4   # or scale the AZ to >= take nodes","handlingStrategy":"validation","validationCode":"func shardSizeOK(nodes []Endpoint, shardSize int) bool {\n    if shardSize == 0 { return true }\n    byAZ := map[string]int{}\n    for _, n := range nodes { byAZ[n.AZ]++ }\n    take := shardSize / len(byAZ)\n    for az, c := range byAZ {\n        if take > c { return false }\n    }\n    return true\n}","typeGuard":null,"tryCatchPattern":"ep, err := ring.Get(tenant)\nif err != nil && strings.Contains(err.Error(), \"shard size\") {\n    // fall back to full ring or alert ops to fix AZ capacity\n    ep = baseRing.Get(tenant)\n}","preventionTips":["Keep shardSize <= min node count across AZs","Monitor per-AZ replica counts; replace dead nodes before tenant lookups fail","Verify AZ labels on endpoints during registration"],"tags":["hashring","shuffle-sharding","capacity"],"backgroundTag":"invalid-config-value","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}