thanos-io/thanos · error

expected in config payload, but got instead

Error message

expected %d in config payload, but got %d instead

What it means

The final validation in parseConfig: the version line plus nodes line must total exactly the byte count declared as configSize in the metadata header. When the declared length differs from what was actually read, the payload is incomplete or misframed and cannot be trusted, so this error is returned. It protects against accepting partially delivered or corrupted cluster membership data.

Solutions

1. Fix the server-side size computation so configSize equals the exact byte length of the version+nodes payload.
2. Check for proxies/load balancers that could truncate or alter the response body.
3. Retry the fetch with backoff — transient inconsistencies occur during cluster reconfiguration.
4. Capture traffic (tcpdump) and compare declared vs actual payload bytes to locate the off-by-N.

Example fix

// before
fmt.Fprintf(conn, "CONFIG cluster 0 %d\r\n%s\r\n", len(body)+2, body) // overcounts
// after
body := versionLine + nodesLine
fmt.Fprintf(conn, "CONFIG cluster 0 %d\r\n%s", len(body), body)

func validatePayloadSize(headerLen int, versionLine, nodesLine string) error {
    if got := len(versionLine) + len(nodesLine); got != headerLen {
        return fmt.Errorf("expected %d payload bytes, got %d", headerLen, got)
    }
    return nil
}

cfg, err := discovery.parseConfig(reader)
if err != nil && strings.Contains(err.Error(), "in config payload, but got") {
    return backoff.Retry(func() error { cfg, err = refetchConfig(addr); return err }, b)
}

When it happens

Trigger: The server writes fewer or more payload bytes than declared in 'CONFIG cluster 0 <len>' — truncation by a proxy, line-ending length accounting bugs (\n vs \r\n), or a concurrent cluster reconfiguration changing the payload mid-write.

Common situations: ElastiCache failover or node add/remove racing with a fetch; custom memcached-compatible servers with off-by-N size computation; load balancer idle timeouts cutting the response.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• expected 4 components in config metadata, and received
• failed to parse config size from metadata
• failed to parser config version
• failed to read nodes
• failed to read config metadata

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/eee122788e7d9d33. Report an issue: GitHub.

Appendix: source

Thrown at pkg/discovery/memcache/resolver.go:95

		return nil, fmt.Errorf("failed to parse config size from metadata: %s, error: %s", configMeta, err)
	}

	configVersion, err := reader.ReadString('\n')
	if err != nil {
		return nil, fmt.Errorf("failed to find config version: %s", err)
	}
	clusterConfig.version, err = strconv.Atoi(strings.TrimSpace(configVersion))
	if err != nil {
		return nil, fmt.Errorf("failed to parser config version: %s", err)
	}

	nodes, err := reader.ReadString('\n')
	if err != nil {
		return nil, fmt.Errorf("failed to read nodes: %s", err)
	}

	if len(configVersion)+len(nodes) != configSize {
		return nil, fmt.Errorf("expected %d in config payload, but got %d instead", configSize, len(configVersion)+len(nodes))
	}

	for host := range strings.SplitSeq(strings.TrimSpace(nodes), " ") {
		dnsIpPort := strings.Split(host, "|")
		if len(dnsIpPort) != 3 {
			return nil, fmt.Errorf("node not in expected format: %s", dnsIpPort)
		}
		port, err := strconv.Atoi(dnsIpPort[2])
		if err != nil {
			return nil, fmt.Errorf("failed to parse port: %s, err: %s", dnsIpPort, err)
		}
		clusterConfig.nodes = append(clusterConfig.nodes, node{dns: dnsIpPort[0], ip: dnsIpPort[1], port: port})
	}

	return clusterConfig, nil
}

View on GitHub (pinned to 35b8b99117)