thanos-io/thanos · error

failed to read nodes

Error message

failed to read nodes: %s

What it means

After the version line, parseConfig reads the nodes line (space-separated 'host|ip|port' entries). The protocol requires that len(versionLine)+len(nodesLine) equals the configSize declared in the metadata; if not, the payload is incomplete or misframed and this error is returned with both expected and actual byte counts.

Solutions

1. Make the server compute configSize as the exact byte length of the version+nodes lines it emits (mind \r\n counting).
2. Compare expected vs got counts: a 1-2 byte delta suggests a line-ending bug; a large delta suggests truncation.
3. Retry Resolve — payloads can be inconsistent during cluster reconfiguration.
4. Capture traffic (tcpdump) against a known-good ElastiCache response to fix framing.

Example fix

// before
payload := fmt.Sprintf("1\n%s", nodes)
header := fmt.Sprintf("CONFIG cluster 0 %d\r\n", len(payload)+len("\r\n")) // overcounts
// after
payload := fmt.Sprintf("1\r\n%s\r\n", nodes)
header := fmt.Sprintf("CONFIG cluster 0 %d\r\n", len(payload))

func payloadLengthMatches(body string, declared int) error {
    if got := len(body); got != declared {
        return fmt.Errorf("declared %d bytes, got %d", declared, got)
    }
    return nil
}

cfg, err := discovery.parseConfig(reader)
if err != nil && strings.Contains(err.Error(), "in config payload") {
    return retryFetchConfig() // likely truncation or reconfiguration race
}

When it happens

Trigger: The server closes the stream before sending the full node list (ReadString returns a partial line at EOF), or the declared configSize does not match the actual version+nodes bytes — e.g. CRLF vs LF counting bugs or cluster membership changing mid-read.

Common situations: ElastiCache returning a payload shorter than advertised during failover; CRLF vs LF length accounting bugs in custom servers; network truncation of the final line; stale configSize after a config change.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• failed to read config metadata
• expected 4 components in config metadata, and received
• expected in config payload, but got instead
• failed to parse config size from metadata
• failed to find config version

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e9daccf089e67850. Report an issue: GitHub.

Appendix: source

Thrown at pkg/discovery/memcache/resolver.go:91

	}

	configSize, err := strconv.Atoi(configMetaComponents[3])
	if err != nil {
		return nil, fmt.Errorf("failed to parse config size from metadata: %s, error: %s", configMeta, err)
	}

	configVersion, err := reader.ReadString('\n')
	if err != nil {
		return nil, fmt.Errorf("failed to find config version: %s", err)
	}
	clusterConfig.version, err = strconv.Atoi(strings.TrimSpace(configVersion))
	if err != nil {
		return nil, fmt.Errorf("failed to parser config version: %s", err)
	}

	nodes, err := reader.ReadString('\n')
	if err != nil {
		return nil, fmt.Errorf("failed to read nodes: %s", err)
	}

	if len(configVersion)+len(nodes) != configSize {
		return nil, fmt.Errorf("expected %d in config payload, but got %d instead", configSize, len(configVersion)+len(nodes))
	}

	for host := range strings.SplitSeq(strings.TrimSpace(nodes), " ") {
		dnsIpPort := strings.Split(host, "|")
		if len(dnsIpPort) != 3 {
			return nil, fmt.Errorf("node not in expected format: %s", dnsIpPort)
		}
		port, err := strconv.Atoi(dnsIpPort[2])
		if err != nil {
			return nil, fmt.Errorf("failed to parse port: %s, err: %s", dnsIpPort, err)
		}
		clusterConfig.nodes = append(clusterConfig.nodes, node{dns: dnsIpPort[0], ip: dnsIpPort[1], port: port})
	}

View on GitHub (pinned to 35b8b99117)