thanos-io/thanos · error

failed to find config version

Error message

failed to find config version: %s

What it means

After the metadata line, the CONFIG payload's first line is the cluster version number. parseConfig reads it with ReadString('\n'); this error is returned when that read fails — usually io.EOF because the stream ended right after the metadata line, or a connection error. It means the response was truncated before the version line arrived.

Solutions

1. Fix server-side framing so the full payload always follows the metadata line.
2. Investigate connection stability (timeouts, keep-alives, LB idle timeouts).
3. Retry Resolve with backoff to rule out transient disconnects.
4. Check for io.EOF in the wrapped error to distinguish truncation from real I/O errors.

Example fix

// before
cfg, err := parseConfig(reader) // EOF after metadata line
// after
cfg, err := parseConfig(reader)
if err != nil && strings.Contains(err.Error(), "config version") {
    return retryFetchConfig() // truncated response; refetch
}

func hasFullPayload(meta string, reader *bufio.Reader) bool {
    parts := strings.Fields(meta)
    if len(parts) != 4 {
        return false
    }
    size, err := strconv.Atoi(parts[3])
    return err == nil && reader.Buffered() >= size
}

cfg, err := discovery.parseConfig(reader)
if err != nil && strings.Contains(err.Error(), "failed to find config version") {
    return backoff.Retry(func() error { cfg, err = refetchConfig(addr); return err }, b)
}

When it happens

Trigger: The reader contains only the metadata line and then ends; the connection drops after the header; the response is cut off by a short read timeout or premature close.

Common situations: Connections closed prematurely under load; servers sending the header but failing to write the payload; flaky networks between client and ElastiCache; LB idle-timeout resets.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• failed to read config metadata
• expected 4 components in config metadata, and received
• failed to parse config size from metadata
• failed to parser config version
• failed to read nodes

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9081d3564602e0c0. Report an issue: GitHub.

Appendix: source

Thrown at pkg/discovery/memcache/resolver.go:82

	if err != nil {
		return nil, fmt.Errorf("failed to read config metadata: %s", err)
	}
	configMeta = strings.TrimSpace(configMeta)

	// First line should be "CONFIG cluster 0 [length-of-payload-]
	configMetaComponents := strings.Split(configMeta, " ")
	if len(configMetaComponents) != 4 {
		return nil, fmt.Errorf("expected 4 components in config metadata, and received %d, meta: %s", len(configMetaComponents), configMeta)
	}

	configSize, err := strconv.Atoi(configMetaComponents[3])
	if err != nil {
		return nil, fmt.Errorf("failed to parse config size from metadata: %s, error: %s", configMeta, err)
	}

	configVersion, err := reader.ReadString('\n')
	if err != nil {
		return nil, fmt.Errorf("failed to find config version: %s", err)
	}
	clusterConfig.version, err = strconv.Atoi(strings.TrimSpace(configVersion))
	if err != nil {
		return nil, fmt.Errorf("failed to parser config version: %s", err)
	}

	nodes, err := reader.ReadString('\n')
	if err != nil {
		return nil, fmt.Errorf("failed to read nodes: %s", err)
	}

	if len(configVersion)+len(nodes) != configSize {
		return nil, fmt.Errorf("expected %d in config payload, but got %d instead", configSize, len(configVersion)+len(nodes))
	}

	for host := range strings.SplitSeq(strings.TrimSpace(nodes), " ") {
		dnsIpPort := strings.Split(host, "|")
		if len(dnsIpPort) != 3 {

View on GitHub (pinned to 35b8b99117)