thanos-io/thanos · error

failed to read config metadata

Error message

failed to read config metadata: %s

What it means

The memcached auto-discovery client fetches cluster configuration via the memcached 'config get cluster' protocol, whose first line is metadata ('CONFIG cluster 0 <len>'). parseConfig reads that first line with bufio.Reader.ReadString('\n'); this error is returned when the read fails — typically EOF because the stream ended before any newline, or a connection error.

Solutions

1. Verify the target is a cluster-enabled ElastiCache/memcached endpoint that supports 'config get cluster'.
2. Check network connectivity and whether the connection closes prematurely (timeouts, proxies, LB idle timeouts).
3. Inspect the underlying error embedded via %s (io.EOF vs socket error) to narrow the cause.
4. Wrap Resolve with retry/backoff for transient connection resets.

Example fix

// before
cfg, err := discovery.parseConfig(bufio.NewReader(conn)) // EOF: server closed early
// after
conn.SetDeadline(time.Now().Add(5 * time.Second))
cfg, err := discovery.parseConfig(bufio.NewReader(conn))
if err != nil {
    return nil, fmt.Errorf("cluster config fetch failed (cluster-mode endpoint?): %w", err)
}

func canFetchConfig(conn net.Conn) error {
    conn.SetReadDeadline(time.Now().Add(5 * time.Second))
    return nil // also verify endpoint is cluster-mode before discovery
}

cfg, err := discovery.parseConfig(reader)
if err != nil {
    if errors.Is(err, io.EOF) || strings.Contains(err.Error(), "failed to read config metadata") {
        return nil, fmt.Errorf("endpoint %s closed connection before config metadata; is it cluster-mode?", addr)
    }
    return nil, err
}

When it happens

Trigger: Calling Resolve on memcachedAutoDiscovery when the server or connection returns an empty/short payload: connection closed before any data, a non-cluster memcached replying without a newline, or an I/O error on the socket.

Common situations: Pointing discovery at a plain (non-cluster-mode) memcached or ElastiCache node that does not support the cluster config command; network disconnects mid-request; proxies buffering or stripping the CONFIG response.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• expected 4 components in config metadata, and received
• failed to find config version
• failed to read nodes
• failed to parse config size from metadata
• failed to parser config version

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8aaac75f21e42290. Report an issue: GitHub.

Appendix: source

Thrown at pkg/discovery/memcache/resolver.go:65

	}
	if err := rw.Flush(); err != nil {
		return nil, err
	}

	config, err = s.parseConfig(rw.Reader)
	if err != nil {
		return nil, err
	}

	return config, err
}

func (s *memcachedAutoDiscovery) parseConfig(reader *bufio.Reader) (*clusterConfig, error) {
	clusterConfig := new(clusterConfig)

	configMeta, err := reader.ReadString('\n')
	if err != nil {
		return nil, fmt.Errorf("failed to read config metadata: %s", err)
	}
	configMeta = strings.TrimSpace(configMeta)

	// First line should be "CONFIG cluster 0 [length-of-payload-]
	configMetaComponents := strings.Split(configMeta, " ")
	if len(configMetaComponents) != 4 {
		return nil, fmt.Errorf("expected 4 components in config metadata, and received %d, meta: %s", len(configMetaComponents), configMeta)
	}

	configSize, err := strconv.Atoi(configMetaComponents[3])
	if err != nil {
		return nil, fmt.Errorf("failed to parse config size from metadata: %s, error: %s", configMeta, err)
	}

	configVersion, err := reader.ReadString('\n')
	if err != nil {
		return nil, fmt.Errorf("failed to find config version: %s", err)
	}

View on GitHub (pinned to 35b8b99117)