thanos-io/thanos · warning

server=

Error message

server=%s

What it means

memcachedClient.Set wraps an error returned by the memcached set operation with the address of the backend server that was picked for the key. The wrap preserves the original error (timeout, server down, etc.) and annotates it with the failing server's address for debugging.

Solutions

1. Inspect the wrapped original error to determine the failure class (timeout vs connection vs server error).
2. Check the health of the memcached server named in the message (server=<addr>).
3. Set a sane memcached timeout and enable the client's server-update loop so the server list stays fresh.
4. Retry the store; Set failures are best-effort in this cache and non-fatal to query serving.

Example fix

// before: hard to tell which node failed
return err
// after: error now annotated
return errors.Wrapf(err, "server=%s", addr)

if err := pingMemcached(addr, 500*time.Millisecond); err != nil { log.Warnf("memcached %s unhealthy before Set: %v", addr, err) }

err := c.Set(ctx, key, val, ttl)
if err != nil {
    var se *memcachedServerError
    if errors.As(err, &se) { /* server rejected item */ } else { /* transient: retry once or degrade */ }
    log.Warnf("cache set failed (best-effort): %v", err)
}

When it happens

Trigger: Calling Set (via the cache Store path) after PickServer succeeded, but the actual memcached SET command failed (connection error, timeout, server error response).

Common situations: Memcached restarted or evicting connections mid-operation; network partition to one memcached node; item larger than the server's item size limit; timeouts under load.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• failed to get object attributes
• sync before first pass of downsampling
• sync before second pass of downsampling
• initializing the query range cache config
• initializing the labels cache config

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4a0a331f84e23e0d. Report an issue: GitHub.

Appendix: source

Thrown at internal/cortex/chunk/cache/memcached_client.go:225

	// Skip hitting memcached at all if the item is bigger than the max allowed size.
	if c.maxItemSize > 0 && len(item.Value) > c.maxItemSize {
		c.skipped.Inc()
		return nil
	}

	err := c.Client.Set(item)
	if err == nil {
		return nil
	}

	// Inject the server address in order to have more information about which memcached
	// backend server failed. This is a best effort.
	addr, addrErr := c.serverList.PickServer(item.Key)
	if addrErr != nil {
		return err
	}

	return errors.Wrapf(err, "server=%s", addr)
}

func (c *memcachedClient) updateLoop(updateInterval time.Duration) {
	defer c.wait.Done()
	ticker := time.NewTicker(updateInterval)
	for {
		select {
		case <-ticker.C:
			err := c.updateMemcacheServers()
			if err != nil {
				level.Warn(c.logger).Log("msg", "error updating memcache servers", "err", err)
			}
		case <-c.quit:
			ticker.Stop()
			return
		}
	}
}

View on GitHub (pinned to 35b8b99117)