failed to unmarshal apiserver addresses from etcd: %v

Error message

failed to unmarshal apiserver addresses from etcd: %v

What it means

The etcd key <program>/apiaddresses (k3s/apiaddresses, pkg/etcd/etcd.go:92) stores a JSON array of apiserver addresses that agents use to find servers. When the key exists and is non-empty but does not unmarshal into []string, this error is returned - the stored value is not valid JSON or not an array of strings.

Source

Thrown at pkg/etcd/etcd.go:1625

func GetAPIServerURLsFromETCD(ctx context.Context, cfg *config.Control) ([]string, error) {
	cl, conn, err := getClient(ctx, cfg)
	if err != nil {
		return nil, err
	}
	defer conn.Close()

	etcdResp, err := cl.KV.Get(ctx, AddressKey)
	if err != nil {
		return nil, err
	}

	if etcdResp.Count == 0 || len(etcdResp.Kvs[0].Value) == 0 {
		return nil, ErrAddressNotSet
	}

	var addresses []string
	if err := json.Unmarshal(etcdResp.Kvs[0].Value, &addresses); err != nil {
		return nil, fmt.Errorf("failed to unmarshal apiserver addresses from etcd: %v", err)
	}

	return addresses, nil
}

// GetMembersClientURLs will list through the member lists in etcd and return
// back a combined list of client urls for each member in the cluster
func (e *ETCD) GetMembersClientURLs(ctx context.Context) ([]string, error) {
	if e.client == nil {
		if err := e.startClient(ctx); err != nil {
			return nil, err
		}
	}

	members, err := e.client.MemberList(ctx)
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to 6ba341e396)

Solutions

Inspect the current value: `etcdctl get k3s/apiaddresses` using the k3s etcd client certs - it should look like ["https://10.0.0.1:6443",...].
Fix it in place with a proper JSON array of server URLs, or delete the key (`etcdctl del k3s/apiaddresses`) so servers re-register and rewrite it on next start.
Restart the k3s servers after fixing/deleting; verify agents reconnect. Avoid writing k3s-internal etcd keys manually going forward.

Example fix

# before (corrupt value)
etcdctl get k3s/apiaddresses
> 10.0.0.1:6443,10.0.0.2:6443        # not JSON -> unmarshal fails

# after
etcdctl del k3s/apiaddresses          # let servers rewrite it on restart
systemctl restart k3s                  # agents reconnect via re-registered addresses

Defensive patterns

Strategy: try-catch

Validate before calling

// Validate the stored value before relying on it:
resp, err := cli.KV.Get(ctx, "k3s/apiaddresses")
if err != nil { log.Fatal(err) }
if resp.Count > 0 {
    var addrs []string
    if json.Unmarshal(resp.Kvs[0].Value, &addrs) != nil || !allHTTPS(addrs) {
        log.Fatal("k3s/apiaddresses holds malformed data - delete the key and let servers re-register")
    }
}

Try / catch

// Catch unmarshal failure, self-heal by deleting the key, then retry:
addrs, err := getAddresses(ctx, cli)
if err != nil {
    if strings.Contains(err.Error(), "failed to unmarshal apiserver addresses") {
        if _, delErr := cli.Delete(ctx, "k3s/apiaddresses"); delErr != nil { log.Fatal(delErr) }
        addrs, err = getAddresses(ctx, cli) // servers rewrite it as they start
    }
    if err != nil { log.Fatal(err) }
}

Prevention

Never write k3s-internal etcd keys manually; use k3s's own registration path.
Audit direct etcd access in automation (etcdctl put) against reserved key prefixes.
Restore only snapshots produced by the same k3s major version line.

When it happens

Trigger: cl.KV.Get(AddressKey) succeeds, Count>0 with non-empty value, but json.Unmarshal into []string fails (pkg/etcd/etcd.go:1621-1626): the value was manually overwritten, corrupted, or written by an incompatible/external tool rather than by k3s itself.

Common situations: Someone ran etcdctl put on k3s/apiaddresses with a non-JSON or differently-shaped value; external automation poking k3s's etcd directly; partial write during an incident; restoring an old/foreign snapshot into a newer k3s.

Related errors

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/5f0531da38c3a226. Report an issue: GitHub.