{"record":{"id":"5f0531da38c3a226","repo":"k3s-io/k3s","slug":"failed-to-unmarshal-apiserver-addresses-from-etcd","errorCode":null,"errorMessage":"failed to unmarshal apiserver addresses from etcd: %v","messagePattern":"failed to unmarshal apiserver addresses from etcd: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/etcd/etcd.go","lineNumber":1625,"sourceCode":"func GetAPIServerURLsFromETCD(ctx context.Context, cfg *config.Control) ([]string, error) {\n\tcl, conn, err := getClient(ctx, cfg)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tdefer conn.Close()\n\n\tetcdResp, err := cl.KV.Get(ctx, AddressKey)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif etcdResp.Count == 0 || len(etcdResp.Kvs[0].Value) == 0 {\n\t\treturn nil, ErrAddressNotSet\n\t}\n\n\tvar addresses []string\n\tif err := json.Unmarshal(etcdResp.Kvs[0].Value, &addresses); err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to unmarshal apiserver addresses from etcd: %v\", err)\n\t}\n\n\treturn addresses, nil\n}\n\n// GetMembersClientURLs will list through the member lists in etcd and return\n// back a combined list of client urls for each member in the cluster\nfunc (e *ETCD) GetMembersClientURLs(ctx context.Context) ([]string, error) {\n\tif e.client == nil {\n\t\tif err := e.startClient(ctx); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t}\n\n\tmembers, err := e.client.MemberList(ctx)\n\tif err != nil {\n\t\treturn nil, err\n\t}","sourceCodeStart":1607,"sourceCodeEnd":1643,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/etcd.go#L1607-L1643","documentation":"The etcd key <program>/apiaddresses (k3s/apiaddresses, pkg/etcd/etcd.go:92) stores a JSON array of apiserver addresses that agents use to find servers. When the key exists and is non-empty but does not unmarshal into []string, this error is returned - the stored value is not valid JSON or not an array of strings.","triggerScenarios":"cl.KV.Get(AddressKey) succeeds, Count>0 with non-empty value, but json.Unmarshal into []string fails (pkg/etcd/etcd.go:1621-1626): the value was manually overwritten, corrupted, or written by an incompatible/external tool rather than by k3s itself.","commonSituations":"Someone ran etcdctl put on k3s/apiaddresses with a non-JSON or differently-shaped value; external automation poking k3s's etcd directly; partial write during an incident; restoring an old/foreign snapshot into a newer k3s.","solutions":["Inspect the current value: `etcdctl get k3s/apiaddresses` using the k3s etcd client certs - it should look like [\"https://10.0.0.1:6443\",...].","Fix it in place with a proper JSON array of server URLs, or delete the key (`etcdctl del k3s/apiaddresses`) so servers re-register and rewrite it on next start.","Restart the k3s servers after fixing/deleting; verify agents reconnect. Avoid writing k3s-internal etcd keys manually going forward."],"exampleFix":"# before (corrupt value)\netcdctl get k3s/apiaddresses\n> 10.0.0.1:6443,10.0.0.2:6443        # not JSON -> unmarshal fails\n\n# after\netcdctl del k3s/apiaddresses          # let servers rewrite it on restart\nsystemctl restart k3s                  # agents reconnect via re-registered addresses","handlingStrategy":"try-catch","validationCode":"// Validate the stored value before relying on it:\nresp, err := cli.KV.Get(ctx, \"k3s/apiaddresses\")\nif err != nil { log.Fatal(err) }\nif resp.Count > 0 {\n    var addrs []string\n    if json.Unmarshal(resp.Kvs[0].Value, &addrs) != nil || !allHTTPS(addrs) {\n        log.Fatal(\"k3s/apiaddresses holds malformed data - delete the key and let servers re-register\")\n    }\n}","typeGuard":null,"tryCatchPattern":"// Catch unmarshal failure, self-heal by deleting the key, then retry:\naddrs, err := getAddresses(ctx, cli)\nif err != nil {\n    if strings.Contains(err.Error(), \"failed to unmarshal apiserver addresses\") {\n        if _, delErr := cli.Delete(ctx, \"k3s/apiaddresses\"); delErr != nil { log.Fatal(delErr) }\n        addrs, err = getAddresses(ctx, cli) // servers rewrite it as they start\n    }\n    if err != nil { log.Fatal(err) }\n}","preventionTips":["Never write k3s-internal etcd keys manually; use k3s's own registration path.","Audit direct etcd access in automation (etcdctl put) against reserved key prefixes.","Restore only snapshots produced by the same k3s major version line."],"tags":["etcd","json","apiserver","networking","k3s"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}