{"record":{"id":"e9daccf089e67850","repo":"thanos-io/thanos","slug":"failed-to-read-nodes-s","errorCode":null,"errorMessage":"failed to read nodes: %s","messagePattern":"failed to read nodes: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/discovery/memcache/resolver.go","lineNumber":91,"sourceCode":"\t}\n\n\tconfigSize, err := strconv.Atoi(configMetaComponents[3])\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to parse config size from metadata: %s, error: %s\", configMeta, err)\n\t}\n\n\tconfigVersion, err := reader.ReadString('\\n')\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to find config version: %s\", err)\n\t}\n\tclusterConfig.version, err = strconv.Atoi(strings.TrimSpace(configVersion))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to parser config version: %s\", err)\n\t}\n\n\tnodes, err := reader.ReadString('\\n')\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to read nodes: %s\", err)\n\t}\n\n\tif len(configVersion)+len(nodes) != configSize {\n\t\treturn nil, fmt.Errorf(\"expected %d in config payload, but got %d instead\", configSize, len(configVersion)+len(nodes))\n\t}\n\n\tfor host := range strings.SplitSeq(strings.TrimSpace(nodes), \" \") {\n\t\tdnsIpPort := strings.Split(host, \"|\")\n\t\tif len(dnsIpPort) != 3 {\n\t\t\treturn nil, fmt.Errorf(\"node not in expected format: %s\", dnsIpPort)\n\t\t}\n\t\tport, err := strconv.Atoi(dnsIpPort[2])\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to parse port: %s, err: %s\", dnsIpPort, err)\n\t\t}\n\t\tclusterConfig.nodes = append(clusterConfig.nodes, node{dns: dnsIpPort[0], ip: dnsIpPort[1], port: port})\n\t}\n","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/discovery/memcache/resolver.go#L73-L109","documentation":"After the version line, parseConfig reads the nodes line (space-separated 'host|ip|port' entries). The protocol requires that len(versionLine)+len(nodesLine) equals the configSize declared in the metadata; if not, the payload is incomplete or misframed and this error is returned with both expected and actual byte counts.","triggerScenarios":"The server closes the stream before sending the full node list (ReadString returns a partial line at EOF), or the declared configSize does not match the actual version+nodes bytes — e.g. CRLF vs LF counting bugs or cluster membership changing mid-read.","commonSituations":"ElastiCache returning a payload shorter than advertised during failover; CRLF vs LF length accounting bugs in custom servers; network truncation of the final line; stale configSize after a config change.","solutions":["Make the server compute configSize as the exact byte length of the version+nodes lines it emits (mind \\r\\n counting).","Compare expected vs got counts: a 1-2 byte delta suggests a line-ending bug; a large delta suggests truncation.","Retry Resolve — payloads can be inconsistent during cluster reconfiguration.","Capture traffic (tcpdump) against a known-good ElastiCache response to fix framing."],"exampleFix":"// before\npayload := fmt.Sprintf(\"1\\n%s\", nodes)\nheader := fmt.Sprintf(\"CONFIG cluster 0 %d\\r\\n\", len(payload)+len(\"\\r\\n\")) // overcounts\n// after\npayload := fmt.Sprintf(\"1\\r\\n%s\\r\\n\", nodes)\nheader := fmt.Sprintf(\"CONFIG cluster 0 %d\\r\\n\", len(payload))","handlingStrategy":"validation","validationCode":"func payloadLengthMatches(body string, declared int) error {\n    if got := len(body); got != declared {\n        return fmt.Errorf(\"declared %d bytes, got %d\", declared, got)\n    }\n    return nil\n}","typeGuard":null,"tryCatchPattern":"cfg, err := discovery.parseConfig(reader)\nif err != nil && strings.Contains(err.Error(), \"in config payload\") {\n    return retryFetchConfig() // likely truncation or reconfiguration race\n}","preventionTips":["Compute configSize from the exact bytes of the version+nodes payload (count \\r\\n consistently).","Prefer writing the payload and its length from one code path to avoid drift.","Retry fetches during ElastiCache reconfiguration windows.","Diff live traffic against declared sizes when debugging framing."],"tags":["memcached","protocol","framing","elasticache"],"backgroundTag":"checksum-mismatch","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}