{"record":{"id":"2e3d4d0c97ef200b","repo":"slackhq/nebula","slug":"sendmmsg-made-no-progress","errorCode":null,"errorMessage":"sendmmsg made no progress","messagePattern":"sendmmsg made no progress","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"udp/udp_linux_writebatch.go","lineNumber":277,"sourceCode":"\t\t// Drain the packed entries without repacking: everything the packing\n\t\t// loop wired (iovecs, names, cmsgs) stays intact until the next chunk\n\t\t// overwrites it, so a partial success resumes the same sendmmsg array\n\t\t// at the first unsent entry, and a rejected entry is skipped in place.\n\t\t// Only the GSO-disable path replans, since its entries change shape.\n\t\tdone := 0\n\t\tfor done < entry {\n\t\t\tsent, serr := w.sendFn(done, entry-done)\n\t\t\tif sent > 0 {\n\t\t\t\t// Count packets per entry; the bufs index span would\n\t\t\t\t// overcount across holes left by skipped runs.\n\t\t\t\tfor e := done; e < done+sent; e++ {\n\t\t\t\t\twritten += w.entryPkts[e]\n\t\t\t\t}\n\t\t\t\tdone += sent\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tif serr == nil {\n\t\t\t\treturn written, fmt.Errorf(\"sendmmsg made no progress\")\n\t\t\t}\n\t\t\t// sent<=0 means the first remaining entry itself failed.\n\t\t\t// EIO on a superpacket means the route cannot carry a GSO send even though the setsockopt probe passed:\n\t\t\t// udp_send_skb() returns EIO when:\n\t\t\t//   * the egress device lacks TX checksum offload (kernels through 6.10)\n\t\t\t//   * or when an xfrm policy covers the route.\n\t\t\t// Persistent, so disable GSO and replay from the failed run as one-packet entries.\n\t\t\tif w.gsoSupported && w.entryPkts[done] >= 2 && errors.Is(serr, unix.EIO) {\n\t\t\t\tw.gsoSupported = false\n\t\t\t\tw.l.Warn(\"udp: kernel rejected GSO send, disabling GSO\", \"error\", serr)\n\t\t\t\trecordCapability(\"udp.gso.enabled\", false)\n\t\t\t\ti = w.entryEnd[done] - w.entryPkts[done]\n\t\t\t\tbreak\n\t\t\t}\n\t\t\t// Any other zero-sent error is a per-entry failure.\n\t\t\t// Transient errnos (EINTR, ENOBUFS) were already retried inside sendFn.\n\t\t\t// These packets are doomed. Log them and move on.\n\t\t\tif w.l.Enabled(context.Background(), slog.LevelDebug) {","sourceCodeStart":259,"sourceCodeEnd":295,"githubUrl":"https://github.com/slackhq/nebula/blob/dd8f660c0ac37903ec4080ca4d3c861ba9342ceb/udp/udp_linux_writebatch.go#L259-L295","documentation":"During a chunked GSO send, WriteBatch retried sendmmsg after a partial/failed send and the syscall returned success but sent zero entries, meaning no forward progress was possible; the library treats this as a stall and aborts the call, returning what was written so far plus this error.","triggerScenarios":"sendmmsg returning sent==0 with no error while entries remain in the chunk — e.g. kernel quirk or edge case in GSO (UDP_SEGMENT) handling where a zero-count success is returned repeatedly.","commonSituations":"GSO sends over routes with odd offload characteristics; kernels with sendmmsg/UDP GSO regressions; networking environments (xfrm, unusual devices) interacting badly with superpacket sends.","solutions":["Update the Linux kernel to a version with current UDP GSO fixes","Disable UDP offloads (listeners.batch/offload settings, or disable GSO/UDP_SEGMENT) to avoid the superpacket path","Treat the returned written count as authoritative — packets before the stall were sent; re-drive remaining sends at the caller if the API permits","Report to upstream (nebula) with kernel version if reproducible, as sent==0 with nil error is unexpected"],"exampleFix":"// before\nlisteners:\n  batch: 64\n// after (avoid GSO path)\nlisteners:\n  batch: 1","handlingStrategy":"retry","validationCode":"// avoid the GSO path if the environment is known-bad\nif kernelSupportsUDPGSOCleanly() == false {\n    disableBatchOffloads()\n}","typeGuard":null,"tryCatchPattern":"n, err := w.WriteBatch(bufs, addrs)\nif err != nil {\n    if strings.Contains(err.Error(), \"sendmmsg made no progress\") {\n        // n packets already sent; re-drive remainder without GSO/batching\n        return n, retryWithoutGSO(bufs[min(n, len(bufs)):], addrs[n:])\n    }\n    return n, err\n}","preventionTips":["Run kernels with current UDP GSO fixes","Disable UDP offload/GSO on routes known to misbehave (xfrm, odd NICs)","Handle the short-count return: never assume bufs were all sent","Watch upstream nebula/kernel changelogs for sendmmsg regressions"],"tags":["udp","linux","sendmmsg","gso","stall"],"backgroundTag":"sendmmsg-no-progress","analyzedSha":"dd8f660c0ac37903ec4080ca4d3c861ba9342ceb","analyzedAt":"2026-09-03T11:13:55.444Z","contentChangedAt":"2026-09-03T11:13:55.444Z","schemaVersion":2},"datasetVersion":"2026-09-10T17:17:09.494Z"}