slackhq/nebula · error

sendmmsg made no progress

Error message

sendmmsg made no progress

What it means

During a chunked GSO send, WriteBatch retried sendmmsg after a partial/failed send and the syscall returned success but sent zero entries, meaning no forward progress was possible; the library treats this as a stall and aborts the call, returning what was written so far plus this error.

Source

Thrown at udp/udp_linux_writebatch.go:277

		// Drain the packed entries without repacking: everything the packing
		// loop wired (iovecs, names, cmsgs) stays intact until the next chunk
		// overwrites it, so a partial success resumes the same sendmmsg array
		// at the first unsent entry, and a rejected entry is skipped in place.
		// Only the GSO-disable path replans, since its entries change shape.
		done := 0
		for done < entry {
			sent, serr := w.sendFn(done, entry-done)
			if sent > 0 {
				// Count packets per entry; the bufs index span would
				// overcount across holes left by skipped runs.
				for e := done; e < done+sent; e++ {
					written += w.entryPkts[e]
				}
				done += sent
				continue
			}
			if serr == nil {
				return written, fmt.Errorf("sendmmsg made no progress")
			}
			// sent<=0 means the first remaining entry itself failed.
			// EIO on a superpacket means the route cannot carry a GSO send even though the setsockopt probe passed:
			// udp_send_skb() returns EIO when:
			//   * the egress device lacks TX checksum offload (kernels through 6.10)
			//   * or when an xfrm policy covers the route.
			// Persistent, so disable GSO and replay from the failed run as one-packet entries.
			if w.gsoSupported && w.entryPkts[done] >= 2 && errors.Is(serr, unix.EIO) {
				w.gsoSupported = false
				w.l.Warn("udp: kernel rejected GSO send, disabling GSO", "error", serr)
				recordCapability("udp.gso.enabled", false)
				i = w.entryEnd[done] - w.entryPkts[done]
				break
			}
			// Any other zero-sent error is a per-entry failure.
			// Transient errnos (EINTR, ENOBUFS) were already retried inside sendFn.
			// These packets are doomed. Log them and move on.
			if w.l.Enabled(context.Background(), slog.LevelDebug) {

View on GitHub (pinned to dd8f660c0a)

Solutions

1. Update the Linux kernel to a version with current UDP GSO fixes
2. Disable UDP offloads (listeners.batch/offload settings, or disable GSO/UDP_SEGMENT) to avoid the superpacket path
3. Treat the returned written count as authoritative — packets before the stall were sent; re-drive remaining sends at the caller if the API permits
4. Report to upstream (nebula) with kernel version if reproducible, as sent==0 with nil error is unexpected

Example fix

// before
listeners:
  batch: 64
// after (avoid GSO path)
listeners:
  batch: 1

// avoid the GSO path if the environment is known-bad
if kernelSupportsUDPGSOCleanly() == false {
    disableBatchOffloads()
}

n, err := w.WriteBatch(bufs, addrs)
if err != nil {
    if strings.Contains(err.Error(), "sendmmsg made no progress") {
        // n packets already sent; re-drive remainder without GSO/batching
        return n, retryWithoutGSO(bufs[min(n, len(bufs)):], addrs[n:])
    }
    return n, err
}

When it happens

Trigger: sendmmsg returning sent==0 with no error while entries remain in the chunk — e.g. kernel quirk or edge case in GSO (UDP_SEGMENT) handling where a zero-count success is returned repeatedly.

Common situations: GSO sends over routes with odd offload characteristics; kernels with sendmmsg/UDP GSO regressions; networking environments (xfrm, unusual devices) interacting badly with superpacket sends.

Related errors

• queue set already closed
• unable to open socket: %w
• unable to set SO_REUSEPORT: %w
• unable to bind to socket: %w
• unsupported sock type: %T

AI-assisted analysis of slackhq/nebula@dd8f660c0a (2026-09-03). Data as JSON: /api/errors/2e3d4d0c97ef200b. Report an issue: GitHub.