ErrSubtypeMismatch

ErrSubtypeMismatch

Error message

packet subtype does not match handshake machine subtype

What it means

ErrSubtypeMismatch is returned by ProcessPacket when the packet's MessageSubType byte does not match the subtype the handshake Machine was constructed for. The machine drops the packet without failing, so a legitimate retransmit can still complete later.

Source

Thrown at handshake/errors.go:21

import "errors"

var (
	ErrInitiateOnResponder     = errors.New("initiate called on responder")
	ErrInitiateAlreadyCalled   = errors.New("initiate already called")
	ErrInitiateNotCalled       = errors.New("initiate must be called before ProcessPacket for initiators")
	ErrPacketTooShort          = errors.New("packet too short")
	ErrPublicKeyMismatch       = errors.New("public key mismatch between certificate and handshake")
	ErrIncompleteHandshake     = errors.New("handshake completed without receiving required content")
	ErrMachineFailed           = errors.New("handshake machine has failed")
	ErrUnknownSubtype          = errors.New("unknown handshake subtype")
	ErrMissingContent          = errors.New("expected handshake content but message was empty")
	ErrUnexpectedContent       = errors.New("received unexpected handshake content")
	ErrInvalidRemoteIndex      = errors.New("peer sent an invalid index in handshake payload")
	ErrIndexAllocation         = errors.New("failed to allocate local index")
	ErrNoCredential            = errors.New("no handshake credential available for cert version")
	ErrAsymmetricCipherKeys    = errors.New("noise produced only one cipher key")
	ErrMultiMessageUnsupported = errors.New("multi-message handshake patterns are not yet supported by the manager")
	ErrSubtypeMismatch         = errors.New("packet subtype does not match handshake machine subtype")
)

View on GitHub (pinned to dd8f660c0a)

Solutions

Verify both peers configure the same handshake message subtype (e.g. HandshakeIXPSK0 vs HandshakeXXPSK0)
Check that the UDP listener isn't receiving handshake packets from other tunnels or hosts (stray traffic); safe to ignore
Inspect packet[1] in the offending packet to confirm the subtype being sent

Example fix

// before
machine := NewMachine(cs, version, initiator, HandshakeXXPSK0)
// after (match the peer)
machine := NewMachine(cs, version, initiator, HandshakeIXPSK0)

Defensive patterns

Strategy: validation

Validate before calling

if header.MessageSubType(packet[1]) != expectedSubtype {
    // drop packet; would cause handshake.ErrSubtypeMismatch
    return
}

Try / catch

_, _, err := m.ProcessPacket(nil, packet)
if errors.Is(err, handshake.ErrSubtypeMismatch) {
    // safe to ignore: machine is not killed, legit retransmits still work
    return
}

Prevention

Use identical handshake subtype config on both peers
Filter stray handshake traffic on shared UDP ports
Never hardcode subtype bytes; use header constants

When it happens

Trigger: Feeding ProcessPacket (handshake/machine.go:216) a handshake packet whose header subtype differs from m.subtype — e.g. an IX PSK0 packet delivered to a machine expecting XX PSK0, or cross-wired UDP ports delivering foreign handshake packets.

Common situations: Two nodes configured with different handshake subtypes trying to connect; stray handshake traffic from another tunnel/overlay landing on the same UDP port; misconfigured firewall/port-forward routing handshake packets to the wrong host.

Understand the failure class

SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

AI-assisted analysis of slackhq/nebula@dd8f660c0a (2026-09-03). Data as JSON: /api/errors/1e3f661dca1ecee6. Report an issue: GitHub.