slackhq/nebula · error

unmarshal handshake: %w

Error message

unmarshal handshake: %w

What it means

processPayload wraps errors from UnmarshalPayload as 'unmarshal handshake: %w' and marks the machine failed. The Noise handshake message decrypted successfully, but its Nebula handshake payload bytes could not be deserialized — the payload structure is not a valid handshake Payload protobuf.

Source

Thrown at handshake/machine.go:297

	m.result.EKey = eKey
	m.result.DKey = dKey
	m.result.MessageIndex = uint64(m.hs.MessageIndex())
	return m.result
}

func (m *Machine) processPayload(msg []byte, flags msgFlags) error {
	if len(msg) == 0 {
		if flags.expectsPayload || flags.expectsCert {
			m.failed = true
			return ErrMissingContent
		}
		return nil
	}

	payload, err := UnmarshalPayload(msg)
	if err != nil {
		m.failed = true
		return fmt.Errorf("unmarshal handshake: %w", err)
	}

	// Assert the payload contains exactly what we expect
	hasPayloadData := payload.InitiatorIndex != 0 || payload.ResponderIndex != 0 || payload.Time != 0
	if hasPayloadData != flags.expectsPayload {
		m.failed = true
		return ErrUnexpectedContent
	}

	hasCertData := len(payload.Cert) > 0
	if hasCertData != flags.expectsCert {
		m.failed = true
		return ErrUnexpectedContent
	}

	// Process payload
	if flags.expectsPayload {
		var remoteIndex uint32

View on GitHub (pinned to dd8f660c0a)

Solutions

1. Upgrade/downgrade peers to a Nebula version with a compatible handshake payload encoding
2. Start a fresh handshake (machine is marked failed; the existing Machine cannot retry)
3. Confirm the remote endpoint actually runs Nebula and not another Noise-protocol service
4. Check for packet corruption on the path (NIC offload bugs, tunnel-in-tunnel MTU issues)

_, _, err := machine.ProcessPacket(pkt)
if err != nil && strings.Contains(err.Error(), "unmarshal handshake:") {
	// machine is marked failed — tear it down and start a new handshake
	connManager.resetHandshake(remote)
}

When it happens

Trigger: ProcessPacket -> processPayload on a successfully decrypted Noise message whose plaintext is not a well-formed Payload (wrong version encoding, corrupted inner bytes, or a foreign implementation's payload layout).

Common situations: Mixed Nebula versions with incompatible payload encodings, a third-party Noise peer speaking the same pattern but different payload schema, bit corruption surviving the AEAD check (rare), faked packets from an attacker holding a derived key.

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• ErrNoPayload
• ErrNoPeerStaticKey
• ErrInitiateOnResponder
• ErrInitiateAlreadyCalled
• ErrInitiateNotCalled

AI-assisted analysis of slackhq/nebula@dd8f660c0a (2026-09-03). Data as JSON: /api/errors/d5e6083c90a2e41a. Report an issue: GitHub.