What causes "invalid handshake message" in slackhq/nebula?

Verify both peers use the same handshake wire format version (protobuf-compatible NebulaHandshake)

invalid handshake message

Error message

invalid handshake message

What it means

errInvalidHandshakeMessage is returned by UnmarshalPayload when the handshake bytes are not a well-formed protobuf wire message — specifically when protowire.ConsumeTag fails (n < 0), meaning the buffer is corrupt, truncated, or not protobuf-encoded at the field-tag level.

Source

Thrown at handshake/payload.go:11

package handshake

import (
	"errors"
	"math"

	"google.golang.org/protobuf/encoding/protowire"
)

var (
	errInvalidHandshakeMessage = errors.New("invalid handshake message")
	errInvalidHandshakeDetails = errors.New("invalid handshake details")
)

// Payload represents the decoded fields of a handshake message.
// Wire format is protobuf-compatible with NebulaHandshake{Details: NebulaHandshakeDetails{...}}.
type Payload struct {
	Cert           []byte
	InitiatorIndex uint32
	ResponderIndex uint32
	Time           uint64
	CertVersion    uint32
}

// Proto field numbers for NebulaHandshakeDetails
const (
	fieldCert           = 1 // bytes
	fieldInitiatorIndex = 2 // uint32
	fieldResponderIndex = 3 // uint32

View on GitHub (pinned to dd8f660c0a)

Solutions

Verify both peers use the same handshake wire format version (protobuf-compatible NebulaHandshake)
Check for network corruption or fragmentation issues (MTU/UDP path) between the peers
Capture the packet and validate it decodes as NebulaHandshake{Details: NebulaHandshakeDetails}

Defensive patterns

Strategy: try-catch

Validate before calling

if len(b) == 0 {
    return fmt.Errorf("empty handshake payload")
}

Try / catch

p, err := UnmarshalPayload(b)
if errors.Is(err, errInvalidHandshakeMessage) {
    // drop malformed packet / count as parse failure
    return nil, err
}

Prevention

Keep handshake wire-format versions in sync across peers
Validate packets before unmarshaling (length, version byte)
Monitor parse-failure rates to detect corruption or hostile traffic

When it happens

Trigger: Passing truncated, corrupted, or non-protobuf bytes to UnmarshalPayload; a peer speaking a different/incompatible wire format; bit flips or MTU fragmentation mangling the handshake datagram.

Common situations: Version skew where one peer sends the legacy Nebula struct-encoded handshake and the other expects protobuf-compatible encoding; packets corrupted in transit; tests feeding random bytes.

Understand the failure class

SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

AI-assisted analysis of slackhq/nebula@dd8f660c0a (2026-09-03). Data as JSON: /api/errors/1652b0ccef63107c. Report an issue: GitHub.