slackhq/nebula · error

unable to unmarshal pubkey

Error message

unable to unmarshal pubkey: %w

What it means

This error is returned by nistCurve.DH in noiseutil when the peer's public key bytes cannot be parsed as a valid public key on the configured NIST curve (P-256/P-384/P-521) via ecdh.Curve.NewPublicKey. It means the Noise handshake received a remote static/ephemeral public key that is not a valid, canonical point on the expected curve. The underlying ecdh error is wrapped with %w so the root cause is preserved.

Solutions

  1. Verify both peers use the same NIST curve (e.g. 'P256' in the Noise config) so key lengths match
  2. Regenerate or re-export the offending key pair with the same library used for parsing (crypto/ecdh)
  3. Check that the public key bytes are the full uncompressed point size for the curve (65/97/133 bytes) and not a truncated or DER-encoded copy
  4. Inspect the wrapped %w cause with errors.Unwrap/errors.As to distinguish length vs point-validity failures

Example fix

// before: mixing curves across peers
key, _ := nistCurve{curve: ecdh.P256()}.DH(priv, peerP384Pub) // fails to unmarshal
// after: ensure both sides use the same curve
if len(pub) != expectedPubLen(curveName) {
    return nil, fmt.Errorf("peer pubkey wrong size for %s", curveName)
}
key, err := nistCurve{curve: ecdh.P256()}.DH(priv, pub)
Defensive patterns

Strategy: validation

Validate before calling

func validNistPub(curveName string, pub []byte) bool {
    var n int
    switch curveName {
    case "P256": n = 65
    case "P384": n = 97
    case "P521": n = 133
    default: return false
    }
    return len(pub) == n && pub[0] == 4
}

Type guard

func isEcdhUnmarshalError(err error) bool {
    var e *ecdh.PublicKey // NewPublicKey failure is generic error; check wrapped text
    _ = e
    return err != nil && strings.Contains(err.Error(), "unable to unmarshal pubkey")
}

Try / catch

shared, err := curve.DH(priv, pub)
if err != nil {
    var cause error
    errors.As(err, &cause)
    return fmt.Errorf("handshake DH failed (check peer curve/key bytes): %w", err)
}

Prevention

When it happens

Trigger: Calling nistCurve.DH(privkey, pubkey) with pubkey bytes that are empty, the wrong length for the curve, or not a valid EC point — e.g. a remote peer configured with a key pair generated on a different curve or with a non-NIST key format.

Common situations: Mismatched curve configuration between peers (one on P-256, other on P-384); corrupted or truncated keys stored in config/certs; keys generated by a different library with a different encoding; a peer sending garbage during a handshake attempt.

Related errors

AI-assisted analysis of slackhq/nebula@dd8f660c0a (2026-09-03). Data as JSON: /api/errors/b80b2d70829c2ced. Report an issue: GitHub.

Appendix: source

Thrown at noiseutil/nist.go:47

	}
}

func (c nistCurve) GenerateKeypair(rng io.Reader) (noise.DHKey, error) {
	if rng == nil {
		rng = rand.Reader
	}
	privkey, err := c.curve.GenerateKey(rng)
	if err != nil {
		return noise.DHKey{}, err
	}
	pubkey := privkey.PublicKey()
	return noise.DHKey{Private: privkey.Bytes(), Public: pubkey.Bytes()}, nil
}

func (c nistCurve) DH(privkey, pubkey []byte) ([]byte, error) {
	ecdhPubKey, err := c.curve.NewPublicKey(pubkey)
	if err != nil {
		return nil, fmt.Errorf("unable to unmarshal pubkey: %w", err)
	}
	ecdhPrivKey, err := c.curve.NewPrivateKey(privkey)
	if err != nil {
		return nil, fmt.Errorf("unable to unmarshal private key: %w", err)
	}

	return ecdhPrivKey.ECDH(ecdhPubKey)
}

func (c nistCurve) DHLen() int {
	// NOTE: Noise Protocol specifies "DHLen" to represent two things:
	// - The size of the public key
	// - The return size of the DH() function
	// But for standard NIST ECDH, the sizes of these are different.
	// Luckily, the flynn/noise library actually only uses this DHLen()
	// value to represent the public key size, so that is what we are
	// returning here. The length of the DH() return bytes are unaffected by
	// this value here.

View on GitHub (pinned to dd8f660c0a)