AlexxIT/go2rtc · error

dtls: server handshake failed

Error message

dtls: server handshake failed: %w

What it means

AVServStart acts as the DTLS server on the back channel and wraps any failure of NewDTLSServer (the PSK-based DTLS server handshake) with this error. It means the DTLS server-side handshake with the camera did not complete, so the AV login exchange cannot proceed.

Solutions

1. Verify the PSK matches the one derived from the device credentials
2. Check that the camera is online and initiating contact on the back channel
3. Confirm UDP traffic is not blocked by firewall/NAT
4. Enable verbose logging to see how far the handshake progressed

if psk == nil || len(psk) == 0 {
	return fmt.Errorf("missing DTLS PSK")
}

if err := conn.AVServStart(); err != nil {
	var he *dtls.HandshakeError
	if errors.As(err, &he) {
		// re-run StartIntercom after verifying PSK
	}
	return err
}

When it happens

Trigger: Calling StartIntercom -> AVServStart when NewDTLSServer fails: wrong PSK, client (camera) never initiates the DTLS handshake, or the underlying pion/dtls listener errors.

Common situations: PSK mismatch between host and camera firmware; camera not attempting to connect on the back channel; UDP path blocked so handshake packets never arrive; certificate/cipher suite incompatibility in the DTLS stack.

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• dtls: client handshake failed
• av login 2 failed
• wrong response:
• rtmp: wrong response %#v
• av login 1 failed

AI-assisted analysis of AlexxIT/go2rtc@c245815e75 (2026-09-07). Data as JSON: /api/errors/6f703351267f5938. Report an issue: GitHub.

Appendix: source

Thrown at pkg/tutk/dtls/conn_dtls.go:232

								ack := c.msgACK()
								c.clientConn.Write(ack)
							}
						}
					}
				}()

				return nil
			}
		case <-timer.C:
			return context.DeadlineExceeded
		}
	}
}

func (c *DTLSConn) AVServStart() error {
	conn, err := NewDTLSServer(c.ctx, iotcChannelBack, c.addr, c.WriteDTLS, c.serverBuf, c.psk)
	if err != nil {
		return fmt.Errorf("dtls: server handshake failed: %w", err)
	}

	if c.verbose {
		fmt.Printf("[DTLS] Server handshake complete on channel %d\n", iotcChannelBack)
		fmt.Printf("[SERVER] Waiting for AV Login request from camera...\n")
	}

	// Wait for AV Login request from camera
	buf := make([]byte, 1024)
	conn.SetReadDeadline(time.Now().Add(5 * time.Second))
	n, err := conn.Read(buf)
	if err != nil {
		go conn.Close()
		return fmt.Errorf("read av login: %w", err)
	}

	if c.verbose {
		fmt.Printf("[SERVER] AV Login request len=%d data:\n%s", n, hexDump(buf[:n]))

View on GitHub (pinned to c245815e75)