What causes "Server response contained invalid data." in jstedfast/MailKit?

Inspect the raw server token sent in the final DIGEST-MD5 step (log Challenge input) and confirm the server actually emits rspauth.

jstedfast/MailKit · error · SaslException

InvalidChallenge

InvalidChallenge

Error message

Server response contained invalid data.

What it means

During the final step of a DIGEST-MD5 SASL exchange, the server's response is scanned for the 'rspauth' key-value pair. MailKit throws this SaslException (SaslErrorCode.InvalidChallenge) when the response contains a valid key=value pair whose key is anything other than 'rspauth', meaning the server sent unexpected authentication data instead of the required response-auth verification.

Solutions

Inspect the raw server token sent in the final DIGEST-MD5 step (log Challenge input) and confirm the server actually emits rspauth.
Verify you are using the correct SASL mechanism for this server; fall back to a simpler mechanism like CRAM-MD5 or PLAIN over TLS.
Update the server software (or MailKit) to a version whose DIGEST-MD5 implementation conforms to RFC 2831.
If the server is behind a proxy/firewall, bypass or reconfigure it to see if it mangles the SASL exchange.
Disable DIGEST-MD5 on the server and use another auth mechanism.

Example fix

// before: mechanism selected as DIGEST-MD5 against a server that omits rspauth
var client = new SmtpClient ();
await client.Connect ("smtp.example.com", 587);
await client.Authenticate ("user", "pass");

// after: prefer a well-supported mechanism / catch and fall back
var client = new SmtpClient ();
await client.Connect ("smtp.example.com", 587, SecureSocketOptions.StartTls);
try {
	await client.Authenticate ("user", "pass");
} catch (SaslException ex) when (ex.ErrorCode == SaslErrorCode.InvalidChallenge) {
	// fall back or report non-conformant server
	throw;
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Verify the server advertises DIGEST-MD5 before choosing it
if (!client.AuthenticationMechanisms.Contains ("DIGEST-MD5"))
	throw new InvalidOperationException ("Server does not support DIGEST-MD5");

Try / catch

try {
	await client.Authenticate (new SaslMechanismDigestMd5 (user, pass));
} catch (SaslException ex) when (ex.ErrorCode == SaslErrorCode.InvalidChallenge) {
	// non-conformant server final response; fall back or report
	logger.LogWarning (ex, "DIGEST-MD5 failed with InvalidChallenge");
	await client.Authenticate (new SaslMechanismLogin (user, pass));
}

Prevention

Only select DIGEST-MD5 when the server advertises it in AuthenticationMechanisms.
Prefer PLAIN/LOGIN over TLS instead of DIGEST-MD5 with servers known to be non-conformant.
Log raw SASL challenges in a debug flag so malformed final responses are diagnosable.
Keep MailKit and server software up to date with RFC 2831-conformant versions.

When it happens

Trigger: Calling Challenge on SaslMechanismDigestMd5 with the final server token when the parsed key is not 'rspauth' (e.g. the server sends another directive like 'realm' or 'nonce' at the final-response stage, or a malformed/garbage token that still parses as key=value).

Common situations: Talking to a non-conformant or buggy SMTP/IMAP/POP3 server whose final DIGEST-MD5 step omits rspauth or sends extra directives; a proxy/MITM altering the final challenge; server software upgrades changing challenge format.

Related errors

AI-assisted analysis of jstedfast/MailKit@9d3859a785 (2026-09-15). Data as JSON: /api/errors/553b8bbeee6fab62. Report an issue: GitHub.

Appendix: source

Thrown at MailKit/Security/SaslMechanismDigestMd5.cs:169

				encoding = challenge.Charset != null ? Encoding.UTF8 : TextEncodings.Latin1;
				cnonce ??= GenerateEntropy (15);

				response = new DigestResponse (challenge, encoding, Uri.Scheme, Uri.DnsSafeHost, AuthorizationId, Credentials.UserName, Credentials.Password, cnonce);
				state = LoginState.Final;

				return response.Encode (encoding);
			case LoginState.Final:
				if (token == null || token.Length == 0)
					throw new SaslException (MechanismName, SaslErrorCode.MissingChallenge, "Server response did not contain any authentication data.");

				var text = encoding!.GetString (token, startIndex, length);
				string? key, value;

				if (!DigestChallenge.TryParseKeyValuePair (text, out key, out value))
					throw new SaslException (MechanismName, SaslErrorCode.IncompleteChallenge, "Server response contained incomplete authentication data.");

				if (!key.Equals ("rspauth", StringComparison.OrdinalIgnoreCase))
					throw new SaslException (MechanismName, SaslErrorCode.InvalidChallenge, "Server response contained invalid data.");

				var expected = response!.ComputeHash (encoding, Credentials.Password, false);
				if (value != expected)
					throw new SaslException (MechanismName, SaslErrorCode.IncorrectHash, "Server response did not contain the expected hash.");

				IsAuthenticated = true;
				break;
			}

			return null;
		}

		/// <summary>
		/// Reset the state of the SASL mechanism.
		/// </summary>
		/// <remarks>
		/// Resets the state of the SASL mechanism.
		/// </remarks>

View on GitHub (pinned to 9d3859a785)