3 AM. You're staring at Thunderbird throwing "SSL handshake failed" while your server logs show nothing useful. Or maybe it's your transactional email script that worked fine for six months and suddenly started timing out on the SMTP connection. The TLS handshake is failing somewhere between your client and the mail server, and the error messages are cryptic enough to make you question your career choices.

I've debugged enough of these to know the pattern. It's almost always one of four things: wrong port for the TLS mode, expired certificate, cipher suite mismatch, or a firewall stripping STARTTLS. The hard part is figuring out which one. (I once spent two hours convinced it was a cipher problem before realizing I'd typo'd the port number. Not my finest moment.)

By the end of this, you'll have a step-by-step approach to diagnose and fix SMTP TLS handshake failures — plus the OpenSSL commands to prove what's actually happening on the wire.

Prerequisites

Before you start debugging:

Terminal access (macOS, Linux, or WSL on Windows)
OpenSSL installed (openssl version should return something — 1.1.1+ is ideal)
The SMTP server hostname and port you're trying to connect to
Your email client or script that's failing (we'll verify the fix there)
Basic understanding of email security terms helps, but isn't required

If you don't have OpenSSL, install it via your package manager: brew install openssl on macOS, apt install openssl on Debian/Ubuntu. On Windows, use WSL or download the installer from slproweb.com.

Step 1: Identify the Failure Type

The error message tells you where to start — if you know how to read it. Here are the common ones:

"SSL routines:ssl3_get_record:wrong version number" You're using STARTTLS on a port that expects implicit TLS, or vice versa.

"certificate verify failed" / "certificate has expired" The server's TLS certificate is expired, self-signed, or doesn't match the hostname.

"no shared cipher" / "handshake failure:ssl3_read_bytes" The server and client can't agree on a cipher suite. Usually means outdated TLS versions.

"Connection reset by peer" during handshake A firewall or proxy is killing the connection, or the server is rejecting your TLS version.

If you're not sure which error you have, move to Step 2 — OpenSSL will tell you exactly what's failing.

Step 2: Test the Connection with OpenSSL

OpenSSL's s_client command shows you the raw TLS handshake. It's the single most useful debugging tool for this.

For implicit TLS (port 465):

openssl s_client -connect mail.example.com:465

For STARTTLS (port 587):

openssl s_client -connect mail.example.com:587 -starttls smtp

The -starttls smtp flag tells OpenSSL to start with unencrypted SMTP, then upgrade via STARTTLS. Miss this flag on port 587 and you'll get garbage output — the server is waiting for an SMTP command, not a TLS ClientHello.

What to look for in the output:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = mail.example.com
verify return:1
---
Certificate chain
 0 s:CN = mail.example.com
   i:C = US, O = Let's Encrypt, CN = R3
...
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
...
    Verify return code: 0 (ok)
---

That's a healthy connection. Verify return code: 0 (ok) is what you want. TLSv1.3, modern cipher. You're good.

If you see Verify return code: 10 (certificate has expired) — that's your problem. Jump to Step 4.

If you see Verify return code: 18 (self-signed certificate) or 21 (unable to verify the first certificate) — the certificate chain is broken. The server's cert isn't trusted by your system's CA bundle.

If you see nothing at all (just hangs), the port is blocked by a firewall or the server isn't listening there. Try nc -zv mail.example.com 587 to check if the port is even open.

Step 3: Check for Port/Mode Mismatch

This is the most common mistake. I've made it dozens of times. Your email client says "SSL/TLS" but you're pointed at port 587, which wants STARTTLS. Or you've selected "STARTTLS" but you're on port 465, which is implicit-TLS-only.

The rule is simple:

Port 587: Use STARTTLS (connection starts unencrypted, upgrades)
Port 465: Use SSL/TLS / Implicit TLS (encrypted from byte one)
Port 25: Server-to-server relay, ISPs block this, don't use it from mail clients

Memorize that. It'll save you hours.

If your email client has a dropdown that says "None / STARTTLS / SSL/TLS" — select STARTTLS for 587, SSL/TLS for 465. Thunderbird labels these well; Apple Mail calls them "TLS/STARTTLS" and "TLS"; Outlook says "STARTTLS" or "SSL/TLS."

Here's a quick test to confirm which mode the server expects:

# Test implicit TLS (should work on 465)
timeout 5 openssl s_client -connect mail.example.com:465 2>&1 | head -5

# Test STARTTLS (should work on 587)
timeout 5 openssl s_client -connect mail.example.com:587 -starttls smtp 2>&1 | head -5

If the first one connects and the second hangs (or vice versa), you know what mode each port expects.

Step 4: Fix Certificate Problems

If OpenSSL showed a certificate error, you need to know what's wrong with the cert.

Check the certificate details:

echo | openssl s_client -connect mail.example.com:465 2>/dev/null | openssl x509 -noout -dates -subject

Output:

notBefore=Apr 15 00:00:00 2026 GMT
notAfter=Jul 14 23:59:59 2026 GMT
subject=CN = mail.example.com

If notAfter is in the past — expired cert. Renew it. Let's Encrypt certs expire every 90 days; if certbot's auto-renewal is broken, this happens.

If the CN (Common Name) or SAN doesn't match the hostname you're connecting to, you'll get a hostname mismatch error. The cert might be for mail.example.com but you're connecting to smtp.example.com. Check the SANs:

echo | openssl s_client -connect mail.example.com:465 2>/dev/null | openssl x509 -noout -ext subjectAltName

Fixing expired certs (Let's Encrypt):

On the mail server:

sudo certbot renew --force-renewal
sudo systemctl reload postfix dovecot  # or your mail server

Fixing self-signed certs:

If you're running a self-signed cert intentionally (internal mail server), you have two options:

Add the cert to your client's trusted CA store
Replace it with a real cert from Let's Encrypt (free, and avoids this headache forever)

For production mail servers, just get a real cert. Life's too short for certificate debugging. Honestly, I think self-signed certs on production SMTP are one of those "seemed clever at the time" decisions that always comes back to bite you. If you're setting up email from scratch on a custom domain, our custom domain email setup guide covers the DNS records you'll need alongside your TLS config.

Step 5: Fix Cipher Suite Mismatches

If OpenSSL shows "no shared cipher" or your client says "cipher mismatch," the server and client can't agree on an encryption algorithm.

Check what ciphers the server supports:

nmap --script ssl-enum-ciphers -p 465 mail.example.com

Or, if you don't have nmap:

openssl s_client -connect mail.example.com:465 -cipher 'ALL' 2>&1 | grep -i cipher

Modern clients expect TLS 1.2+ with ciphers like ECDHE-RSA-AES256-GCM-SHA384. If the server only offers older ciphers (RC4, DES, anything with SHA1), modern clients will refuse to connect.

Fixing cipher configuration (Postfix example):

In /etc/postfix/main.cf:

smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_mandatory_ciphers = medium
smtpd_tls_ciphers = medium

Then reload: sudo systemctl reload postfix

This disables TLS 1.0 and 1.1 (which modern clients reject anyway) and enables the "medium" cipher suite, which includes the modern ECDHE and AES-GCM ciphers.

If you're connecting TO a server that only supports old ciphers (legacy corporate mail server, for example), your options are limited. You can try forcing a specific older cipher in your client, but that's a security tradeoff. Ideally, tell the server admin to update their config.

The frustrating thing? Some organizations refuse to update because "it's been working for 10 years." Cool. Your cipher suite is older than some of your employees.

Step 6: Verify the Fix

Once you've made changes, re-run the OpenSSL test:

openssl s_client -connect mail.example.com:587 -starttls smtp

Look for Verify return code: 0 (ok) and confirm the protocol is TLSv1.2 or TLSv1.3.

Then test from your actual email client. Send a test email. If it goes through without errors, you're done.

For transactional email scripts, you can test SMTP auth without sending mail:

openssl s_client -connect mail.example.com:587 -starttls smtp -quiet
EHLO test
AUTH LOGIN
# (it'll ask for base64-encoded credentials)
QUIT

If AUTH LOGIN works, your TLS handshake is healthy.

Common Errors Reference

"SSL routines:ssl3_get_record:wrong version number"

error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Cause: STARTTLS on an implicit-TLS port (or vice versa).

Fix: Switch port 587 + STARTTLS, or port 465 + SSL/TLS. Don't mix them.

"certificate has expired"

Verify return code: 10 (certificate has expired)

Cause: Server cert is past its notAfter date.

Fix: Renew the cert on the server. certbot renew --force-renewal for Let's Encrypt.

"unable to verify the first certificate"

Verify return code: 21 (unable to verify the first certificate)

Cause: Missing intermediate certificate in the chain.

Fix: Server admin needs to include the full certificate chain, not just the leaf cert. For Let's Encrypt, use fullchain.pem, not cert.pem.

"no shared cipher"

error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small

Cause: Server offers weak ciphers that modern clients reject.

Fix: Update server TLS config to support TLS 1.2+ and modern cipher suites.

Connection hangs / times out

Cause: Firewall blocking the port, or server not listening.

Fix: Check nc -zv host port to verify the port is open. If it's closed, check server config or firewall rules. Corporate firewalls sometimes block 465/587 outbound. I've wasted entire afternoons on this one before thinking to check with IT. If you're running your own Postfix server, see our comparison of managed email vs self-hosted Postfix for when it's worth the operational burden.

Next Steps

If you're setting up email infrastructure from scratch, MTA-STS ensures senders always use TLS — see our MTA-STS setup guide. For a deeper understanding of when to use each port, check Email Ports Explained: 25, 465, 587, 993.

On the authentication side, TLS encrypts the connection, but DMARC prevents spoofing. You need both — don't let anyone tell you TLS alone is enough. And if you're running transactional email from your app, SMTP vs API breaks down when each approach makes sense.

For multi-domain setups where you're managing TLS across dozens of client domains, the overhead adds up. JustEmails handles TLS configuration automatically — valid certs, modern ciphers, both STARTTLS and implicit TLS — so you're not debugging handshake failures at 3 AM. $49/year for unlimited domains and mailboxes, if you want to skip the operational headache. For click fraud on ads that get those email signups, ClickzProtect is our sibling tool — same "catch the problem before it costs you" philosophy. And if you're automating email infrastructure with AI agents, DevOS can help orchestrate deployments across your stack.

Frequently Asked Questions

What causes "SSL routines:ssl3_get_record:wrong version number" errors?

This error means you're trying to use STARTTLS on a port that expects implicit TLS (like 465), or vice versa. Port 587 expects STARTTLS — the connection starts unencrypted and upgrades. Port 465 expects implicit TLS — encrypted from the first byte. Match your client's security setting to the port: STARTTLS for 587, SSL/TLS for 465.

How do I check if my mail server's TLS certificate is valid?

Run: openssl s_client -connect mail.yourdomain.com:465 or openssl s_client -connect mail.yourdomain.com:587 -starttls smtp. Look for "Verify return code: 0 (ok)" at the bottom. If you see error 10 (certificate has expired) or error 18 (self-signed certificate), that's your problem.

Why does my email client say "cipher mismatch" or "no shared cipher"?

The server and client can't agree on an encryption algorithm. This usually means the server only supports older ciphers (TLS 1.0/1.1) that modern clients reject, or the server requires ciphers the client doesn't support. Update the server's TLS configuration to support TLS 1.2+ with modern cipher suites like ECDHE-RSA-AES256-GCM-SHA384.

Does JustEmails handle TLS configuration automatically?

Yes. JustEmails servers run TLS 1.2 and 1.3 with modern cipher suites, valid certificates, and both STARTTLS (port 587) and implicit TLS (port 465). If you're getting TLS handshake errors connecting to JustEmails, the issue is almost always on the client side — wrong port/security combination or an outdated email app.

Try JustEmails

Unlimited custom domain email hosting for $49/year flat — unlimited domains, unlimited email accounts, 10 GB storage (expandable), full IMAP/SMTP. Built for agencies, freelancers, and anyone managing email across more than one domain.

Start your 7-day free trial → · How it compares

Fix TLS Handshake Failures on SMTP Connections