8192-bit Keys Stall the Handshake; 1024-bit DH Gets Refused

One server has clients that hang for a while and then reconnect, and another refuses connections immediately with a message about key size. They look like different bugs and they are the same bug pointing in opposite directions. Handshake cryptography has a cost that scales with parameter size, and it also has a floor below which modern libraries will not go. Sizing the material is the checklist, and the first item is deciding which of the two faults you actually have.

Read the failure before regenerating anything

The timing tells you most of it. A handshake that consumes several seconds and then dies usually means the computation outlasted the peer's patience, which points at parameters that are too large for the hardware or the timeout. An immediate rejection that names a key or group size points the other way, at material the library will not accept at all.

Those two need opposite corrections, and this is where most attempts go wrong. Regenerating 8192-bit parameters on a client that already refused a small group makes the slow case worse while doing nothing for the fast one. Write down which symptom you observed before you touch a file.

Then check what the peer is being asked to do. A server configured for a large finite-field group forces every client to spend the same CPU time on every renegotiation, not just at connect. If the problem appears at a regular interval rather than at startup, that interval is a strong hint about which setting is expensive. Renegotiation repeats the whole computation, so a stall that recurs every hour is usually the same cost being paid again.

The slow case: material the hardware cannot afford

Diffie-Hellman and RSA work grows quickly with key size, and all of it happens before any tunnel traffic flows. On a small router or a low-power client, a large enough group can push the exchange past the other side's timeout, so the connection retries or simply appears to hang with no error at all.

The fix is to come down to a size that every peer supports and every device can compute comfortably. A 2048-bit RSA key and a 2048-bit group remain a broadly compatible baseline; going larger helps only when all peers agree and the CPU can pay for it on every renegotiation. Everything above that baseline is a deliberate trade, not an upgrade.

Size is not a score, and the largest number in a comparison table is not automatically the correct choice. Pick the smallest value your policy allows that all peers accept, then measure the handshake rather than assuming it improved. A smaller group that still times out tells you the size was never the whole story.

The fast case: a group the library refuses

Modern OpenSSL builds reject the smallest finite-field groups outright, and a raised security level pushes the floor higher still. Router firmware that bundles a decades-old parameter file is the usual source, and the symptom is a connection that fails at once with a message naming the group or key size. Nothing about that failure is intermittent, which is what makes it easy to recognize.

Replace the file rather than editing the configuration to skip the check. Generate a current group, validate it, then install it where the daemon actually reads it. Keep the old file until a client has connected successfully with the new one:

openssl dhparam -out /etc/openvpn/ffdhe2048.pem 2048
openssl dhparam -in /etc/openvpn/ffdhe2048.pem -check -noout
ls -l /etc/openvpn/ffdhe2048.pem

The check command validates the file and prints nothing useful when it passes; the listing confirms you replaced the file the service loads rather than a stale copy beside it. Point the configuration at that path, restart the daemon, and confirm that a client which previously failed now completes the handshake.

If the server also sets a security level or a certificate profile, read those before concluding the group was wrong. A profile that requires a stricter class can reject material that the default would have accepted, and removing that setting is a narrower change than disabling the check globally.

Choose a size for the peer and the hardware

Find out what the far end can handle before generating anything exotic. A device too old for a 2048-bit group is a device to replace, since pointing the server at a weaker group only converts a compatibility complaint into an outage. Durable fixes move toward newer software rather than older parameters. Every temporary exception in this area has a way of becoming the permanent configuration.

Where the server CPU is the limit, change the algorithm rather than shrinking below policy. An elliptic-curve key exchange such as X25519 is dramatically cheaper than a large finite-field group and is widely supported, which makes it the better answer to a performance complaint than a smaller RSA modulus.

Whatever you choose, test with one real connection and watch how long the handshake takes, not just whether it eventually connects. A connection that succeeds in twelve seconds has the same root cause as one that fails at fifteen.

Record the choice so a rebuild cannot undo it

Keep the parameter file under configuration management and write the size, the reason and the date beside it. A rebuilt server that quietly generates its own defaults is how the same fault returns six months later with nobody able to explain the original decision.

Record the same information next to the endpoint and the profile, so a review does not have to reverse-engineer it from a file size. When the next connection stalls, that note is the difference between a five-minute check and a fresh investigation. It also stops the next engineer from solving the slow case with a fix meant for the fast one.