>samit_hota
Back to security news

Security News · SN-2026-598

MEDIUMOPEN

OpenSSL 4.1.0 Ships DTLS 1.3 Alongside Four Unpatched Listener Bugs

Affected: OpenSSL 4.1.0

Samit Hota·
#news#vulnerability-disclosure#openssl

The OpenSSL project has released OpenSSL 4.1.0, introducing long-awaited support for Datagram Transport Layer Security version 1.3 (DTLS 1.3) alongside performance enhancements for post-quantum cryptographic algorithms. However, the release arrives with a significant caveat for network engineering teams: four documented bugs in the new DTLS code remain unpatched at launch. Two of these flaws misreport connection state during socket operations, creating conditions that can leave server threads hanging indefinitely or cause applications to drop healthy connections.

DTLS adapts Transport Layer Security (TLS) for datagram-based protocols like UDP, which are commonly used in low-latency environments such as real-time voice, video streaming, VPN tunnels, and gaming. While DTLS 1.3 (standardized in RFC 9147) brings the reduced latency and modern key exchange mechanics of TLS 1.3 to datagram transport, implementing stateful cryptography over an inherently unreliable transport layer requires meticulous socket management. The state handling issues present in OpenSSL 4.1.0 directly impact how non-blocking network event loops process network errors and session tickets.

Logic Flaws in Connection and Error Handling

The primary issues in OpenSSL 4.1.0 center on how the cryptographic library translates underlying socket errors to high-level application status codes, as well as how it handles post-handshake messaging under packet loss conditions:

  • Listener Socket Hangs (SSL_ERROR_WANT_READ): When a permanent receive failure occurs on a DTLS listener, OpenSSL can return SSL_ERROR_WANT_READ. In non-blocking network architectures (such as epoll or kqueue event loops), this code instructs the application to pause and await further incoming data. Because subsequent polling or call attempts fail to surface the underlying fatal socket error, worker threads can remain permanently blocked on dead connections.
  • Premature Termination (SSL_ERROR_SYSCALL): On accepted non-blocking connections, a temporary, retryable write failure can inadvertently surface as SSL_ERROR_SYSCALL. Applications that interpret SSL_ERROR_SYSCALL as a fatal transport termination will immediately tear down active connections that are otherwise functional.
  • Handshake Abort on Packet Loss: If a DTLS 1.3 client acknowledges part or all of a server’s initial handshake flight before transmitting its own response—a situation frequently triggered by UDP packet loss or reordering—the server may incorrectly drop the negotiation and issue an unexpected_message alert.
  • Retransmission Buffer Erasure During KeyUpdate: When processing a KeyUpdate message (used to rotate encryption keys mid-session), OpenSSL purges all pending post-handshake records from its retransmission buffer. If a NewSessionTicket record sent by the server was lost in transit prior to the KeyUpdate, OpenSSL will fail to retransmit it, silently breaking session resumption capabilities for affected clients.

The OpenSSL team currently has a fix in progress for the KeyUpdate retransmission bug, but has not provided a resolution status for the three remaining state-handling flaws.

Cryptographic Upgrades and Build Target Deprecations

Beyond DTLS 1.3 support, OpenSSL 4.1.0 incorporates notable updates to post-quantum cryptography (PQC) and hardware acceleration. Key-encapsulation (ML-KEM) and digital signature (ML-DSA) algorithms receive optimized Number Theoretic Transform (NTT) math operations on ppc64le architectures. For x86_64 deployments, ML-DSA gains AVX-512 SHAKE x4 implementation code, while symmetric AES-CBC decryption receives AVX-512 and VAES instruction optimizations.

The release also integrates RFC 8701 GREASE (Generates Random Extensions And Sustains Extensibility) mechanism support to prevent protocol ossification by injecting reserved, unknown extension values into handshakes. Additional features include an IKEv2 key derivation function, initial target support for the Elbrus2000 architecture, and the inclusion of the dedicated ossl-guide-dtlsv13(7) manual page.

Administrators and developers should also note several build system cleanups and removals:

  • Ineffective Configure Flags Removed: The no-ecdsa and no-ecdh options have been removed from the Configure script. These flags previously failed to exclude their respective implementation code from the final binaries. Teams wishing to completely remove elliptic curve functionality must now use the no-ec configuration flag.
  • Dropped Architecture Targets: Support for Windows-on-Itanium and Windows CE build targets has been officially discontinued.
  • Dependency Changes: The tsget time-stamping utility now depends on the modern Net::Curl::Easy Perl module, replacing the abandoned WWW::Curl::Easy.

Practical Guidance for Developers

Organizations evaluating OpenSSL 4.1.0 should treat the new DTLS features with caution until follow-up maintenance releases are issued:

  • Delay DTLS Listener API Adoption: Applications reliant on high-volume UDP traffic or OpenSSL’s expanded SSL listener API should delay adopting DTLS 1.3 in production until the state-misreporting bugs are resolved upstream.
  • Audit Error Handling Logic: If deploying OpenSSL 4.1.0 in environments using DTLS listeners, do not take return codes like SSL_ERROR_WANT_READ or SSL_ERROR_SYSCALL at face value. Custom logic should double-check underlying socket statuses before dropping sockets or entering wait states.
  • Update Tooling Dependencies: Systems using tsget for cryptographic timestamping must ensure Net::Curl::Easy is installed in the local Perl environment prior to upgrading to prevent broken scripts.

Found something similar in your stack?

Let's find out before it becomes an incident.

Book an advisory call