Researchers at the Okta Red Team have disclosed a subtle but potentially serious weakness in OpenSSL that allows a remote, unauthenticated attacker to gradually choke a server’s memory using a payload as small as 11 bytes. The issue, nicknamed “HollowByte,” doesn’t crash a server outright in a single request. Instead, it exploits how OpenSSL handles memory allocation during the TLS handshake to produce a slow-building denial-of-service condition that can eventually force an out-of-memory kill.

An Allocation That Trusts Too Much

Every TLS handshake begins with a ClientHello message, wrapped in a record whose 4-byte header declares how large the incoming message body will be. In older OpenSSL versions, the library takes that declared size at face value and allocates a receive buffer for it before a single byte of the actual payload has arrived.

That’s the crux of the problem. When a crafted 11-byte payload arrives, the TLS state machine reads the header and walks through an allocation chain that ends in a call to malloc(), sized entirely by the attacker’s claim rather than by any verified data. Since nothing validates the declared length at this stage, a single small packet can force the server to reserve up to 131 KB of memory. The connection’s worker thread then sits idle, waiting indefinitely for data that was never going to arrive.

Why This Is Worse Than a Simple Connection-Exhaustion Attack

Classic denial-of-service techniques like Slowloris work by holding connections open to exhaust available server threads. HollowByte adds a second, nastier layer on top: memory fragmentation rooted in how glibc manages freed memory. When an attacking connection eventually drops, OpenSSL frees the oversized buffer, but glibc doesn’t hand small-to-medium allocations back to the operating system right away. It holds onto them in case they’re reused.

An attacker can exploit that behavior by launching repeated waves of connections, each declaring a different, randomized claimed size. That pattern prevents the memory allocator from ever cleanly reusing the freed chunks, so the server’s resident memory footprint climbs steadily and doesn’t shrink back down, even after the attacking connections have long since disconnected. According to the researchers, the only real remedy at that point is killing and restarting the affected process.

Measured Impact

Okta’s testing against unpatched OpenSSL running behind NGINX illustrated just how effective the technique can be:

• On a server with 1 GB of RAM, the process was killed by the operating system’s out-of-memory handler after accumulating roughly 547 MB of frozen, fragmented memory.
• On a server with 16 GB of RAM, the attack locked up around 25 percent of total system memory while staying well under typical connection-count limits, meaning standard connection-throttling defenses failed to catch or stop it.

Because OpenSSL sits underneath an enormous share of internet infrastructure, the practical blast radius is wide. It touches web servers like Apache and NGINX, language runtimes including Node.js, Python, Ruby, and PHP, and databases such as MySQL and PostgreSQL, any of which could be exposed if they bundle a vulnerable OpenSSL build.

A Quiet Fix, Not a Loud One

OpenSSL resolved the underlying issue by switching to incremental buffer growth: rather than trusting the declared header size outright, the library now expands its buffer only as bytes actually arrive on the wire, so an empty or exaggerated claim costs the server nothing. The change was merged through three pull requests and folded into OpenSSL v4.0.1, with silent backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21.

Notably, OpenSSL classified this as a hardening improvement rather than issuing a formal CVE, a pattern that has shown up in other recent OpenSSL disclosures for denial-of-service-class issues. That decision has a real consequence for defenders: because HollowByte didn’t receive a CVE, many organizations’ standard vulnerability scanning and patch-tracking processes may never flag it.

What Defenders Should Do

• Upgrade to OpenSSL 4.0.1, or the relevant backported release: 3.6.3, 3.5.7, 3.4.6, or 3.0.21.
• Audit embedded OpenSSL copies bundled inside language runtimes and applications, since patching the operating system alone won’t fix statically linked or vendored versions.
• Monitor resident memory trends on TLS-terminating servers for signs of slow, unexplained growth consistent with a fragmentation-style attack.

Given OpenSSL’s near-universal footprint, security teams should treat this as a priority patch regardless of its lack of a formal CVE designation, and should specifically check patch-tracking workflows that rely solely on CVE feeds, since those workflows are exactly the ones likely to miss this fix.