Authors: Harshad Sane, Andrew Halaney

Imagine this — you click play on Netflix on a Friday night and behind the scenes hundreds of containers spring to action in a few seconds to answer your call. At Netflix, scaling containers efficiently is critical to delivering a seamless streaming experience to millions of members worldwide. To keep up with responsiveness at this scale, we modernized our container runtime, only to hit a surprising bottleneck: the CPU architecture itself.

Let us walk you through the story of how we diagnosed the problem and what we learned about scaling containers at the hardware level.

The Problem

When application demand requires that we scale up our servers, we get a new instance from AWS. To use this new capacity efficiently, pods are assigned to the node until its resources are considered fully allocated. A node can go from no applications running to being maxed out within moments of being ready to receive these applications.

As we migrated more and more from our old container platform to our new container platform, we started seeing some concerning trends. Some nodes were stalling for long periods of time, with a simple health check timing out after 30 seconds. An initial investigation showed that the mount table length was increasing dramatically in these situations, and reading it alone could take upwards of 30 seconds. Looking at systemd’s stack it was clear that it was busy processing these mount events as well and could lead to complete system lockup. Kubelet also timed out frequently talking to containerd in this period. Examining the mount table made it clear that these mounts were related to container creation.

The affected nodes were almost all r5.metal instances, and were starting applications whose container image contained many layers (50+).

Challenge

Mount Lock Contention

The flamegraph in Figure 1 clearly shows where containerd spent its time. Almost all of the time is spent trying to grab a kernel-level lock as part of the various mount-related activities when assembling the container’s root filesystem!

Looking closer, containerd executes the following calls for each layer if using user namespaces:

1. open_tree() to get a reference to the layer / directory
2. mount_setattr() to set the idmap to match the container’s user range, shifting the ownership so this container can access the files
3. move_mount() to create a bind mount on the host with this new idmap applied

These bind mounts are owned by the container’s user range and are then used as the lowerdirs to create the overlayfs-based rootfs for the container. Once the overlayfs rootfs is mounted, the bind mounts are then unmounted since they are not necessary to keep around once the overlayfs is constructed.

If a node is starting many containers at once, every CPU ends up busy trying to execute these mounts and umounts. The kernel VFS has various global locks related to the mount table, and each of these mounts requires taking that lock as we can see in the top of the flamegraph. Any system trying to quickly set up many containers is prone to this, and this is a function of the number of layers in the container image.

For example, assume a node is starting 100 containers, each with 50 layers in its image. Each container will need 50 bind mounts to do the idmap for each layer. The container’s overlayfs mount will be created using those bind mounts as the lower directories, and then all 50 bind mounts can be cleaned up via umount. Containerd actually goes through this process twice, once to determine some user information in the image and once to create the actual rootfs. This means the total number of mount operations on the start up path for our 100 containers is 100 * 2 * (1 + 50 + 50) = 20200 mounts, all of which require grabbing various global mount related locks!

Diagnosis

What’s Different In The New Runtime?

As alluded to in the introduction, Netflix has been undergoing a modernization of its container runtime. In the past a virtual kubelet + docker solution was used, whereas now a kubelet + containerd solution is being used. Both the old runtime and the new runtime used user namespaces, so what’s the difference here?

1. Old Runtime:
   All containers shared a single host user range. UIDs in image layers were shifted at untar time, so file permissions matched when containers accessed files. This worked because all containers used the same host user.
2. New Runtime:
   Each container gets a unique host user range, improving security — if a container escapes, it can only affect its own files. To avoid the costly process of untarring and shifting UIDs for every container, the new runtime uses the kernel’s idmap feature. This allows efficient UID mapping per container without copying or changing file ownership, which is why containerd performs many mounts.

Figure 2 below is a simplified example of how this idmap feature looks like:

Why Does Instance Type Matter?

As noted earlier, the issue was predominantly occurring on r5.metal instances. Once we identified the root issue we could easily reproduce by creating a container image with many layers and sending hundreds of workloads using the image to a test node.

To better understand why this bottleneck was more profound on some instances compared to others, we benchmarked container launches on different AWS instance types:

• r5.metal (5th gen Intel, dual-socket, multiple NUMA domains)
• m7i.metal-24xl (7th gen Intel, single-socket, single NUMA domain)
• m7a.24xlarge (7th gen AMD, single-socket, single NUMA domain)

Baseline Results

Figure 3 shows the baseline results from scaling containers on each instance type

• At low concurrency (≤ ~20 containers), all platforms performed similarly
• As concurrency increased, r5.metal began to fail around 100 containers
• 7th generation AWS instances maintained lower launch times and higher success rates as concurrency grew
• m7a instances showed the most consistent scaling behavior with the lowest failure rates even at high concurrency

Deep Dive

Using perf record and custom microbenchmarks, we can see the hottest code path was in the Linux kernel’s Virtual Filesystem (VFS) path lookup code — specifically, a tight spin loop waiting on a sequence lock in path_init(). The CPU spent most of its time executing the pause instruction, indicating many threads were spinning, waiting for the global lock, as shown in the disassembly snippet below

path_init():
…
mov mount_lock,%eax
test $0x1,%al
je 7c
pause
…

Using Intel’s Topdown Microarchitecture Analysis (TMA), we observed:

• 95.5% of pipeline slots were stalled on contested accesses (tma_contested_accesses).
• 57% of slots were due to false sharing (multiple cores accessing the same cache line).
• Cache line bouncing and lock contention were the primary culprits.

Given a high amount of time being spent in contested accesses, the natural thinking from a perspective of hardware variations led to investigation of NUMA and Hyperthreading impact coming from the architecture to this subset

NUMA Effects

Non-Uniform Memory Access (NUMA) is a system design where each processor has its own local memory for faster access but relies on an interconnect to access the memory attached to a remote processor. Introduced in the 1990s to improve scalability in multiprocessor systems, NUMA boosts performance but also introduces higher latency when a CPU needs to access memory attached to another processor. Figure 4 is a simple image describing local vs remote access patterns of a NUMA architecture

AWS instances come in a variety of shapes and sizes. To obtain the largest core count, we tested the 2-socket 5th generation metal instances (r5.metal), on which containers were orchestrated by the titus agent. Modern dual-socket architectures implement NUMA design, leading to faster local but higher remote access latencies. Although container orchestration can maintain locality, global locks can easily run into high latency effects due to remote synchronization. In order to test the impact of NUMA, we tested an AWS 48xl sized instance with 2 NUMA nodes or sockets versus an AWS 24xl sized instance, which represents a single NUMA node or socket. As seen from Figure 5, the extra hop introduces high latencies and hence failures very quickly.

Hyperthreading Effects

• Hyperthreading (HT): Disabling HT on m7i.metal-24xl (Intel) improved container launch latencies by 20–30% as seen in Figure 6, since hyperthreads compete for shared execution resources, worsening the lock contention. When hyperthreading is enabled, each physical CPU core is split into two logical CPUs (hyperthreads) that share most of the core’s execution resources, such as caches, execution units, and memory bandwidth. While this can improve throughput for workloads that are not fully utilizing the core, it introduces significant challenges for workloads that rely heavily on global locks. By disabling hyperthreading, each thread runs on its own physical core, eliminating this competition for shared resources between hyperthreads. As a result, threads can acquire and release global locks more quickly, reducing overall contention and improving latency for operations that generally share underlying resources.

Why Does Hardware Architecture Matter?

Centralized Cache Architectures

Some modern server CPUs use a mesh-style interconnect to link cores and cache slices, with each intersection managing cache coherence for a subset of memory addresses. In these designs, all communication passes through a central queueing structure, which can only handle one request for a given address at a time. When a global lock (like the mount lock) is under heavy contention, all atomic operations targeting that lock are funneled through this single queue, causing requests to pile up and resulting in memory stalls and latency spikes.

In some well-known mesh-based architectures as shown in Figure 7 below, this central queue is called the “Table of Requests” (TOR), and it can become a surprising bottleneck when many threads are fighting for the same lock. If you’ve ever wondered why certain CPUs seem to “pause for breath” under heavy contention, this is often the culprit.

Distributed Cache Architectures

Some modern server CPUs use a distributed, chiplet-based architecture (Figure 8), where multiple core complexes, each with their own local last-level cache — are connected via a high-speed interconnect fabric. In these designs, cache coherence is managed within each core complex, and traffic between complexes is handled by a scalable control fabric. Unlike mesh-based architectures with centralized queueing structures, this distributed approach spreads contention across multiple domains, making severe stalls from global lock contention less likely. For those interested in the technical details, public documentation from major CPU vendors provides deeper insight into these distributed cache and chiplet designs.

Here is a comparison of the same workload run on m7i (centralized cache architecture) vs m7a (distributed cache architecture). Note that, in order to make it closely comparable, Hyperthreading (HT) was disabled on m7i, given previous regression seen in Figure 6, and experiments were run using same core counts. The result clearly shows a fairly consistent difference in performance of approximately 20% as shown in Figure 9

Microbenchmark Results

To prove the above theory related to NUMA, HT and micro-architecture, we developed a small microbenchmark which basically invokes a given number of threads that then spins on a globally contended lock. Running the benchmark at increasing thread counts reveals the latency characteristics of the system under different scenarios. For example, Figure 10 below is the microbenchmark results with NUMA, HT and different microarchitectures.

Results from this custom synthetic benchmark (pause_bench) confirmed:

• On r5.metal, eliminating NUMA by only using a single socket significantly drops latency at high thread counts
• On m7i.metal-24xl, disabling hyperthreading further improves scaling
• On m7a.24xlarge, performance scales the best, demonstrating that a distributed cache architecture handles cache-line contention in this case of global locks more gracefully.

Improving Software Architecture

While understanding the impacts of the hardware architecture is important for assessing possible mitigations, the root cause here is contention over a global lock. Working with containerd upstream we came to two possible solutions:

1. Use the newer kernel mount API’s fsconfig() lowerdir+ support to supply the idmap’ed lowerdirs as fd’s instead of filesystem paths. This avoids the move_mount() syscall mentioned prior which requires global locks to mount each layer to the mount table
2. Map the common parent directory of all the layers. This makes the number of mount operations go from O(n) to O(1) per container, where n is the number of layers in the image

Since using the newer API requires using a new kernel, we opted to make the latter change to benefit more of the community. With that in place, no longer do we see containerd’s flamegraph being dominated by mount-related operations. In fact, as seen in Figure 11 below we had to highlight them in purple below to see them at all!

Conclusion

Our journey migrating to a modern kubelet + containerd runtime at Netflix revealed just how deeply intertwined software and hardware architecture can be when operating at scale. While kubelet/containerd’s usage of unique container users brought significant security gains, it also surfaced new bottlenecks rooted in kernel and CPU architecture — particularly when launching hundreds of many layered container images in parallel. Our investigation highlighted that not all hardware is created equal for this workload: centralized cache management amplified cache contention while distributed cache design smoothly scaled under load.

Ultimately, the best solution combined hardware awareness with software improvements. For an immediate mitigation we chose to route these workloads to CPU architectures that scaled better under these conditions. By changing the software design to minimize per-layer mount operations, we eliminated the global lock as a launch-time bottleneck — unlocking faster, more reliable scaling regardless of the underlying CPU architecture. This experience underscores the importance of holistic performance engineering: understanding and optimizing both the software stack and the hardware it runs on is key to delivering seamless user experiences at Netflix scale.

We trust these insights will assist others in navigating the evolving container ecosystem, transforming potential challenges into opportunities for building robust, high-performance platforms.

Special thanks to the Titus and Performance Engineering teams at Netflix.

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A few hours before Donald Trump gave his State of the Union address, Republican sources told the PBS correspondent Lisa Desjardins that the speech would break records. The president would speak for more than two hours, she reported on X, and one reliable source claimed he might ramble on for 180 minutes.

The post went viral. At about the same time, the market started to move on Kalshi, an online platform where people can invest money in the outcome of a given news event. (Don’t call it gambling.) Forecasts on “How long will Trump speak for at the State of the Union?” shot up by 10 minutes after Desjardins posted: Armed with what they perceived as insider information, users thought they could make a buck by accurately “predicting” the outcome of his speech.

But others speculated in a different direction. “They’re leaking a bunch of stuff about a super long speech and he’ll go about 2 minutes short of the supposed mark and everyone in the white house will make $200k on it,” one Bluesky user, ‪@danvogfan, posted a few hours after Desjardin’s post went viral. In other words, maybe the sources really did have good information—but they were throwing others off track to manipulate the market and profit for themselves.

Prediction markets such as Kalshi and Polymarket have ushered in a moment when anyone with access to exclusive information related to a major news event can do this, even as the platforms themselves prohibit market manipulation. Trump ultimately didn’t speak for as long as the sources had said: He ended after an hour and 47 minutes. Anyone who had bet according to the information that Desjardins had reported would have lost money. “We live in such a profound dystopia,” another popular Bluesky user wrote above @danvogfan’s post after the fact.

[Read: America is slow-walking into a Polymarket disaster]

We can’t say definitively that any insider trading has actually happened, though other suspicious incidents have occurred. In early January, one Polymarket user bet more than $30,000 on Venezuelan President Nicolás Maduro being ousted just hours before he was captured by the U.S. military. (The bet paid out $400,000 and led Representative Ritchie Torres to introduce a bill that would ban federal workers from using prediction markets.) Last month, Israeli authorities charged two people on suspicion of using classified information to bet on military operations on Polymarket. And this past weekend, an anonymous trader who goes by the name Magamyman made more than $550,000 on Polymarket by betting on the timing of U.S. and Israeli strikes on Iran and the fate of its supreme leader.

Welcome to the democratization of insider trading, brought to us by platforms that let people wager on election outcomes, sports, and “Taylor Swift pregnant before marriage?” The prediction markets frame bets as tradable “shares” that rise and fall like stocks, financializing every current event and piece of online ephemera and generating a pervasive hum of paranoia: The world as a hedge fund, where everything can be a derivative. Greed is good.

Prediction markets claim to harness the wisdom of crowds to provide reliable public data: Because people are putting real money behind their opinions, they are expressing what they actually believe is most likely to happen, which, according to the reasoning of these platforms, means that events will unfold accordingly. Many news organizations, and Substack, now have partnerships with prediction markets—the subtext being that they provide some kind of news-gathering function. Some users who distrust mainstream media turn to the markets in place of traditional journalism.

But in reality, prediction markets produce the opposite of accurate, unbiased information. They encourage anyone with an informational edge to use their knowledge for personal financial gain. In this way, prediction markets are the perfect technology for a low-trust society, simultaneously exploiting and reifying an environment in which believing the motives behind any person or action becomes harder.

Polymarket did not respond to my request for comment. In response to concerns about trading on the outcome of the Iran strikes, the company said that it aims to “create accurate, unbiased forecasts for the most important events to society.” Jack Such, a spokesperson for Kalshi, told me, “War markets put Americans at risk and have absolutely no place in prediction markets.” Unlike Polymarket, which technically operates outside of the United States, Kalshi is subject to U.S. government regulation. It does not allow bets on wars or assassinations, though it did host a vaguely worded market pertaining to whether Iranian Supreme Leader Ayatollah Ali Khamenei would be “out.” After he was killed in the conflict, Kalshi did not resolve the market to “Yes,” enraging some users.

[Read: Your phone is a slot machine]

“We share the concerns about war markets, death markets, and insider trading. We don’t allow any of these on Kalshi,” Such also said. “Having a market on whether or not the U.S. will enter a civil war is insane. Not all prediction markets are the same.” (Polymarket has indeed hosted such markets.)

But these differences, though relevant in specific instances, do not have much bearing on the larger problems that these platforms contribute to. Prediction markets are selling a philosophy: Tarek Mansour, Kalshi’s CEO, said that the company is “replacing debate, subjectivity, and talk with markets, accuracy, and truth,” and Polymarket’s CEO, Shayne Coplan, said his company is “the most accurate thing we have as mankind right now.”

On X, both Polymarket and Kalshi have their own accounts that act as news feeds, where they post engagement-baiting and occasionally misleading headlines and speculate about world events that, conveniently, one can bet on via their platforms. On Tuesday morning, Polymarket’s X account looked an awful lot like a news wire, posting, “BREAKING: Ken Paxton projected to win today’s Texas Republican Senate Primary” and putting his odds at 83 percent. Paxton ended up with about 40 percent of the vote, slightly less than his opponent John Cornyn, who was polling at 18 percent on Polymarket at the time of the post; the two will compete in a runoff election in May.

The markets also encourage a kind of meta-game. People are betting on outcomes, but they are also hedging with side bets. For example, this winter, the Polymarket entry titled “Will Jesus Christ return before 2027?” climbed from 1.8 percent betting yes to roughly 4 percent betting yes in one month. The bizarre spike made the rounds online before a perceptive X user noted that the real reason for the change was that Polymarket traders had created a secondary market to bet on whether the odds of Christ returning would climb above 5 percent. Those traders were then manipulating the original “Will Jesus Christ return before 2027?” market to try to make money on their secondary bets.

[Read: You’ve never seen Super Bowl betting like this before]

This means that the markets don’t always reflect what people think will happen as much as they reflect what people think other people think will happen. This certainly gives the lie to the promise of accurate, truthful information from these platforms, as do the suspected incidents of insider training. When a market is manipulated by people with exclusive information, it does not provide clear, actionable intelligence to everyone else. That’s because many of these markets come and go quickly. Somebody who was up at 2 a.m. and happened to be paying attention to Polymarket’s Iran-air-strike market may have been able to pick up on Magamyman’s big bet and gain a subtle informational edge, but it is absurd to compare this subtle signal to credible reporting or intelligence. Kalshi, at least, seems to realize this; Such told me that the platform “bans insider trading not only because it’s unfair, but also because it erodes trust.” (Insider trading is also illegal, a point that a spokesperson for the White House repeatedly pointed out to me, without addressing my questions about whether the administration has its own rules forbidding government workers from participating in prediction markets.)

So they’re specious forecasting tools. Yet the prediction markets are bad for another, much more obvious, corrosive reason: Beneath the veneer of forecasting, the platforms are funneling gamblers to markets to bet on human suffering and acts of war. The top market on Polymarket’s homepage as I wrote this sentence was “Will the Iranian regime fall by June 30?” More than $6.7 million has been wagered on it so far. Betting on geopolitics and military operations allows traders to profit off of death, and it transforms people, politics, death, trauma, everything into commodities. In the wake of the first strikes on Iran, Polymarket briefly allowed trades on when a nuclear weapon was likely to be detonated. Current events, no matter how heinous, become entertainment, a business plan, or both—what Jason Koebler of 404 Media recently dubbed a “depravity economy.”

As the depravity economy grows, it will break whatever trust we have left in one another: If prop bets spur athletes to play differently, this poses an existential threat to the integrity of live sports. If people believe that anonymous government insiders are profiting off of classified information, what reason is there to trust anything that the administration says? There’s a term called the liar’s dividend, which describes an information environment where mis- and disinformation such as deepfakes become so prevalent that anyone accused of doing something awful can simply use them to cast doubt on genuine evidence. Prediction markets offer an insider’s dividend, creating an environment where the prevalence of prediction markets and insider trading becomes great enough that everyone assumes a given decision was made to enrich those with an edge.

This is the central lie of prediction markets: They claim to get us closer to the truth but, in the end, they make us less certain about the world. But this erosion of trust is a feature, not a bug, for these platforms. A world where people are suspicious of every motive is a world where the cold logic of gambling feels more rational. A zero-trust society is one where the prediction markets’ dubious “wisdom of crowds” marketing seems extra appealing.

In this way, prediction markets are a system that justifies its own existence—a well-oiled machine chipping away at societal trust while offering a convenient solution to its own problem. The prediction markets have done what any savvy trader or firm might—they’ve hedged their bets. The house can’t lose.

If you are interested in historical big computers, you probably think of IBM, with maybe a little thought of Sperry Rand or, if you go smaller, HP, DEC, and companies like Data General. But you may not have heard of Tandem Computers unless you have dealt with systems where downtime was unacceptable. Printing bills or payroll checks can afford some downtime while you reboot or replace a bad board. But if your computer services ATM machines, cash registers, or a factory, that’s another type of operation altogether. That was where Tandem computers made their mark, and [Asianometry] recounts their history in a recent video that you can watch below.

When IBM was king, your best bet for having a computer running nonstop was to have more than one computer. But that’s pricey. Computers might have some redundancy, but it is difficult to avoid single points of failure. For example, if you have two computers with a single network connection and a single disk drive. Then failures in the network connection or the disk drive will take the system down.

The idea started with an HP engineer, but HP wasn’t interested. Tandem was founded on the idea of building a computer that would run continuously. In fact, the name was “the non-stop.” The idea was that smaller computer systems could be combined to equal the performance of a big computer, while any single constituent system failing would still allow the computer to function. It was simply slower. Even the bus that tied the computers together was redundant. Power supplies had batteries so the machines would keep working even through short power failures.

Not only does this guard against failures, but it also allows you to take a single computer down for repair or maintenance without stopping the system. You could also scale performance by simply adding more computers.

Citibank was the first customer, and the ATM industry widely adopted the system. The only issue was that Tandem programs required special handling to leverage the hardware redundancy. Competitors were able to eat market share by providing hardware-only solutions.

The changing computer landscape didn’t help Tandem, either. Tandem was formed at a time when computer hardware was expensive, so using a mostly software solution to a problem made sense. But over time, hardware became both more reliable and less expensive. Software, meanwhile, got more expensive. You can see where this is going.

The company flailed and eventually would try to reinvent itself as a software company. Before that transition could work or fail, Compaq bought the company in 1997. Compaq, of course, would also buy DEC, and then it was all bought up by HP — oddly enough, where the idea for Tandem all started.

There’s a lot of detail in the video, and if you fondly remember Tandem, you’ll enjoy all the photos and details on the company. If you need redundancy down at the component level, you’ll probably need voting.

Before the modern Internet existed, there were still plenty of ways of connecting with other computer users “online”, although many of them might seem completely foreign to those of us in the modern era. One of those systems was the Bulletin Board System, or BBS, which would have been a single computer, often in someone’s home, connected to a single phone line. People accessing the BBS would log in if the line wasn’t busy, leave messages, and quickly log out since the system could only support one user at a time. While perhaps a rose-tinted view, this was a more wholesome and less angsty time than the modern algorithm-driven Internet, and it turns out these systems are making a bit of a comeback as a result.

The video by [The Retro Shack] sets up a lot of this history for context, then, towards the end, uses a modern FPGA-based recreation called the Commodore 64 Ultimate to access a BBS called The Old Net, a modern recreation of what these 80s-era BBS systems were like. This involves using a modern networking card that allows the C64 to connect to Wi-Fi access points to get online instead of an old phone modem, and then using a terminal program called CCGMS to connect to the BBS itself. Once there, users can access mail, share files, and even play a few games.

While the video is a very basic illustration of how these BBS systems worked and how to access one, it is notable in that it’s part of a trend of rejecting more modern technology and systems in favor of older ones, where the users had more control. A retro machine like a C64 or Atari is not required either; modern operating systems can access these with the right terminal program, too. A more in-depth guide to the BBS can be found here for those looking to explore, and we’ve also seen other modern BBS systems recently.

Thanks to [Charlie] for the tip!