It’s not that we’ve discovered some magic bullet, but rather that JAX, or at least the open source version of it, is mostly optimized for small to medium-sized training runs on Google TPUs, whereas we need to massive training runs on Nvidia GPUs.
Pipeline parallelism is essential and crushes fully-sharded data parallelism at scale.
And C will compile to the most efficient binary short of assembly. Maybe we will do a little assembly too.
💡 Inside Track & Deep Insight
In a rare technical clarification, Elon Musk downplayed the notion of a 'magic bullet' in AI training, instead emphasizing that xAI's large-scale workloads on Nvidia GPUs demand raw performance from C (and even assembly) rather than high-level frameworks like JAX, which he argued is optimized for Google TPUs and medium-sized runs. This underscores a critical engineering divide: while JAX excels in research environments, production mega-training on Nvidia hardware often requires lower-level control to maximize throughput and minimize overhead.
Musk's specific mention of pipeline parallelism 'crushing' fully-sharded data parallelism at scale reveals xAI's architecture strategy. As models grow beyond trillions of parameters, communication bottlenecks in data parallelism become prohibitive; pipeline parallelism partitions model layers across devices, keeping each device's compute busy while reducing inter-device communication. This aligns with trends at leading labs like DeepMind and OpenAI, where custom parallelism strategies are increasingly critical.
The offhand remark about using assembly signals a no-holds-barred optimization culture, far from the typical AI startup reliance on existing libraries. For investors and competitors, it hints at xAI's potential cost or speed advantages if they succeed in extracting maximum efficiency from Nvidia's H100/B200 hardware. The comment also subtly distances xAI from the JAX ecosystem, suggesting a potential divergence in toolchains that could shape future AI hardware and software stack investments.
👇 Original Post on X
Yes.
It’s not that we’ve discovered some magic bullet, but rather that JAX, or at least the open source version of it, is mostly optimized for small to medium-sized training runs on Google TPUs, whereas we need to massive training runs on Nvidia GPUs.
Pipeline parallelism is…
— Elon Musk (@elonmusk) May 28, 2026