The AI industry has a hardware problem that money alone cannot solve. Nvidia controls roughly 80 percent of the market for chips that train and run large language models, and the company has wielded that dominance to extract margins that would make a luxury-goods house blush. Customers pay, grumble, and pay again—because until now, the software stack has been as locked-in as the silicon.
ZML, a Paris-based startup that has operated in relative obscurity, is betting that inference—the computationally intensive process of actually running a trained model—can be liberated from any single vendor's hardware. This week the company released a free, open-source product designed to optimize inference workloads across a menagerie of chips: Nvidia GPUs, yes, but also AMD's accelerators, Intel's Gaudi line, and even custom ASICs from cloud hyperscalers.
The inference bottleneck
Training a frontier model remains eye-wateringly expensive, but it is a one-time cost amortized across millions of users. Inference is the recurring bill—every query to ChatGPT, every image generated by Midjourney, every code suggestion from Copilot. As AI applications move from novelty to infrastructure, inference spend is projected to eclipse training spend within two years. Whoever controls the efficiency of that workload controls the economics of the entire stack.
ZML's optimizer sits between the model and the metal, translating high-level instructions into hardware-specific code paths. The pitch is straightforward: write once, run anywhere, and let the software figure out which chip in your heterogeneous fleet is best suited to each task. For enterprises juggling on-premise servers, multiple cloud providers, and looming export restrictions on certain Nvidia parts, the appeal is obvious.
Why open source, why free
Giving away the core product is a calculated bet on ecosystem gravity. ZML plans to monetize through enterprise support, managed services, and premium features—the playbook that turned Red Hat into a $34 billion acquisition target. Open-sourcing the optimizer also invites contributions from hardware vendors eager to see their chips better supported. AMD and Intel have every incentive to help ZML succeed; so do sovereign AI initiatives in Europe and Asia that view Nvidia dependency as a strategic vulnerability.
The timing is not accidental. Export controls have made certain high-end Nvidia chips unavailable in China and under scrutiny elsewhere. Companies building AI infrastructure want optionality, and ZML is offering a credible path to it.
Our take
Nvidia's moat has always been as much about CUDA—its proprietary software ecosystem—as about transistor density. ZML is attempting to drain that moat one inference call at a time. Whether a young French company can outmaneuver a $3 trillion incumbent remains to be seen, but the mere existence of a viable alternative shifts negotiating leverage across the industry. For anyone tired of writing checks to Santa Clara, that alone is worth watching.
