Microsoft Is Finally Making Custom AI Chips — But They're Not for Displacing Nvidia

Microsoft's custom AI chip could be utilized for training large language models, potentially reducing the need for expensive dependence on $NVIDIA(NVDA.US)$. Azure Maia AI chip and Arm-powered Azure Cobalt CPU are arriving in 2024, on the back of a surge in demand this year for Nvidia's H100 GPUs that are widely used to train and operate generative image tools and large language models. There's such high demand for these GPUs that some have even fetched more than $40,000 on eBay.

Maia 100 is currently being tested on GPT 3.5 Turbo, the same model that powers ChatGPT, Bing AI workloads, and GitHub Copilot. Manufactured on a 5-nanometer TSMC process, Maia has 105 billion transistors — around 30 percent fewer than the 153 billion found on $Advanced Micro Devices(AMD.US)$'s MI300X AI GPU. "Maia supports our first implementation of the sub 8-bit data types, MX data types, in order to co-design hardware and software," says Borkar. "This helps us support faster model training and inference times."

That diversification of supply chains is important to Microsoft, particularly when Nvidia is the key supplier of AI server chips right now and companies have been racing to buy up these chips. Estimates have suggested OpenAI needed more than 30,000 of Nvidia's older A100 GPUs for the commercialization of ChatGPT, so Microsoft's own chips could help lower the cost of AI for its customers. Microsoft has also developed these chips for its own Azure cloud workloads, not to sell to others like Nvidia, AMD, $Intel(INTC.US)$, and $Qualcomm(QCOM.US)$ all do.