MuonClip Optimizer for LLM Training (used in Kimi 2)

People are calling this the most beautiful thing in AI training. It's also the secret behind a new trillion parameter open- source model named KI. It's a smooth loss curve.

Now, let me break down why this matters and what Moonshot AI did differently. Training a neural network is a lot like navigating a landscape. You're trying to reach the lowest point because the lowest point is the place of low of the least error.

So we typically track this with a loss curve which should steadily fall as trading progresses. What you don't want is curves that go back up, air increases or spikes. All of this is trading instability.

To mathematically reach that low point, we use optimizers. Optimizers typically follow the slope of the landscape, taking careful steps based on how steep the path is. That's what something like Adam does.

It's widely used. But now we have a different approach for Moonshot AI. Instead of following the slope, they reshape the landscape itself.

Their approach called muon clips works by rescaling the query and key matrices inside attention layers after every optimizer step. By rescaling it, they prevent dangerous cliffs that could destabilize large scales models. So instead of carefully like stepping around cliffs, clip just removes them.

And with this, Moonshot AI was able to train Kimi on 15. 5 trillion tokens without a single training spike. Now, this isn't a general purpose fix for all neural networks.

It's really built for large language models.