So DeepSeek is the most discussed thing in AI now and the main reason is basically this. It shows that the DeepSeek startup spent about 2. 8 million GPU hours on training their language model.
That would take one GPU more than 300 years to do this alone. But of course, they used about 2,000 GPUs so it took them about 2 months to train their model. Now, is this number good?
Is it bad? Well, let's compare it with Llama, for example. The largest Lama 3 model was trained for almost 31 million GPU hours.
That's about 11 times more than DeepSeek! Now, on top of all of this, they managed to train their model on relatively inferior GPU chips. Because of the US export regulations, Nvidia cannot currently sell their most powerful GPUs to China.
So, to bypass this problem, Nvidia nerfed their chips and instead of H100s China now gets less powerful H800s. And these nerfed versions seem cheaper to run, apparently only $2 per hour, which gets us to this cost of 5. 6 million dollars.
That's orders of magnitude less than the training cost for models like GPT and Gemini. And the thing is that until now, only these big players like OpenAI, Google, and Meta could could be training such large models due to the costs, and the rest of us had to wait. And that might change because of DeepSeek now.
So we have much less training time, on cheaper GPUs, and the DeepSeek models are also dominating the benchmarks. The Chatbot Arena shows that they are currently on par even with the latest models from OpenAI and Google. So how is this all possible?
Well, there are actually a few caveats to these numbers and things to unpack like the actual cost of training, which is higher than this. . .
But before I get to that, let's first cover how DeepSeek models work and why they are so much faster. Because, there have been lots of things circulating in the media recently, and sometimes I feel like the authors haven't even read any of the papers. So, DeepSeek recently published two models- V3 and R1.
And let's start with V3, which is the third iteration of their main foundation language model, and it's basically another big language model like Gemini or Llama. One of the major changes that made V3 so much faster is that it uses something called "Mixture-of-Experts", or MoE. And the idea behind MoE is that instead of processing input by a single large model, we divide the model into smaller networks, called experts, where each expert specializes in something unique - like a different domain or a different type of syntax.
And when we process the input, we don't pass it to all experts. Instead, we use an additional smaller network, called router, which looks at the given input and maybe says something like: "Hmm, I think only the first and the last experts should process this input". And for the next input, it picks a different set of experts and so on.
MoE is applied on every layer of the transformer network, and it actually replaces all dense layers. And so if this works, then this is great, because first of all, we're processing the input in these specialized sub-networks that can do the task better. And second, and that's even more important here, we're saving a lot of computation.
Because all other experts are being completely ignored. MoE as a concept is nothing new. It dates back to the early '90s, and for example, it was recently used in an LLM called Mixtral.
But, the problem is that it can be hard to get it working well as many things can go wrong here. For example, how can you actually separate the knowledge and make sure that each expert is specialized? Also, there is a well-known problem called "routing collapse", where the router learns to use only one or very few experts every single time and then ignores the rest.
So, you need to somehow encourage the router to select different experts and spread the knowledge. But, deep seek implemented several improvements here! First, they increase the number of experts.
. . I mean, sure why not?
But then, they also added additional shared experts, which are always selected. And the idea behind this is that these shared experts will learn the general knowledge across different contexts, which will leave the other non-shared experts to learn more specialized knowledge. These ideas are nothing novel and have been implemented in NLP many times before, but it's quite impressive that they've got it working on such a large scale.
So, now, you will also know what the term "activated parameters" means. Maybe you've noticed that the DeepSeek website mentions their model having a total of 671 billion parameters but only 37 billion of them are activated. That simply means that for each input token, the network selects only a specific number of experts and the total number of parameters across the selected experts is always 37 billion.
However, that doesn't mean you can just load 37 billion parameters on your GPU. You still need to load the entire model if you want to use it because as you generate tokens, you'll be using different sets of experts each time and will eventually need all of them. For instance, the paper for Mixtral has a very nice example with MoE for code generation.
Here, they colored each expert output in a different color and you can see that it's not all just one color, right? The model is using all sorts of experts all the time, and here it seems that it's more aligned with syntax rather than domain, actually. So, when you see charts like this on NBC news, for example, I think they can be quite misleading.
It shows that DeepSeek uses fewer parameters than Llama and Qwen during "interactions", which is a very ambiguous word. When you interact with a model, you will need all of them. And some charts in papers can be a bit confusing too.
Like this one from the DeepSeek paper on mixture of experts. They're showing that their model uses less than three billion parameters, and it is as good as Llama-2 with 7 billion parameters. But when it comes to performance, you will still end up using all 16 billion of them.
Just not at the same time. However, when it comes to speed, that's a different story because MoE allows us to do far less computation. And lowering their computational requirements and moving less data between their GPUs was actually essential to their success.
Because, as I've already mentioned, China doesn't have access to the most powerful GPUs right now. Last thing I want to mention about V3 is that they train the model mostly in FP8 precision. That means, each parameter was represented by 8 bits.
People usually use FP16 or BF16 for training, which is much more stable, but it requires double the memory - 16 bits. Well, not something DeepSeek wanted to do, right? So, yeah.
. . this on its own can basically double your training speed.
But, it's not so simple. They had to use this mixed-precision framework, where the most compute density operations are done in FP8 and some other more important computations are done in higher precisions. And, it seems like it worked well!
If we can consistently replicate this, everyone will most likely start doing training in FP8 now. The second model released by DeepSeek is called R1. And, R1 is a reasoning model, similar to OpenAI's o1 or o3.
The difference between non-reasoning models and reasoning models is that when we use a non-reason model like Llama, we kind of ask it to spit out the answer right away. Whereas the reasoning model will first generate lots of reasoning text, which gives the model space and time to self-reflect and explore different reasoning paths. And the final answer is generated at the end.
But, OpenAI hides this internal reasoning output from us. So, it's harder to understand how it works, verify the reasoning, and also to replicate OpenAI's work. And DeepSeek is the first company that figured it out!
They published a model that matches the performance of the OpenAI's reasoning models, and they told us everything about it! So how does it work? First, R1 builds on top of V3.
They already have this huge 700 billion parameter language model. And spend those $6 million on it. And so now DeepSeek just wants to teach it how to reason better.
One way of learning this complex reasoning behaviour would be to train the model on examples of such reasoning. For example, if you want high-quality data, you could have human annotators solving math problems, explaining their reasoning step by step, writing it down, and then you take that and use it for supervised learning. But, that would be quite expensive!
So, instead, they were like: What if we only use reinforcement learning? Basically, they led the model to generate lots of solutions for a given problem, and then they used a rule-based reward system to identify correct answers and correct reasoning steps, and then they reinforced those outputs so that the model is more likely to generate them again. And, as they train their model, they noticed one interesting behaviour.
The average length of response was slowly increasing and increasing. Basically, the model was learning to reason for longer and longer. Of course, that could just mean that it's producing longer and longer nonsense.
But, no! These are the final results, and they are very close to OpenAI's reasoning models. But, their conclusion is that this model still struggles with challenges like poor readability and language mixing.
So, they actually decided to collect a small amount of high-quality data and use it to fine-tune the V3 model first before reinforcement learning. And that's the final R1 model, which scores very high on all benchmarks. Now, you might think, surely that's all we got from DeepSeek.
Nope, they actually did one more thing that's really good for us! They took R1, asked it a few thousand questions, and then they trained models like Llama and Qwen on those outputs. So, they basically distilled the reasoning from the large model into these smaller models.
And these distilled models are what you actually want to be using if you plan to try DeepSeek locally because the original models are just way too big. But, using these distilled models is super easy. You can just use Ollama, choose any of these options, and.
. . I will do "ollama run deepseek-r1:8b" and then just wait.
. . and wait.
. . and eventually, we get their model running!
Now, a few thoughts on the training cost and DeepSeek as a company. $5. 6 million is just the cost of their final training run - a single run!
To get there, they had to do lots of research. And they had to run many trainings in parallel to test everything this fast. We can see in their papers lots of interesting innovations.
But, when you have so many moving parts, you have to ablate all of them to be sure that those decisions are justified before they get into the final product. That all had to be very expensive. Also, the data processing of almost 15 trillion tokens couldn't be cheap either.
But, fortunately, they released the model and many of their findings, so the AI community can now benefit from all of this! Regarding DeepSeek, I hear people calling them a random startup and that this model is just a side project. That's definitely not true.
They've been publishing lots of research papers over the past year or two, such as all of these. . .
And, lots of excellent people from DeepSeek, and from academia were involved in these projects. So, there has been lots of intentional work behind this. They also did many advanced optimizations on PTX, which is an intermediate instruction set that sits between the GPU's native machine code and CUDA.
And, this is very hard to do! And, it reflects the skills of those people, and I would say a lot of money and time was invested into this. Overall, this is definitely a win for open research.
Before DeepSeek, only the largest companies like OpenAI, Google, and Meta could train their models, and now it levels the field a bit and allows more new players. For example, Hugging Face already started reproducing the R1 model, and I'm very excited to see where this will go! And, maybe, even universities will be able to join the race and come up with their own models now.
Like, I'm doing my PhD at Cambridge now, and there is no way Cambridge would have invested billions into training models like "o1". Well, 5 million is still a lot, but that's a different kind of discussion. The point is that it's getting more affordable, and this is definitely a win for open research.
So that's all for today - please consider subscribing if you liked the video and thank you for watching!
