Last week, Chinese research lab DeepSeek (founded in 2023 by quantitative trader Liang Wenfeng) rocked the AI world with the launch of their open source reasoning model, R1. This model scores exceptionally well on key capability benchmarks. What’s causing waves, however, isn’t just the capability of this reasoning model, it’s the cost and time required to build it. In a world dominated by multi-billion dollar investments, over years, DeepSeek managed to build a leading reasoning model in a few months for under $6m. This is incredible.
Comparisons are being drawn to OpenAI’s reasoning model (O1) and also to Google’s Gemini 2.0 Flash model. Both of these are promoted for their reasoning capabilities, perhaps due to the absence of compelling AGI. These models are built by the best minds Silicon Valley has to offer, with decades of experience, and the creds from the best universities. DeepSeek however, has a focus on hiring talented graduates. The difference is stark, but the outcomes are compelling.
With all these developments in AI, whether they’re driven by billions of dollars at OpenAI and Google, or lean efficient development at DeepSeek, surely AI safety has improved too? As Yoshua Bengio notes, AI safety must be taken seriously and Nobel Prize winner Geoffrey Hinton calls for urgent research into AI safety.
Using our automated AI safety testing software, we have tested DeepSeek’s R1 reasoning model and added it to our public results here: https://chatterbox.co/ai-safety.
And here’s where it gets real: even with reasoning models, there’s significantly more to do on AI safety:
Let’s look at the metrics behind these results for the three reasoning models under discussion: OpenAI O1, Google Gemini 2.0 Flash and DeepSeek R1. In these charts red signifies lower AI safety and security.
Starting with OpenAI, whilst they have achieved good progress over their prior models, there’s still work to do in some key areas:
Google Gemini 2.0 Flash however show weaknesses across the board:
And, whilst DeepSeek have created a very capable model, this model also exhibits AI safety weaknesses across the board:
And note, these models are all being accessed from the vendor’s own cloud environment using their chosen default settings.
It will be interesting to take note of how this plays out. At the speed with which DeepSeek develop and release models, will they end up as the leader in AI safety for reasoning models in the near future?
And how does the Enterprise respond? Given what DeepSeek have shown is possible, will Enterprise organizations start to develop their own AI models, based on open source, to control their own AI? This would bring the AI stack under their own in house control, including AI hardware, AI data, AI models, AI guardrails and AI safety testing? Let’s see…