On July 18th, OpenAI announced the launch of GPT-4o mini, a highly cost-effective small model.

Small and excellent

OpenAI describes GPT-4o mini as a small model with excellent text intelligence and multimodal reasoning capabilities.

According to OpenAI, GPT-4o mini has a score of 82% on MMLU and currently outperforms GPT-4 in terms of chat preferences on the LMSYS leaderboard.

GPT-4o mini has surpassed GPT-3.5 Turbo and other small models in academic benchmark tests for text intelligence and multimodal reasoning, and supports the same language range as GPT-4o.

GPT-4o mini also performs well in function calling, allowing developers to build applications that retrieve data or take action using external systems. In addition, compared to the GPT-3.5 Turbo, the GPT-4o mini has improved long context performance.

The GPT-4o mini has been evaluated on several key benchmarks.Inference task: GPT-4o mini outperforms other small models in text and visual inference tasks, with a score of 82.0% on the text intelligence and inference benchmark MMLU, compared to 77.9% for Gemini Flash and 73.8% for Claude Haiku.

Mathematical and coding abilities: GPT-4o mini performs better than previous small models on the market in mathematical reasoning and coding tasks. On the MGSM for measuring mathematical reasoning, GPT-4o mini scored 87.0%, Gemini Flash scored 75.5%, and Claude Haiku scored 71.7%. On HumanEval, which measures encoding performance, GPT-4o mini scored 87.2%, Gemini Flash scored 71.5%, and Claude Haiku scored 75.9%.

Multimodal reasoning: GPT-4o mini also performed well in the multimodal reasoning evaluation of MMMU, scoring 59.4%, while Gemini Flash scored 56.1% and Claude Haiku scored 50.2%.

OpenAI also mentioned that when the company worked with companies such as Ramp, it found that GPT-4o mini performed significantly better than GPT-3 when performing tasks such as extracting structured data from receipt files or generating high-quality email responses when providing thread history..5 Turbo。

Lower cost

Despite having excellent performance, this is not the most eye-catching aspect of the GPT-4o mini, its biggest highlight is the significant price reduction achieved.

OpenAI states that when developers use GPT-4o mini, they need to pay 15 cents for every 1 million input tokens and 60 cents for every 1 million output tokens, which is an order of magnitude cheaper than the previous Frontier model and over 60% cheaper than the GPT-3.5 Turbo.

The company stated that cost reduction will help develop applications that are affected by activity levels.

GPT-4o mini will enable multitasking with low cost and low latency, such as linking or parallelizing multiple model calls (such as calling multiple APIs), passing a large amount of context to the model (such as a complete code base or conversation history), or interacting with customers through fast, real-time text responses (such as customer support chatbots).

At present, the GPT-4o mini's application programming interface supports text and visual input, and will also support text, image, video, and audio input and output in the future. The context window of this model can accommodate 128K input tokens, with a maximum of 16K output tokens per request, and has knowledge up to October 2023. Due to the improved tokenizer shared with GPT-4o, processing non English text is now more cost-effective.

In terms of security, the GPT-4o mini is equipped with the same security mitigation measures as the GPT-4o. It is understood that over 70 external experts from fields such as social psychology and misinformation have tested GPT-4o to identify potential risks, and OpenAI has addressed these risks. OpenAI stated that the team will also be committed to utilizing newly developed technologies to enhance the security of GPT-4o mini.

OpenAI stated that GPT-4o mini has now been launched to both free and paid ChatGPT Plus and Team users, and will be available to enterprise customers next week. The GPT-4o mini will replace the old model GPT-3.5 Turbo in ChatGPT.