On March 16, local time, Google DeepMind announced the launch of a new AI system SIMA (Scalable Instructable Multiworld Agent) that is "scalable, instructable, and multi-world," which claims to be the first universal AI agent that can follow natural language instructions in a wide range of 3D virtual environments and video games.。

According to reports, SIMA is an AI agent that trains and learns game skills and can follow instructions and perform tasks in real time in the game.。For games that have never been played before, even open-world games that do not have a linear end path, it can work with the player.。

Simply put, SIMA in the game is not a "hard-working" AI, more like "another player"。Tim Harley, Google DeepMind researcher and co-head of SIMA, said: "SIMA is not trained to win games; it is trained to win games.。"

AIGC's "Pathfinder"

Since OpenAI released ChatGPT in November 2022, a number of technology companies such as Microsoft, Adobe, Meta and Anthropoic have launched their own AIGC tools.。Recently, development in this field has expanded from text writing to images, audio and video, and games.。

Google says video games are a great training ground for AI systems。As a sandbox, video games provide a safe and accessible test method that not only allows AI to learn to play games, but also learn to translate abstract language into practical actions, making it more practical in various environments。

SIMA is currently in the research phase.。To give SIMA more exposure to the game environment, the development team says it has now worked with eight game studios to train and test it on nine video games.。Frederic Besse, a research engineer at Google DeepMind who worked on the project, said: "SIMA was able to take advantage of shared concepts in the game, learn better skills and learn to execute instructions better.。"

In the field of games and AI, Google has been regarded as the "veteran" level。From the early collaboration with the gaming platform Atari to the development of the AlphaStar system that plays StarCraft II at the level of a human grandmaster, the birth of SIMA marks a new milestone in Google's AI development.。

It focuses on moving from a single game to a general-purpose, instructable game agent, with the extraordinary ability to understand natural language instructions and perform tasks across multiple virtual environments, which has far-reaching implications for the development of intelligent robots and interactive AI systems.。

Real gold is not afraid of fire

To make SIMA learning and training more effective, the DeepMind team chose games that focus more on open-ended games rather than narrative, which are random and spontaneous and can maximize SIMA's access to environmental information.。

In addition, SIMA does not require a custom API to play games or access source code, and can activate learning progress only with screen images and simple natural language instructions from users, thereby improving SIMA's versatility。

The development team also used four research environments, including the Unity engine to create a new environment called "Construction Lab," in which agents need to build models to test their understanding of object operations and the physical world.。In addition, the researchers avoided games with violent behavior to comply with Google's AI ethics guidelines.。

At the data level, the DeepMind team collected keyboard and mouse data from a variety of game operations, and then entered that data into the robot's language model, which trained and strengthened SIMA's language processing capabilities by digesting a huge text database.。After the evaluation of the human review, SIMA fine-tunes the performance based on manual data。

Currently, SIMA has completed the evaluation of 600 basic skills, can achieve 10 seconds of navigation, object interaction and menu use and other operations, and adapt to various scenarios。And even in untrained games, SIMA's performance is equivalent to that of a trained agent, proving its ability to generalize in a completely new environment.。

OpenAI Concept Continuation

In 2016, the concept of "game agent" appeared in the public view。OpenAI's first version of the Universe platform has been supported by Microsoft, Nvidia and many other well-known companies since its inception, working together to make it use computers like humans.。

Universe is capable of simulating a variety of games and applications and allows users to train and test their performance in a variety of environments.。In a variety of simulation environments (including Flash games, browser tasks, etc.), the agent can interact with the environment by observing screen pixels, simulating keyboard and mouse operations, etc.。Through this platform, researchers can benchmark the performance of algorithms with humans and test the algorithms of agents and compare their performance in various simulation environments。

In contrast, SIMA provides a flexible and customizable platform for researchers to explore various aspects of the agent (such as virtual reality, game development, intelligent assistants, etc.), demonstrating the potential of developing a new generation of general-purpose, language-driven AI agents, and opening up new possibilities for the development of future intelligent systems.。