On Wednesday, Microsoft introduced Muse, a generative AI model designed to transform how games are conceptualized, developed, and preserved. Built on the World and Human Action Model (WHAM), Muse can generate game visuals, predict controller inputs, or even combine both to create dynamic gameplay sequences. This innovation, developed in collaboration with Xbox Game Studios’ Ninja Theory, aims to empower game developers and storytellers by offering new tools to enhance their creative processes.
The research behind Muse, published in the international journal Nature, was spearheaded by Microsoft Research’s Game Intelligence and Teachable AI Experiences (Tai X) teams. The model was trained on over one billion images and controller actions from Bleeding Edge, a multiplayer game developed by Ninja Theory. This dataset represents more than seven years of continuous gameplay, giving Muse a deep understanding of 3D game worlds, physics, and player interactions.
How Muse works
Muse’s capabilities are rooted in its ability to generate consistent, diverse, and persistent gameplay sequences. For instance, if given a prompt, Muse can create a two-minute gameplay clip that adheres to the game’s dynamics (consistency), introduces variations (diversity), and maintains key elements throughout (persistency). Early versions of the model struggled with accuracy, but iterative training on advanced GPU clusters, including NVIDIA’s H100s, significantly improved its performance.
One of Muse’s standout features is its potential to revive classic games. By analyzing gameplay data and visuals, Muse could optimize older titles for modern devices, making them accessible to new generations of players. “Countless classic games tied to aging hardware are no longer playable,” said Fatima Kardar, Microsoft’s corporate vice president of gaming AI. “Muse could change how we preserve and experience these games in the future.”
AI in gaming: A tool for developers, not a replacement
Despite Muse’s capabilities, Microsoft insists AI is not meant to replace human game developers. Dom Matthews, head of Ninja Theory, emphasized that Muse is a tool to enhance creativity, not take it over. Still, AI’s role in gaming remains a controversial topic. As game studios continue to embrace AI-driven tools, concerns about automation and job displacement persist. Microsoft has reassured developers that game creators will be at the center of its AI initiatives.
Future applications of Muse
Microsoft is already exploring Muse’s potential in real-time playable AI models and prototyping new gameplay experiences. The company also envisions applications beyond gaming, such as in interior design and architectural modeling, thanks to Muse’s ability to visualize and navigate 3D spaces.
“Beyond gaming, I’m excited by the potential of this capability to enable AI assistants that understand and help visualize things, from reconfiguring the kitchen in your home to redesigning a retail space to building a digital twin of a factory floor to test and explore different scenarios,” Peter Lee, president, Microsoft Research, said in a blog post.