What Cosmos 3 Is and Why Physical AI Needs It
Cosmos 3 is an open world physical AI foundation model that combines multimodal scene understanding, world modeling, and action generation so robots and autonomous systems can perceive complex environments, predict what will happen next, and output realistic behaviors in a single architecture. Instead of training separate perception and control networks, Cosmos 3 gives developers one model that can read text, see images and video, and then generate future frames, descriptive language, or action sequences. This matters because physical AI foundation models must understand environments before taking meaningful actions; a robot arm or autonomous vehicle cannot plan safely without a mental model of how objects move and interact. By centering world modeling AI as the bridge between sensing and control, Cosmos 3 aims to cut physical AI development cycles from long, brittle pipelines into more unified, adaptable systems that can be tuned per domain.
Inside the Mixture-of-Transformers Architecture
At the core of Cosmos 3 is a mixture-of-transformers design split into two coordinated towers that share a unified modeling space. The reasoner tower is a vision-language model that takes multimodal input—images, video, and text—and uses an autoregressive transformer to understand motion, object interactions, and higher-level physical context. It acts as the model’s “brain,” forming a coherent internal description of the scene before anything is generated. The generator tower then conditions a diffusion-based process on that understanding to create physics-aware video or action outputs. This mixture-of-transformers setup lets the same physical AI foundation model support text-only reasoning, world simulation, or full action prediction without external orchestration. Developers can call the reasoner alone for analysis, or activate both towers to create guided synthetic data, edge-case scenarios, or action-conditioned rollouts for autonomous robot training and evaluation.
From Scene Understanding to Real-World Robot Actions
Cosmos 3 is designed for applications where perception, prediction, and control must stay tightly coupled, including Cosmos 3 robotics workflows, autonomous vehicles, and smart spaces. Supported modality pairings show how the model spans the full loop: it can take text and images to generate videos for world prediction, accept video and text to output video plus actions as a world action model, or map action, video, and text into action-conditioned world simulations. This allows developers to build policy models for robot learning that are grounded in visually rich, physics-aware rollouts. In autonomous driving, for example, Cosmos 3 can generate rare edge-case videos from text prompts, then attach candidate control sequences, accelerating data generation for safety testing. For warehouses and smart facilities, it can simulate human motion and object interactions, enabling world modeling AI that supports monitoring, planning, and incident prevention.
Open Models, Datasets, and Deployment for Physical AI
Nvidia is releasing Cosmos 3 as a family of open physical AI foundation models, with Cosmos 3 Nano at 16B parameters for workstation-scale inference and Cosmos 3 Super at 64B parameters aimed at data centers. According to Nvidia, Cosmos 3 is “the world’s first fully open omnimodel that can natively understand and generate text, images, video, ambient sound and actions with leading physics accuracy.” The company is publishing checkpoints on Hugging Face, code on GitHub, and open post-training scripts so teams can adapt the mixture-of-transformers model to their own embodiment and tasks. Six synthetic world modeling datasets covering robotics, driving, warehouse safety, and spatial reasoning further support customization. Paired with Cosmos NIM microservices and the OpenMDW-1.1 distribution framework, developers gain a single legal and technical stack to train, modify, and deploy reasoning, world, and action models for real-world systems.