Simplify End-to-End Autonomous Vehicle Development with New NVIDIA Cosmos World Foundation Models

NVIDIA Cosmos World Foundation Models in Autonomous Vehicle Development

• Introduction
  • NVIDIA Cosmos world foundation models (WFMs)can simplify end-to-end autonomous vehicle (AV) development by adapting and post-training models to the AV domain.
• Developing Synthetic Data Generation Pipelines
  • NVIDIA Research post-trained Cosmos WFMs on 20,000 hours of driving data for AV development workflows, improving performance in AV model training.
• AV-Specific Models
  • Cosmos WFMs optimize synthetic data generation for AV training, especially through data augmentation with models like Cosmos-Transfer-1-7B-Sample-AV and Cosmos-Transfer-1-7B-Single2Multiview-Sample_AV.
• Synthetic Data Pipelines
  • A pipeline using Cosmos models starts with text prompts and real-world data, outputting physically based multi-view videos for development purposes.
• Integrating Cosmos into Existing AV Workflows
  • Open-source simulators and AV companies are integrating Cosmos models into their toolchains, enabling accelerated synthetic data generation for developers worldwide.
• Cosmos Transfer
  • Introduced at GTC Paris, Cosmos Transfer NIM is a containerized version for accelerated inference and is being integrated into simulators like CARLA and AV development environments like Mcity.
• Cosmos Predict
  • Also announced at GTC Paris, Cosmos Predict-2 is a high-performing model for future world state prediction with enhanced fidelity and improved control in video generation.
• AV Industry Adoption
  • The AV industry is embracing end-to-end WFMs for AV development, emphasizing the importance of vast, diverse, and physically accurate sensor data.

For more details, explore the NVIDIA research papers to be presented at CVPR 2025 and watch the NVIDIA GTC Paris keynote by NVIDIA founder and CEO Jensen Huang.