đ Blogs & Articles â Awesome Generative Models
Loading postsâŚ
Loading postsâŚ
45 posts, articles, and resources from across the field.
Recent developments focus on creating unified models that integrate various data types.
3 postsResearch is exploring the integration of generative models with robotics for enhanced performance.
2 postsThe field is advancing techniques for improving the performance of vision-language models.
2 postsWhy it matters â GRASP introduces a novel gradient-based planning technique that enhances long-horizon planning in learned dynamics, crucial for developing more effective world models in generative AI.
Why it matters â PLAID's ability to generate both protein sequences and structures using latent diffusion models represents a significant advancement in multimodal generative modeling, with implications for bioinformatics and drug discovery.
Why it matters â This guide provides essential strategies for prompt engineering in vision language models, which is vital for optimizing their performance in image and video understanding tasks.
Why it matters â The rise of world-action models highlights the integration of pretrained vision-language models into robotic policies, enhancing their adaptability and performance in real-world tasks.
Why it matters â VSAS-Bench provides a framework for real-time evaluation of visual streaming assistant models, which is essential for developing responsive and adaptive generative models in visual and language processing.
Why it matters â BalCapRL presents a balanced approach to image captioning using reinforcement learning and multimodal large language models, addressing the challenge of generating detailed captions from images.
Why it matters â This post discusses the integration of generative AI with computer vision pipelines, showcasing how generative techniques can transform video analytics from simple counting to complex content generation.
Why it matters â The NVIDIA AI Blueprint illustrates how vision language models can enhance video analytics, providing a framework for richer contextual understanding and broader perception in AI applications.
Why it matters â This post outlines the potential of VLMs in creating real-time multimodal applications, emphasizing their role in advancing visual computing technologies.
Why it matters â Exploring the robustness and consistency of RL-finetuned VLMs is critical for improving their performance on reasoning-intensive tasks, which is a key challenge in generative models.
Why it matters â NVIDIA Cosmos Reason introduces a customizable reasoning model for robotics, emphasizing the importance of adaptability in physical AI applications for enhanced performance.
Why it matters â This benchmarking post provides insights into evaluating LLM and VLM reasoning capabilities in gaming, which is essential for understanding their performance in interactive environments.
Why it matters â The discussion on open research and AI safety is crucial for generative models researchers, as it addresses the ethical implications and safety measures necessary for responsible AI development.
Why it matters â Gemma 4 12B introduces a unified multimodal model, which is significant for researchers looking to streamline generative tasks across different modalities without the complexity of encoders.
Why it matters â Nemotron 3.5 focuses on customizable content safety in multimodal AI, which is increasingly important for enterprises looking to deploy generative models responsibly.
Why it matters â This post highlights the shift towards multimodal AI applications, emphasizing the need for efficient processing across diverse data types, which is key for generative model integration.
Why it matters â The introduction of long-context multimodal intelligence in the NVIDIA Nemotron 3 Nano Omni model is significant for enhancing generative capabilities in handling complex documents and media.
Why it matters â Nano Banana 2's advancements in image generation speed and quality are crucial for researchers focused on real-time applications in generative modeling.
Why it matters â Mistral 3's family of open-source multimodal models represents an important contribution to the accessibility and diversity of generative modeling tools available to researchers.
Why it matters â Whole-Body Conditioned Egocentric Video Prediction addresses the challenge of predicting future video frames based on human actions, which is vital for advancing generative models in video synthesis.
Why it matters â Mistral Small 3.1 sets a new standard for multimodal and multilingual capabilities in generative models, which is essential for broadening their applicability across languages and contexts.
Why it matters â The latest addition to Microsoft Phi SLMs showcases advancements in large language models that are more practical for real-world applications, a key consideration for generative model researchers.
Why it matters â Segment Anything's development towards a foundation model for image segmentation is a significant step for researchers focused on enhancing generative capabilities in visual tasks.
Why it matters â This guide on Edge AI emphasizes the importance of deploying advanced AI models on power-efficient devices, which is crucial for the practical application of generative models in robotics and smart devices.
Why it matters â The upcoming livestream on fine-tuning NVIDIA Cosmos Reason for visual AI agents is a practical opportunity for researchers to learn about customizing generative models for specific tasks.
Why it matters â The focus on multimodal extraction for efficient AI pipelines highlights the need for generative models that can handle diverse data types, which is increasingly relevant in enterprise contexts.
Why it matters â The BAIR Graduate Showcase highlights the contributions of new Ph.D. graduates, underscoring the ongoing research and innovation in generative models and AI as a whole.
Apple is presenting new research at the annual IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), which takes place in person in Denver at the Colorado Convention Center from June 3 to June 7. We are proud to sponsor the conference, which brings together the scientific and industrial research communities in computer vision and pattern recognition. Below is an overview of Appleâs participation at CVPR 2026.
As a founding member of the NVIDIA Nemotron Coalition, Mistral AI is contributing large-scale model development and multimodal capabilities.
Why it matters â This post challenges the notion of multimodality as a requirement for AGI, prompting researchers to reconsider the foundational principles underlying generative models and their relationship to human-like intelligence.
Explore visually perceptive AI agents, the latest vision AI technologies, hands-on training, and inspiring deployments.
Empowering conservation efforts through innovative technologies and global collaborationA vessel captured by NASAâs Landsat 8. Skylightâs computer vision models leverage this imagery to identify suspicious behavior, such as illegal fishing, helping authorities act quickly to protect marine ecosystems.As our world faces unprecedented environmental challengesââârising global temperatures, increasingly severe extreme weather events, and a rapidly deepening biodiversity crisisâââthe need for innovative, scalable solutions has never been more urgent. AI is already playing a transformative role in addressing these global issues. For instance, AI camera traps equipped with satellite and cellular connectivity have reduced the data-to-action timeline from months to minutes, transforming how conservation is done. Meanwhile, advanced computer vision models are being used to detect and stop illegal fishing, estimate potential wildfire risks, and understand our changing planet. As a leader in developing state-of-the-art conservation technologies, Ai2 is heading to this yearâs United Nations Biodiversity Conference of the Parties (COP16) to showcase how open, collaborative AI can galvanize communities and solve the worldâs biggest problems.Taking place from October 21 to November 1 in Cali, Colombia, COP16 is one of the largest and most important global gatherings focused on addressing the urgent crisis of biodiversity loss. Current estimates indicate that at least one million plant and animal species are at risk of extinction, underscoring the need for immediate action. COP16 marks a pivotal moment as the first major opportunity to assess how national governments are moving from commitments to action.âWe are at a critical juncture where technology, collaboration, and urgency must intersect to drive real change for our planetââââAi2âs Ted SchmittWith 190 countries agreeing to protect 30 percent of the planetâs land and oceans by 2030, this yearâs conference will focus on the progress toward fulfilling that goal. Growing in just six years from 18 percent of land and 8 percent of the ocean currently protected, the need for tools to close this gap is urgent and critical. Ai2âs free and open-source technologies, like Skylight and EarthRanger, are uniquely positioned to empower governments and NGOs to turn their national action plans into reality by providing real-time data and insights to drive smarter, faster decisions in conservation efforts. This yearâs COP is a crucial event in assessing whether countries are on track and creating momentum to deliver on their promises and safeguard our planetâs biodiversity for future generations.A researcher points at her EarthRanger instance. Ai2âs EarthRanger is a cutting-edge platform designed for wildlife conservation, enabling real-time monitoring and data collection to protect threatened species and manage protected areas and landscapes effectively.At COP16, we will concentrate on two primary objectives: fostering and enhancing partnerships with global conservation stakeholders and inspiring recognition of the transformative potential of AI and conservation technologies in safeguarding biodiversity. While AI and technology are not a one-size-fits-all solution to the pressing challenges we face, they significantly accelerate the ability of conservationists and governments to adapt and respond to these rising threats. By co-hosting a pavilion in the Blue Zone with our partners, the Group on Earth Observations (GEO), GEO BON, and ESRI, we will demonstrate how nongovernmental organizations, collectively, are unlocking the data urgently needed to advance ecosystem protection and deliver on the ambitious targets set out in the global biodiversity framework.During COP16, we will host several presentations at the pavilion, including EarthRanger, Skylight, and our newly launched Earth System programs. We invited additional partners, including iNaturalist, leading AI for Conservation researchers at the Smithsonian Institution, MIT, WildLabs, and more, to share the latest technology developments in delivering conservation and environmental solutions. We will also join partners such as GEO, Maldivesâ Ministry of Climate Change, Environment, and Energy, Esri, Planet Labs, USGS, and Maxar to announce how the Global Ecosystems Atlas is empowering the Maldives to make better decisions faster. This initiative leverages advanced satellite imagery and AI to create the first-ever comprehensive map of ecosystems in the Maldives, providing a scalable, AI-powered resource that delivers detailed intelligence.In partnership with global spatial data providers, Ai2 will develop a globally accurate, regularly updated cropland map that will be immediately useful to a wide range of stakeholders seeking to better understand the vital food systems they manage. These AI-driven crop maps will strengthen agricultural monitoring to secure the availability of high-quality, nutritious food and economic stability.âWe are at a critical juncture where technology, collaboration, and urgency must intersect to drive real change for our planet,â said Ted Schmitt, Senior Director of Conservation at Ai2. âWe are excited to announce the launch of our latest philanthropic conservation technology program. EarthSystem will build on the highly impactful EarthRanger, Skylight, WildLands, and Climate programs. Through those programs, Ai2 is delivering real solutions to a range of environmental and biodiversity challenges. Leveraging our world-class AI researchers and engineers focused fully on conservation impact, EarthSystem will transform the power and promise of AI from dream to reality; we look forward to working with a range of partners across conservation and environmental domains to create game-changing solutions.âThese presentations will give attendees insights into how various technologies are helping to transform wildlife conservation, ocean health, and environmental monitoring. Presentations will take place throughout the conference, and we invite you to join us at the following sessions:Ai2 at COP16(all times Colombia Standard Time)Location- 22 October11:45â1 pm: Earth SystemâââHarnessing AI for Environmental Impact6:00â7:20 pm: Monitoring, control, and surveillance of future high seas MPAs: What role for emerging technologies (Skylight)- 23 October4:45â6:15 pm: Earth SystemâââHarnessing AI for Environmental Impact- 24 October11:40 am: Charting the FutureâââDeveloping Conservation Technology for Biodiversity Monitoring (EarthRanger and Ai2)3:00â4:30 pm: Transforming Wildlife Conservation with the Support of EarthRanger- 26 October1:15â2:45 pm: Transforming Wildlife Conservation with the Support of EarthRanger- 27 October1:15â2:45 pm: Earth SystemâââHarnessing AI for Environmental Impact6:30â7:30 pm: Transforming Wildlife Conservation with the Support of EarthRanger- 28 October3:00â4:30 pm: SkylightâââCombatting IUU Fishing with Cutting-Edge AI- 29 October11:45â1 pm: SkylightâââCombatting IUU Fishing with Cutting-Edge AI (presented in Spanish)- 30 October11:45â1 pm: SkylightâââCombatting IUU Fishing with Cutting-Edge AI1:15â2:45 pm: Earth SystemâââHarnessing AI for Environmental Impact6:00 pm: Mapping Natureâs Footprint: The Global Ecosystem AtlasAi2 at COP 16: Harnessing AI and Conservation Tech to Protect Our Planet was originally published in Ai2 Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Vision Language Models (VLMs) offer the exciting possibility of processing text as rendered images, bypassing the need for tokenizing the text into long token sequences. Since VLM image encoders map fixed-size images to a fixed number of visual tokens, varying rendering resolution provides a fine-grained compression knob. However, accuracy deteriorates quickly as compression increases: characters shrink below the vision encoderâs effective resolution, making them indistinguishable. To address this, we propose LensVLM, an inference framework and post-training recipe that enables VLMs to scanâŚ
NVIDIA XR AI is now available in public beta, giving developers a framework for building multimodal AI agents for AR glasses and XR devices.  
Normalizing Flows (NFs) are a classical family of likelihood-based methods that have received revived attention. Recent efforts such as TARFlow have shown that NFs are capable of achieving promising performance on image modeling tasks, making them viable alternatives to other methods such as diffusion models. In this work, we further advance the state of Normalizing Flow generative models by introducing iterative TARFlow (iTARFlow). Unlike diffusion models, iTARFlow maintains a fully end-to-end, likelihood-based objective during training. During sampling, it performs autoregressive generationâŚ
True spatial intelligence for multimodal agents transcends low-level geometric perception, evolving from knowing where things are to understanding what they are for. While existing benchmarks, such as VSI-Bench, effectively evaluate this foundational geometric stage, they fall short of probing the higher-order cognitive abilities essential for grounded intelligence. To bridge this gap, we introduce the Spatial-Functional Intelligence Benchmark (SFI-Bench), a video-based benchmark with over 1700 questions derived from diverse, egocentric indoor video scans. SFI-Bench is designed toâŚ
The automotive cockpit is undergoing a fundamental shift from rule-based interfaces to agentic, multimodal AI systems capable of reasoning, planning, and...
Agentic systems often reason across screens, documents, audio, video, and text within a single perceptionâtoâaction loop. However, they still rely on...
AI agent systems today juggle separate models for vision, speech and language â losing time and context as they pass data from one model to the other. Unveiled today, NVIDIA Nemotron 3 Nano Omni is an open multimodal model that brings these capabilities together into one system, enabling agents to deliver faster, smarter responses with […]
Training Diffusion Models with Reinforcement Learning We deployed 100 reinforcement learning (RL)-controlled cars into rush-hour highway traffic to smooth congestion and reduce fuel consumption for everyone. Our goal is to tackle "stop-and-go" waves, those frustrating slowdowns and speedups that usually have no clear cause but lead to congestion and significant energy waste. To train efficient flow-smoothing controllers, we built fast, data-driven simulations that RL agents interact with, learning to maximize energy efficiency while maintaining throughput and operating safely around human drivers. Overall, a small proportion of well-controlled autonomous vehicles (AVs) is enough to significantly improve traffic flow and fuel efficiency for all drivers on the road. Moreover, the trained controllers are designed to be deployable on most modern vehicles, operating in a decentralized manner and relying on standard radar sensors. In our latest paper, we explore the challenges of deploying RL controllers on a large-scale, from simulation to the field, during this 100-car experiment. The challenges of phantom jams A stop-and-go wave moving backwards through highway traffic. If you drive, youâve surely experienced the frustration of stop-and-go waves, those seemingly inexplicable traffic slowdowns that appear out of nowhere and then suddenly clear up. These waves are often caused by small fluctuations in our driving behavior that get amplified through the flow of traffic. We naturally adjust our speed based on the vehicle in front of us. If the gap opens, we speed up to keep up. If they brake, we also slow down. But due to our nonzero reaction time, we might brake just a bit harder than the vehicle in front. The next driver behind us does the same, and this keeps amplifying. Over time, what started as an insignificant slowdown turns into a full stop further back in traffic. These waves move backward through the traffic stream, leading to significant drops in energy efficiency due to frequent accelerations, accompanied by increased CO2 emissions and accident risk. And this isnât an isolated phenomenon! These waves are ubiquitous on busy roads when the traffic density exceeds a critical threshold. So how can we address this problem? Traditional approaches like ramp metering and variable speed limits attempt to manage traffic flow, but they often require costly infrastructure and centralized coordination. A more scalable approach is to use AVs, which can dynamically adjust their driving behavior in real-time. However, simply inserting AVs among human drivers isnât enough: they must also drive in a smarter way that makes traffic better for everyone, which is where RL comes in. Fundamental diagram of traffic flow. The number of cars on the road (density) affects how much traffic is moving forward (flow). At low density, adding more cars increases flow because more vehicles can pass through. But beyond a critical threshold, cars start blocking each other, leading to congestion, where adding more cars actually slows down overall movement. Reinforcement learning for wave-smoothing AVs RL is a powerful control approach where an agent learns to maximize a reward signal through interactions with an environment. The agent collects experience through trial and error, learns from its mistakes, and improves over time. In our case, the environment is a mixed-autonomy traffic scenario, where AVs learn driving strategies to dampen stop-and-go waves and reduce fuel consumption for both themselves and nearby human-driven vehicles. Training these RL agents requires fast simulations with realistic traffic dynamics that can replicate highway stop-and-go behavior. To achieve this, we leveraged experimental data collected on Interstate 24 (I-24) near Nashville, Tennessee, and used it to build simulations where vehicles replay highway trajectories, creating unstable traffic that AVs driving behind them learn to smooth out. Simulation replaying a highway trajectory that exhibits several stop-and-go waves. We designed the AVs with deployment in mind, ensuring that they can operate using only basic sensor information about themselves and the vehicle in front. The observations consist of the AVâs speed, the speed of the leading vehicle, and the space gap between them. Given these inputs, the RL agent then prescribes either an instantaneous acceleration or a desired speed for the AV. The key advantage of using only these local measurements is that the RL controllers can be deployed on most modern vehicles in a decentralized way, without requiring additional infrastructure. Reward design The most challenging part is designing a reward function that, when maximized, aligns with the different objectives that we desire the AVs to achieve: Wave smoothing: Reduce stop-and-go oscillations. Energy efficiency: Lower fuel consumption for all vehicles, not just AVs. Safety: Ensure reasonable following distances and avoid abrupt braking. Driving comfort: Avoid aggressive accelerations and decelerations. Adherence to human driving norms: Ensure a ânormalâ driving behavior that doesnât make surrounding drivers uncomfortable. Balancing these objectives together is difficult, as suitable coefficients for each term must be found. For instance, if minimizing fuel consumption dominates the reward, RL AVs learn to come to a stop in the middle of the highway because that is energy optimal. To prevent this, we introduced dynamic minimum and maximum gap thresholds to ensure safe and reasonable behavior while optimizing fuel efficiency. We also penalized the fuel consumption of human-driven vehicles behind the AV to discourage it from learning a selfish behavior that optimizes energy savings for the AV at the expense of surrounding traffic. Overall, we aim to strike a balance between energy savings and having a reasonable and safe driving behavior. Simulation results Illustration of the dynamic minimum and maximum gap thresholds, within which the AV can operate freely to smooth traffic as efficiently as possible. The typical behavior learned by the AVs is to maintain slightly larger gaps than human drivers, allowing them to absorb upcoming, possibly abrupt, traffic slowdowns more effectively. In simulation, this approach resulted in significant fuel savings of up to 20% across all road users in the most congested scenarios, with fewer than 5% of AVs on the road. And these AVs donât have to be special vehicles! They can simply be standard consumer cars equipped with a smart adaptive cruise control (ACC), which is what we tested at scale. Smoothing behavior of RL AVs. Red: a human trajectory from the dataset. Blue: successive AVs in the platoon, where AV 1 is the closest behind the human trajectory. There is typically between 20 and 25 human vehicles between AVs. Each AV doesnât slow down as much or accelerate as fast as its leader, leading to decreasing wave amplitude over time and thus energy savings. 100 AV field test: deploying RL at scale Our 100 cars parked at our operational center during the experiment week. Given the promising simulation results, the natural next step was to bridge the gap from simulation to the highway. We took the trained RL controllers and deployed them on 100 vehicles on the I-24 during peak traffic hours over several days. This large-scale experiment, which we called the MegaVanderTest, is the largest mixed-autonomy traffic-smoothing experiment ever conducted. Before deploying RL controllers in the field, we trained and evaluated them extensively in simulation and validated them on the hardware. Overall, the steps towards deployment involved: Training in data-driven simulations: We used highway traffic data from I-24 to create a training environment with realistic wave dynamics, then validate the trained agentâs performance and robustness in a variety of new traffic scenarios. Deployment on hardware: After being validated in robotics software, the trained controller is uploaded onto the car and is able to control the set speed of the vehicle. We operate through the vehicleâs on-board cruise control, which acts as a lower-level safety controller. Modular control framework: One key challenge during the test was not having access to the leading vehicle information sensors. To overcome this, the RL controller was integrated into a hierarchical system, the MegaController, which combines a speed planner guide that accounts for downstream traffic conditions, with the RL controller as the final decision maker. Validation on hardware: The RL agents were designed to operate in an environment where most vehicles were human-driven, requiring robust policies that adapt to unpredictable behavior. We verify this by driving the RL-controlled vehicles on the road under careful human supervision, making changes to the control based on feedback. Each of the 100 cars is connected to a Raspberry Pi, on which the RL controller (a small neural network) is deployed. The RL controller directly controls the onboard adaptive cruise control (ACC) system, setting its speed and desired following distance. Once validated, the RL controllers were deployed on 100 cars and driven on I-24 during morning rush hour. Surrounding traffic was unaware of the experiment, ensuring unbiased driver behavior. Data was collected during the experiment from dozens of overhead cameras placed along the highway, which led to the extraction of millions of individual vehicle trajectories through a computer vision pipeline. Metrics computed on these trajectories indicate a trend of reduced fuel consumption around AVs, as expected from simulation results and previous smaller validation deployments. For instance, we can observe that the closer people are driving behind our AVs, the less fuel they appear to consume on average (which is calculated using a calibrated energy model): Average fuel consumption as a function of distance behind the nearest engaged RL-controlled AV in the downstream traffic. As human drivers get further away behind AVs, their average fuel consumption increases. Another way to measure the impact is to measure the variance of the speeds and accelerations: the lower the variance, the less amplitude the waves should have, which is what we observe from the field test data. Overall, although getting precise measurements from a large amount of camera video data is complicated, we observe a trend of 15 to 20% of energy savings around our controlled cars. Data points from all vehicles on the highway over a single day of the experiment, plotted in speed-acceleration space. The cluster to the left of the red line represents congestion, while the one on the right corresponds to free flow. We observe that the congestion cluster is smaller when AVs are present, as measured by computing the area of a soft convex envelope or by fitting a Gaussian kernel. Final thoughts The 100-car field operational test was decentralized, with no explicit cooperation or communication between AVs, reflective of current autonomy deployment, and bringing us one step closer to smoother, more energy-efficient highways. Yet, there is still vast potential for improvement. Scaling up simulations to be faster and more accurate with better human-driving models is crucial for bridging the simulation-to-reality gap. Equipping AVs with additional traffic data, whether through advanced sensors or centralized planning, could further improve the performance of the controllers. For instance, while multi-agent RL is promising for improving cooperative control strategies, it remains an open question how enabling explicit communication between AVs over 5G networks could further improve stability and further mitigate stop-and-go waves. Crucially, our controllers integrate seamlessly with existing adaptive cruise control (ACC) systems, making field deployment feasible at scale. The more vehicles equipped with smart traffic-smoothing control, the fewer waves weâll see on our roads, meaning less pollution and fuel savings for everyone! Many contributors took part in making the MegaVanderTest happen! The full list is available on the CIRCLES project page, along with more details about the project. Read more: [paper]