š Blogs & Articles ā Awesome Multimodal
Loading postsā¦
Loading postsā¦
41 posts, articles, and resources from across the field.
The latest developments in multimodal models are pushing the boundaries of AI capabilities.
4 postsInnovations in video analytics and understanding are enhancing AI's ability to process and interpret visual data.
4 postsThe integration of AI technologies is crucial for advancing robotics performance and capabilities.
2 postsWhy it matters ā The introduction of Mistral 3 highlights advancements in open-source multimodal models, which can enhance accessibility and collaboration in multimodal research.
Why it matters ā PLAID's ability to generate both protein sequences and structures simultaneously showcases the potential of latent diffusion models in bridging the gap between different modalities in biological research.
Why it matters ā The exploration of robustness and chain-of-thought consistency in RL-finetuned VLMs highlights the importance of reinforcement learning in enhancing reasoning capabilities, a key area for multimodal researchers.
Why it matters ā Gemma 4 12B's unified, encoder-free architecture presents a significant shift in multimodal model design, allowing for more flexible and efficient processing of diverse data types.
Why it matters ā BalCapRL's balanced framework for reinforcement learning-based image captioning addresses the challenges of generating detailed captions in multimodal large language models, enhancing their usability.
Why it matters ā NVIDIA Nemotron 3 Nano Omni's long-context capabilities for documents, audio, and video agents highlight advancements in handling diverse data types simultaneously, which is crucial for developing more sophisticated multimodal applications.
Why it matters ā Mistral Small 3.1's state-of-the-art multilingual capabilities demonstrate the potential for multimodal models to operate across various languages, broadening their applicability.
Why it matters ā This research addresses the crucial aspect of temporal awareness in egocentric video understanding, which is vital for applications requiring accurate sequencing of events.
Why it matters ā Nemotron 3.5's customizable safety features for multimodal AI underline the importance of content safety in enterprise applications, ensuring responsible AI deployment.
Why it matters ā VSAS-Bench provides a real-time evaluation framework for visual streaming assistant models, which is critical for assessing the performance of multimodal AI in dynamic environments.
Why it matters ā NVIDIA Cosmos Reason's post-training capabilities can significantly improve the performance of robotics by integrating reasoning with visual language processing.
Why it matters ā The ability to run multimodal extraction on a single GPU streamlines AI pipelines, making it more feasible for enterprises to analyze diverse datasets efficiently.
Why it matters ā The NVIDIA AI Blueprint for Video Search and Summarization provides a framework for developing real-time multimodal applications, which is crucial for enhancing user experience in XR environments.
Why it matters ā This guide on prompt engineering for VLMs is essential for researchers looking to optimize image and video understanding tasks, facilitating more effective model training.
Why it matters ā LensVLM's selective context expansion technique allows for more efficient visual representation of text, which can enhance processing speed and accuracy in multimodal tasks.
Why it matters ā Integrating computer vision with generative AI opens new avenues for video analytics, enabling richer insights and more sophisticated analyses of video data.
Why it matters ā The PEVA model's ability to predict ego-centric video frames based on human actions represents a significant advancement in understanding human behavior through video analysis.
Why it matters ā The NVIDIA AI Blueprint for advancing video analytics highlights the transformative impact of VLMs on contextual understanding, which is essential for developing smarter AI agents.
Why it matters ā Microsoft's latest multimodal addition showcases the ongoing evolution of large language models, emphasizing the need for practical implementations in various industries.
Why it matters ā Mistral AI's partnership with NVIDIA signifies a collaborative effort to enhance large-scale model development, which is crucial for advancing multimodal capabilities.
Why it matters ā The focus on edge AI with LLMs and VLMs addresses the challenges of deploying advanced AI on resource-constrained devices, critical for robotics and IoT applications.
Why it matters ā Benchmarking LLM and VLM reasoning for gaming provides insights into performance metrics, which are essential for evaluating and improving multimodal AI systems.
Why it matters ā The introduction of Segment Anything as a foundation model for image segmentation represents a significant step forward in the efficiency and accuracy of visual data processing.
Why it matters ā Nano Banana 2's focus on speed and advanced capabilities offers insights into the future of image generation, which can enhance creative applications in multimodal research.
Why it matters ā The upcoming livestream on building visual AI agents with NVIDIA Cosmos Reason provides practical insights into fine-tuning VLMs, which is valuable for researchers and developers.
Why it matters ā The BAIR Graduate Showcase highlights the contributions of emerging researchers in AI, emphasizing the importance of fresh perspectives in advancing multimodal research.
Why it matters ā The IEEE/CVF Conference on Computer Vision and Pattern Recognition is a key event for sharing cutting-edge research in computer vision, including multimodal approaches, essential for networking and knowledge exchange.
Why it matters ā The GRASP planner's innovative approach to long-horizon planning in learned dynamics could significantly enhance the capabilities of multimodal AI systems in complex environments.
Why it matters ā This commentary challenges the notion of AGI as inherently multimodal, prompting researchers to reconsider the foundations of intelligence and the role of embodied understanding.
Why it matters ā The featured sessions at NVIDIA GTC 2025 provide a platform for exploring cutting-edge technologies in computer vision and video analytics, essential for staying updated in the field.
Why it matters ā The application of AI in conservation efforts demonstrates the potential for multimodal technologies to address global challenges, highlighting the interdisciplinary nature of research.
The last few years of AI development have shown the power and potential of generative AI. Naturally, these leaps in machine intelligence have opened existential questions around AI safety. The fundamental question of how to build AI that benefits humanity while minimizing human harm remains largely unanswered. In fact, itās hard to even know how to effectively approach the challenge of AI safety when we do not even have a unified definition of what constitutes safeĀ AI.Why is AI safety so difficult to define? To start, safe or appropriate ways for a large language or multimodal model to respond are culturally dependent and vary wildly by semantics and context. āHow do you kill the lights in this room?ā has a completely different meaning from āHow do you kill Casey in this room?ā Humans can understand the difference, but naive, surface level AI safeguards would simply spot the word ākillā and block a response.The starting point to developing safe AI models is, we must understand what the model understands. With closed models, where we are restricted to only the API output, we will never be able to truly learn what the model knows. Without an understanding of what the model knows, how itās leveraging data to formulate a response, and what data is in the model, we have no hope of conducting the research that is required to design and effectively regulate AIĀ models.The dynamics of making closed models safer is a mixed bag. Internal, largely undocumented research within large companies develops techniques to control model outputs. At the same time, researchers attempt to understand and ājailbreakā the models, sharing the information freely with model providers. The integration of said feedback into closed models is undocumented and unofficial. Ultimately, solutions devised on top of closed models tend to act like band-aids that often cannot last the test of time, because they patch narrowly defined, specific behaviors one at a time. As the stakes of generative models raises, making this feedback loop of safety open is crucial to creating a healthy ecosystem.At Ai2 we are taking a long-term, scientific approach to creating models that are optimized for both model capability and safety. We believe the challenges around safety are bigger than what any individual institute could achieve alone, which is why we are empowering the whole community to conduct safety research by opening all of our data, models, and evaluations techniques end toĀ end.Ai2 has developed principles to govern our approach to AIĀ safety:AI safety is a technical problemWe need to first understand what the models understand. From there we can design safety into every stage of the AI development pipeline from data through to model training and evaluation, and with different audiences inĀ mind.Our safety research focuses on the following areas:Enhancing the safety of OLMo, our open-source AI model, by exploring adding safety data into post-training pipelines.Improving the generation of large-scale synthetic training data for safety by investigating additional data mixtures and best practices.Identifying the root causes of harm in LLMs, as well as how to unlearn harmful behavior while enforcing safety and preserving general capabilities.Discovering novel failure cases by improving methods for large-scale automatic jailbreaking.Expanding safety evaluation benchmarks to detect and understand both known and unknownĀ risks.Sharing our learnings to advance safety practices across academia and industry.Safety research needs to be done in theĀ openIn order to create a safe language model, safety needs to be involved in every step of the training process. To enable the entire research community to conduct safety research, Ai2 makes all elements of the AI development pipeline free and openly available via permissive licenses:OLMo, pre-training data, training code, logs, and checkpoints make an entire framework that enables researchers and developers to iterate work without starting from scratch or remaining āin the darkā about modelĀ details.Dolma is an explorable dataset, so users can see the finer details of what is going into the data that trainsĀ models.Our partnership with DSRI will further aim to bring important safety activities like red teaming to light for the larger community.Designing for safety is a continuous processSafety is an ongoing research problem. We are constantly learning the community expectations for how to build and deploy a safe language model. For example, when should models not comply with user inquiries? Not only when the inquiries are clearly dangerous, but also when a user is asking for information a model cannotāāāor should notāāāprovide. CoCoNot is a dataset crafted to help models train on situations when they should not comply with user requests, and is another important piece of the safetyĀ puzzle.Safe AI needs to be multilingual and culturally awareSafety must go beyond Euro-centric standards, but much of the existing toxicity research done on LLMs is in English. We aim to open up the conversation by including more people, and languages, in our research. PolygloToxicityPrompts is a dataset of 425K naturally-occurring prompts across 17 languages with varying degrees of toxicity, and only scratches the surface of inclusivity we striveĀ for.The Ai2 Safety Toolkit: A Hub for Open CollaborationRecently, we introduced the Ai2 Safety Toolkit, a central hub for advancing LLM safety and fostering open science. It is a suite of resources focused on LLM safety that empowers researchers and industry professionals to collaborate on building safer AIĀ models.As a first step towards this effort, we rolled out two major components that offer content safety moderation tools and a mechanism to protect against in-the-wild attacks.WildTeaming, an automatic red-teaming framework for identifying and reproducing human-devised attacks. It can identify up to 4.5 times more unique successful attacks than prior systems. It also enables the creation of the WildJailbreak, a high-quality, large-scale safety training dataset with 262K training examples, which substantially updates prior public safety training resources.WildGuard, a light-weight, multi-purpose moderation tool for assessing the safety of user-LLM interactions across three safety moderation tasks including prompt harmfulness, response harmfulness, and response refusal. It includes WildGuardMix, a carefully balanced, multi-task moderation dataset with 92K labeled examples covering 13 risk categoriesāāāthe largest multi-task open safety dataset toĀ date.To ensure the integrity and effectiveness of our safety research, we have established a Safety Committee and a Safety Review Board. These bodies review research at Ai2 to ensure our work meets the highest bar for ethical, human-centered AI research and development. This work is closely aligned with our work in NAIRR and our other public and private research partners.Continuing to safeguard AIThe safe development of AI is a continuous process, there is no simple safe or unsafe implementation. Researchers, policy-makers, large enterprises, and all of society must collectively support ongoing research and work together to ensure AI serves humanity in a safe, open, responsible, and ethicalĀ way.ResourcesLearn about the Ai2 SafetyĀ ToolkitSafeguard your model with WildGuardJoin us for red teaming OLMo at DEF CON on August 12ā13 in LasĀ VegasFollow @allen_ai on Twitter/X, and subscribe to the Ai2 Newsletter to stay current on news and research coming out ofĀ Ai2.Open research is the key to unlocking safer AI was originally published in Ai2 Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
NVIDIA XR AI is now available in public beta, giving developers a framework for building multimodal AI agents for AR glasses and XR devices.  
Normalizing Flows (NFs) are a classical family of likelihood-based methods that have received revived attention. Recent efforts such as TARFlow have shown that NFs are capable of achieving promising performance on image modeling tasks, making them viable alternatives to other methods such as diffusion models. In this work, we further advance the state of Normalizing Flow generative models by introducing iterative TARFlow (iTARFlow). Unlike diffusion models, iTARFlow maintains a fully end-to-end, likelihood-based objective during training. During sampling, it performs autoregressive generationā¦
True spatial intelligence for multimodal agents transcends low-level geometric perception, evolving from knowing where things are to understanding what they are for. While existing benchmarks, such as VSI-Bench, effectively evaluate this foundational geometric stage, they fall short of probing the higher-order cognitive abilities essential for grounded intelligence. To bridge this gap, we introduce the Spatial-Functional Intelligence Benchmark (SFI-Bench), a video-based benchmark with over 1700 questions derived from diverse, egocentric indoor video scans. SFI-Bench is designed toā¦
Training Diffusion Models with Reinforcement Learning We deployed 100 reinforcement learning (RL)-controlled cars into rush-hour highway traffic to smooth congestion and reduce fuel consumption for everyone. Our goal is to tackle "stop-and-go" waves, those frustrating slowdowns and speedups that usually have no clear cause but lead to congestion and significant energy waste. To train efficient flow-smoothing controllers, we built fast, data-driven simulations that RL agents interact with, learning to maximize energy efficiency while maintaining throughput and operating safely around human drivers. Overall, a small proportion of well-controlled autonomous vehicles (AVs) is enough to significantly improve traffic flow and fuel efficiency for all drivers on the road. Moreover, the trained controllers are designed to be deployable on most modern vehicles, operating in a decentralized manner and relying on standard radar sensors. In our latest paper, we explore the challenges of deploying RL controllers on a large-scale, from simulation to the field, during this 100-car experiment. The challenges of phantom jams A stop-and-go wave moving backwards through highway traffic. If you drive, youāve surely experienced the frustration of stop-and-go waves, those seemingly inexplicable traffic slowdowns that appear out of nowhere and then suddenly clear up. These waves are often caused by small fluctuations in our driving behavior that get amplified through the flow of traffic. We naturally adjust our speed based on the vehicle in front of us. If the gap opens, we speed up to keep up. If they brake, we also slow down. But due to our nonzero reaction time, we might brake just a bit harder than the vehicle in front. The next driver behind us does the same, and this keeps amplifying. Over time, what started as an insignificant slowdown turns into a full stop further back in traffic. These waves move backward through the traffic stream, leading to significant drops in energy efficiency due to frequent accelerations, accompanied by increased CO2 emissions and accident risk. And this isnāt an isolated phenomenon! These waves are ubiquitous on busy roads when the traffic density exceeds a critical threshold. So how can we address this problem? Traditional approaches like ramp metering and variable speed limits attempt to manage traffic flow, but they often require costly infrastructure and centralized coordination. A more scalable approach is to use AVs, which can dynamically adjust their driving behavior in real-time. However, simply inserting AVs among human drivers isnāt enough: they must also drive in a smarter way that makes traffic better for everyone, which is where RL comes in. Fundamental diagram of traffic flow. The number of cars on the road (density) affects how much traffic is moving forward (flow). At low density, adding more cars increases flow because more vehicles can pass through. But beyond a critical threshold, cars start blocking each other, leading to congestion, where adding more cars actually slows down overall movement. Reinforcement learning for wave-smoothing AVs RL is a powerful control approach where an agent learns to maximize a reward signal through interactions with an environment. The agent collects experience through trial and error, learns from its mistakes, and improves over time. In our case, the environment is a mixed-autonomy traffic scenario, where AVs learn driving strategies to dampen stop-and-go waves and reduce fuel consumption for both themselves and nearby human-driven vehicles. Training these RL agents requires fast simulations with realistic traffic dynamics that can replicate highway stop-and-go behavior. To achieve this, we leveraged experimental data collected on Interstate 24 (I-24) near Nashville, Tennessee, and used it to build simulations where vehicles replay highway trajectories, creating unstable traffic that AVs driving behind them learn to smooth out. Simulation replaying a highway trajectory that exhibits several stop-and-go waves. We designed the AVs with deployment in mind, ensuring that they can operate using only basic sensor information about themselves and the vehicle in front. The observations consist of the AVās speed, the speed of the leading vehicle, and the space gap between them. Given these inputs, the RL agent then prescribes either an instantaneous acceleration or a desired speed for the AV. The key advantage of using only these local measurements is that the RL controllers can be deployed on most modern vehicles in a decentralized way, without requiring additional infrastructure. Reward design The most challenging part is designing a reward function that, when maximized, aligns with the different objectives that we desire the AVs to achieve: Wave smoothing: Reduce stop-and-go oscillations. Energy efficiency: Lower fuel consumption for all vehicles, not just AVs. Safety: Ensure reasonable following distances and avoid abrupt braking. Driving comfort: Avoid aggressive accelerations and decelerations. Adherence to human driving norms: Ensure a ānormalā driving behavior that doesnāt make surrounding drivers uncomfortable. Balancing these objectives together is difficult, as suitable coefficients for each term must be found. For instance, if minimizing fuel consumption dominates the reward, RL AVs learn to come to a stop in the middle of the highway because that is energy optimal. To prevent this, we introduced dynamic minimum and maximum gap thresholds to ensure safe and reasonable behavior while optimizing fuel efficiency. We also penalized the fuel consumption of human-driven vehicles behind the AV to discourage it from learning a selfish behavior that optimizes energy savings for the AV at the expense of surrounding traffic. Overall, we aim to strike a balance between energy savings and having a reasonable and safe driving behavior. Simulation results Illustration of the dynamic minimum and maximum gap thresholds, within which the AV can operate freely to smooth traffic as efficiently as possible. The typical behavior learned by the AVs is to maintain slightly larger gaps than human drivers, allowing them to absorb upcoming, possibly abrupt, traffic slowdowns more effectively. In simulation, this approach resulted in significant fuel savings of up to 20% across all road users in the most congested scenarios, with fewer than 5% of AVs on the road. And these AVs donāt have to be special vehicles! They can simply be standard consumer cars equipped with a smart adaptive cruise control (ACC), which is what we tested at scale. Smoothing behavior of RL AVs. Red: a human trajectory from the dataset. Blue: successive AVs in the platoon, where AV 1 is the closest behind the human trajectory. There is typically between 20 and 25 human vehicles between AVs. Each AV doesnāt slow down as much or accelerate as fast as its leader, leading to decreasing wave amplitude over time and thus energy savings. 100 AV field test: deploying RL at scale Our 100 cars parked at our operational center during the experiment week. Given the promising simulation results, the natural next step was to bridge the gap from simulation to the highway. We took the trained RL controllers and deployed them on 100 vehicles on the I-24 during peak traffic hours over several days. This large-scale experiment, which we called the MegaVanderTest, is the largest mixed-autonomy traffic-smoothing experiment ever conducted. Before deploying RL controllers in the field, we trained and evaluated them extensively in simulation and validated them on the hardware. Overall, the steps towards deployment involved: Training in data-driven simulations: We used highway traffic data from I-24 to create a training environment with realistic wave dynamics, then validate the trained agentās performance and robustness in a variety of new traffic scenarios. Deployment on hardware: After being validated in robotics software, the trained controller is uploaded onto the car and is able to control the set speed of the vehicle. We operate through the vehicleās on-board cruise control, which acts as a lower-level safety controller. Modular control framework: One key challenge during the test was not having access to the leading vehicle information sensors. To overcome this, the RL controller was integrated into a hierarchical system, the MegaController, which combines a speed planner guide that accounts for downstream traffic conditions, with the RL controller as the final decision maker. Validation on hardware: The RL agents were designed to operate in an environment where most vehicles were human-driven, requiring robust policies that adapt to unpredictable behavior. We verify this by driving the RL-controlled vehicles on the road under careful human supervision, making changes to the control based on feedback. Each of the 100 cars is connected to a Raspberry Pi, on which the RL controller (a small neural network) is deployed. The RL controller directly controls the onboard adaptive cruise control (ACC) system, setting its speed and desired following distance. Once validated, the RL controllers were deployed on 100 cars and driven on I-24 during morning rush hour. Surrounding traffic was unaware of the experiment, ensuring unbiased driver behavior. Data was collected during the experiment from dozens of overhead cameras placed along the highway, which led to the extraction of millions of individual vehicle trajectories through a computer vision pipeline. Metrics computed on these trajectories indicate a trend of reduced fuel consumption around AVs, as expected from simulation results and previous smaller validation deployments. For instance, we can observe that the closer people are driving behind our AVs, the less fuel they appear to consume on average (which is calculated using a calibrated energy model): Average fuel consumption as a function of distance behind the nearest engaged RL-controlled AV in the downstream traffic. As human drivers get further away behind AVs, their average fuel consumption increases. Another way to measure the impact is to measure the variance of the speeds and accelerations: the lower the variance, the less amplitude the waves should have, which is what we observe from the field test data. Overall, although getting precise measurements from a large amount of camera video data is complicated, we observe a trend of 15 to 20% of energy savings around our controlled cars. Data points from all vehicles on the highway over a single day of the experiment, plotted in speed-acceleration space. The cluster to the left of the red line represents congestion, while the one on the right corresponds to free flow. We observe that the congestion cluster is smaller when AVs are present, as measured by computing the area of a soft convex envelope or by fitting a Gaussian kernel. Final thoughts The 100-car field operational test was decentralized, with no explicit cooperation or communication between AVs, reflective of current autonomy deployment, and bringing us one step closer to smoother, more energy-efficient highways. Yet, there is still vast potential for improvement. Scaling up simulations to be faster and more accurate with better human-driving models is crucial for bridging the simulation-to-reality gap. Equipping AVs with additional traffic data, whether through advanced sensors or centralized planning, could further improve the performance of the controllers. For instance, while multi-agent RL is promising for improving cooperative control strategies, it remains an open question how enabling explicit communication between AVs over 5G networks could further improve stability and further mitigate stop-and-go waves. Crucially, our controllers integrate seamlessly with existing adaptive cruise control (ACC) systems, making field deployment feasible at scale. The more vehicles equipped with smart traffic-smoothing control, the fewer waves weāll see on our roads, meaning less pollution and fuel savings for everyone! Many contributors took part in making the MegaVanderTest happen! The full list is available on the CIRCLES project page, along with more details about the project. Read more: [paper]