📊 Datasets — Awesome Multimodal

The COCO dataset is a large-scale benchmark containing images with corresponding captions, used to evaluate the performance of models in understanding and generating fine-grained visual and textual concepts.

Visual Question Answering — open-ended questions about images requiring joint vision and language understanding.

The LIBERO benchmark is a dataset used to evaluate the efficiency and performance of Vision-Language-Action models in terms of inference cost and task success rates.

""" _HOMEPAGE = "https://shannon.cs.illinois.edu/DenotationGraph/" # TODO: Victor _LICENSE = "" _ANNOTATION_URL = "http://cs.stanford.edu/people/karpathy/deepimagesent/caption_datasets.zip" _FEATURES = datasets.Features( { "image": datasets.Image(), "filename": datasets.Value("string"), "imgid": datasets.Value("int32"), "sentids": [datasets.Value("int32")], "sentences": [ { "tokens": [datasets.Value("string")], "raw": datasets.Value("string"), "imgid": datasets.Value("int32"), "sentid": datasets.Value("int32") } ] } ) class Flickr30k(datasets.GeneratorBasedBuilder):

Large-scale Multi-modality Models Evaluation Suite Accelerating the development of large-scale multi-modality models (LMMs) with lmms-eval 🏠 Homepage | 📚 Documentation | 🤗 Huggingface Datasets This Dataset This is a formatted version of POPE. It is used in our lmms-eval pipeline to allow for one-click evaluations of large multi-modality models. @article{li2023evaluating, title={Evaluating object hallucination in large vision-language models}… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-encoder/POPE.

A compositional visual-question-answering benchmark generated from Visual Genome scene graphs.

The 'nuScenes' dataset is a comprehensive benchmark for autonomous driving that contains diverse sensor data and annotations from real-world driving scenarios, used to evaluate the performance of perception and planning algorithms in complex environments.

A massive multi-discipline multimodal benchmark of college-level questions requiring image understanding plus expert reasoning.

OK-VQA is a new dataset for visual question answering that requires methods which can draw upon outside knowledge to answer questions. - 14,055 open-ended questions - 5 ground truth answers per question - Manually filtered to ensure all questions require outside knowledge (e.g. from Wikipeida) - Reduced questions with most common answers to reduce dataset bias

The 'MME' dataset/benchmark is used to evaluate the performance of Multi-modal Large Language Models (MLLMs) in mitigating visual hallucination by comparing their generated responses against accurate visual cues from provided images.

Dataset Card for "refcoco" More Information needed

ImageNet-R This repo is made to facilitate the evaluation of various pretraining models. It's constructed from the source file provided by official implementation. Usage from datasets import load_dataset dataset = load_dataset('axiong/imagenet-r') Dataset Summary ImageNet-R(endition) contains art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video… See the full description on the dataset page: https://huggingface.co/datasets/axiong/imagenet-r.

Dataset Card for "MMBench" More Information needed

The 'R-2R' dataset is a benchmark used to evaluate Vision-and-Language Navigation (VLN) systems, containing diverse trajectories and instructions derived from real-world indoor environments.

This is the ScientificQA dataset by Saikh et al (2022). @article{10.1007/s00799-022-00329-y, author = {Saikh, Tanik and Ghosal, Tirthankar and Mittal, Amish and Ekbal, Asif and Bhattacharyya, Pushpak}, title = {ScienceQA: A Novel Resource for Question Answering on Scholarly Articles}, year = {2022}, journal = {Int. J. Digit. Libr.}, month = {sep} }

Dataset Card for "msr-vtt" More Information needed

Dataset Card for MathVista Dataset Description Paper Information Dataset Examples Leaderboard Dataset Usage Data Downloading Data Format Data Visualization Data Source Automatic Evaluation License Citation Dataset Description MathVista is a consolidated Mathematical reasoning benchmark within Visual contexts. It consists of three newly created datasets, IQTest, FunctionQA, and PaperQA, which address the missing visual domains and are tailored to evaluate logical… See the full description on the dataset page: https://huggingface.co/datasets/AI4Math/MathVista.

Dataset Card for VQA-RAD Dataset Description VQA-RAD is a dataset of question-answer pairs on radiology images. The dataset is intended to be used for training and testing Medical Visual Question Answering (VQA) systems. The dataset includes both open-ended questions and binary "yes/no" questions. The dataset is built from MedPix, which is a free open-access online database of medical images. The question-answer pairs were manually generated by a team of clinicians.… See the full description on the dataset page: https://huggingface.co/datasets/flaviagiammarino/vqa-rad.

Dataset arXiv Website Code VSI-Bench VSI-Bench-Debiased [!IMPORTANT] [Nov. 7, 2025] UPDATE: This Dataset has been updated to include a "Debiased" subset following the TsT Pruning Methodology Visual-Spatial Intelligence Benchmark (VSI-Bench & VSI-Bench-Debiased) This repository contains the visual spatial intelligence benchmark (VSI-Bench), introduced in Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces, and its… See the full description on the dataset page: https://huggingface.co/datasets/nyu-visionx/VSI-Bench.

The 'NExT-QA' dataset/benchmark is designed for evaluating Video Question Answering (VideoQA) systems by providing a structured framework for assessing their ability to identify critical moments in videos and reason about causal relationships to answer complex questions.

The 'CHAIR' dataset/benchmark is used to evaluate the performance of vision-language models by measuring their ability to generate accurate and coherent captions while minimizing hallucinations.

GenEval is a benchmark used to evaluate the performance of models in unified multimodal understanding and generation tasks.

The MIMIC-CXR dataset is a large collection of chest X-ray images and associated radiology reports used to evaluate and improve medical report generation models.

TextVQA requires models to read and reason about text in images to answer questions about them. Specifically, models need to incorporate a new modality of text present in the images and reason over it to answer TextVQA questions. TextVQA dataset contains 45,336 questions over 28,408 images from the OpenImages dataset.

Dataset Card for REVERIE Dataset Details Dataset Type: REVERIE is the first large-scale visual instruction-tuning dataset with ReflEctiVE RatIonalE annotations. REVERIE comprises 115k machine-generated reasoning instructions, each meticulously annotated with a corresponding pair of correct and confusing responses, alongside comprehensive rationales elucidating the justification behind the correctness or erroneousness of each response. Data Collection: REVERIE was… See the full description on the dataset page: https://huggingface.co/datasets/zjr2000/REVERIE.

A-OKVQA is a reasoning dataset used to evaluate the visual reasoning capabilities of vision-language models (VLMs) through complex question-answering tasks.

VQA-2.0 is a benchmark dataset that contains images paired with questions and answers, used to evaluate the performance of models in visual question answering tasks.

Dataset Card for ImageNet Dataset Summary ILSVRC 2012, commonly known as 'ImageNet' is an image dataset organized according to the WordNet hierarchy. Each meaningful concept in WordNet, possibly described by multiple words or word phrases, is called a "synonym set" or "synset". There are more than 100,000 synsets in WordNet, majority of them are nouns (80,000+). ImageNet aims to provide on average 1000 images to illustrate each synset. Images of each concept are… See the full description on the dataset page: https://huggingface.co/datasets/ILSVRC/imagenet-1k.

InfoSeek is a benchmark dataset used to evaluate knowledge-based visual question answering (KB-VQA) systems by providing relevant external documents for conditioning answer generation.

Visual Question Answering (VQA) is a benchmark that evaluates the ability of models to answer questions about images, integrating visual and textual understanding.

Large-scale Multi-modality Models Evaluation Suite Accelerating the development of large-scale multi-modality models (LMMs) with lmms-eval 🏠 Homepage | 📚 Documentation | 🤗 Huggingface Datasets This Dataset This is a formatted version of ChartQA. It is used in our lmms-eval pipeline to allow for one-click evaluations of large multi-modality models. @article{masry2022chartqa, title={ChartQA: A benchmark for question answering about charts with visual and… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-encoder/ChartQA.

Dataset Card for Conceptual Captions Dataset Summary Conceptual Captions is a dataset consisting of ~3.3M images annotated with captions. In contrast with the curated style of other image caption annotations, Conceptual Caption images and their raw descriptions are harvested from the web, and therefore represent a wider variety of styles. More precisely, the raw descriptions are harvested from the Alt-text HTML attribute associated with web images. To arrive at the… See the full description on the dataset page: https://huggingface.co/datasets/google-research-datasets/conceptual_captions.

The Natural Language for Visual Reasoning corpora are two language grounding datasets containing natural language sentences grounded in images. The task is to determine whether a sentence is true about a visual input. The data was collected through crowdsourcings, and solving the task requires reasoning about sets of objects, comparisons, and spatial relations. This includes two corpora: NLVR, with synthetically generated images, and NLVR2, which includes natural photographs.

Images densely annotated with objects, attributes, relationships, region descriptions, and QA.

Dataset Card for LongVideoBench Large multimodal models (LMMs) are handling increasingly longer and more complex inputs. However, few public benchmarks are available to assess these advancements. To address this, we introduce LongVideoBench, a question-answering benchmark with video-language interleaved inputs up to an hour long. It comprises 3,763 web-collected videos with subtitles across diverse themes, designed to evaluate LMMs on long-term multimodal understanding. The… See the full description on the dataset page: https://huggingface.co/datasets/longvideobench/LongVideoBench.

audiocaps HuggingFace mirror of official data repo.

The ActivityNet Captions dataset connects videos to a series of temporally annotated sentence descriptions. Each sentence covers an unique segment of the video, describing multiple events that occur. These events may occur over very long or short periods of time and are not limited in any capacity, allowing them to co-occur. On average, each of the 20k videos contains 3.65 temporally localized sentences, resulting in a total of 100k sentences. We find that the number of sentences per video follows a relatively normal distribution. Furthermore, as the video duration increases, the number of sentences also increases. Each sentence has an average length of 13.48 words, which is also normally distributed. You can find more details of the dataset under the ActivityNet Captions Dataset section, and under supplementary materials in the paper.

CLIP (Contrastive Language-Image Pretraining) is a dataset and model that contains paired images and text, used to evaluate multimodal alignment and preferences in vision-language tasks.

MMMU-Pro (A More Robust Multi-discipline Multimodal Understanding Benchmark) 🌐 Homepage | 🏆 Leaderboard | 🤗 Dataset | 🤗 Paper | 📖 arXiv | GitHub 🔔News 🛠️[2026-07-10] Fixed incorrect ground-truth answer labels. (validation_Design_15; validation_Art_Theory_4) 🛠️[2026-05-30] Fixed the option augmentation issue in Vision and Standard (10 options) settings. (validation_Diagnostics_and_Laboratory_Medicine_17) 🛠️[2025-03-08] Fixed mismatch between inner image… See the full description on the dataset page: https://huggingface.co/datasets/MMMU/MMMU_Pro.

The 'Charades-STA' dataset is a benchmark used to evaluate temporal reasoning in long videos by providing a collection of videos annotated with natural language queries and corresponding temporal segments.

MLVU is a benchmark dataset used to evaluate the performance of multimodal large language models (MLLMs) in video understanding tasks.

Dataset Card for "clavin" More Information needed

'SimplerEnv' is an 80-task benchmark designed to evaluate closed-loop control and high-level instruction understanding in vision-language-action models.

Dataset Info: SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering [ISBI 2021 oral] Project Page: click Corresponding Authors: Bo Liu, Xiao-Ming Wu Any questions, please contact us. Thank you! Modification: In the Huggingface Repo, we have changed the name of validate.json to validation.json to better display in the Dataset Card.

LLaVA is a pretraining dataset used to evaluate Vision-Language Models (VLMs) by providing image-text pairs in various languages.

The Multi-30K dataset contains image-caption pairs in multiple languages and is used to evaluate multimodal machine translation performance.

Dataset Card for MathVerse Dataset Description Paper Information Dataset Examples Leaderboard Citation Dataset Description The capabilities of Multi-modal Large Language Models (MLLMs) in visual math problem-solvingremain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To… See the full description on the dataset page: https://huggingface.co/datasets/AI4Math/MathVerse.

The 'AMBER' dataset/benchmark is used to evaluate the reliability of vision-language models by assessing their ability to generate consistent and accurate responses to queries.

CLEVR is a dataset designed to evaluate the ability of models to perform visual reasoning and understand complex relationships in images through a series of structured questions and answers.

MMEB (Massive Multimodal Embedding Benchmark) is a comprehensive benchmark that includes 4 meta-tasks and 36 datasets designed to evaluate universal multimodal embedding models across various downstream tasks.

MVBench is a benchmark dataset used to evaluate the performance of vision-language models (VLMs) in tasks related to video understanding and temporal reasoning, specifically focusing on counting repetitions in video clips.

The VCR (Visual Commonsense Reasoning) dataset is used to evaluate a model's reasoning ability in understanding the semantics of visual content and natural language through tasks that require fine-grained visual and textual information.

Large-scale Multi-modality Models Evaluation Suite Accelerating the development of large-scale multi-modality models (LMMs) with lmms-eval 🏠 Homepage | 📚 Documentation | 🤗 Huggingface Datasets This Dataset This is a formatted version of DocVQA. It is used in our lmms-eval pipeline to allow for one-click evaluations of large multi-modality models. @article{mathew2020docvqa, title={DocVQA: A Dataset for VQA on Document Images. CoRR abs/2007.00398 (2020)}… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-encoder/DocVQA.

About DiDeMo contains 10K long-form videos from Flickr. For each video, ~4 short sentences are annotated in temporal order. We follow the existing works to concatenate those short sentences and evaluate ‘paragraph-to-video’ retrieval on this benchmark. We adopt the official split: Train: 8,395 videos, 8,395 captions (concatenate from 33,005 short captions) Val: 1,065 videos, 1,065 captions (concatenate from 4,290 short captions) (We don't have the collection yet.) Test: 1… See the full description on the dataset page: https://huggingface.co/datasets/friedrichor/DiDeMo.

SNLI-VE is the dataset proposed for the Visual Entailment (VE) task investigated in Visual Entailment Task for Visually-Grounded Language Learning accpeted to NeurIPS 2018 ViGIL workshop). SNLI-VE is built on top of SNLI and Flickr30K. The problem that VE is trying to solve is to reason about the relationship between an image premise Pimage and a text hypothesis Htext. Specifically, given an image as premise, and a natural language sentence as hypothesis, three labels (entailment, neutral and contradiction) are assigned based on the relationship conveyed by the (Pimage, Htext) entailment holds if there is enough evidence in Pimage to conclude that Htext is true. contradiction holds if there is enough evidence in Pimage to conclude that Htext is false. Otherwise, the relationship is neutral, implying the evidence in Pimage is insufficient to draw a conclusion about Htext.

The VLN-CE dataset/benchmark contains a collection of tasks for evaluating Vision-and-Language Navigation models by assessing their ability to navigate in environments using language instructions and spatial representations.

The 'RxR' dataset is a benchmark used to evaluate Vision-Language Navigation (VLN) tasks, containing navigation instructions paired with visual data to assess the performance of models in understanding and executing these instructions.

UCF-101 is a benchmark dataset that contains 13,320 videos across 101 action categories, used to evaluate fine-grained video reasoning and action recognition in models.