🧰 Open Source Tools — Awesome Multimodal

Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience

The Open-Source Multimodal AI Agent Stack: Connecting Cutting-Edge AI Models and Agent Infra

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

☁️ Build multimodal AI applications with cloud-native stack

YC (S26) | AI that knows what you've seen, said, or heard. Records everything you do, say, hear 24/7, local, private, secure. Connect to OpenClaw, Hermes agent and 100+ apps

Janus-Series: Unified Multimodal Understanding and Generation Models

Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).

Agent S: an open agentic framework that uses computers like a human

Visualize, query, and stream to train on multimodal robotics data.

Production ready toolkit to run AI locally

[CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型

A Python library for anomaly detection across tabular, time series, graph, text, image, and audio data. 60+ detectors, benchmark-backed ADEngine orchestration, and an agentic workflow for AI agents.

Effortless data labeling with AI support from Segment Anything and other awesome models.

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

🔍大模型应用开发实战一：RAG 技术全栈指南，在线阅读地址：https://datawhalechina.github.io/all-in-rag/

Deeplake is AI Data Runtime for Agents. It provides serverless postgres with a multimodal datalake, enabling scalable retrieval and training.

Mobile-Agent: The Powerful GUI Agent Family

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

👀「大模型」2小时从0训练65M参数的视觉多模态VLM！Train a 65M-parameter VLM from scratch in just 2h!

A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.

The official repo of Qwen-VL (通义千问-VL) chat & pretrained large vision language model proposed by Alibaba Cloud.

The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/

This repository is a curated collection of links to various courses and resources about Artificial Intelligence (AI)

notes for software engineers getting up to speed on new AI developments. Serves as datastore for https://latent.space writing, and product brainstorming, but has cleaned up canonical references under the /Resources folder.

Open-source framework for building AI-powered apps in JavaScript, Go, and Python, built and used in production by Google

Solve Visual Understanding with Reinforced VLMs

Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.

A visual playground for agentic workflows: Iterate over your agents 10x faster

A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

SUPIR aims at developing Practical Algorithms for Photo-Realistic Image Restoration In the Wild. Our new online demo is also released at suppixel.ai.

A framework for efficient model inference with omni-modality models

MineContext is your proactive context-aware AI partner（Context-Engineering+ChatGPT Pulse）

A Next-Generation Training Engine Built for Ultra-Large MoE Models

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

开放源码的无App推送服务，iOS14+扫码即用。亦支持快应用/iOS和Mac客户端、Android客户端、自制设备

Align Anything: Training All-modality Model with Feedback

[NeurIPS 2025] SpatialLM: Training Large Language Models for Structured Indoor Modeling

Plug in and Play Implementation of Tree of Thoughts: Deliberate Problem Solving with Large Language Models that Elevates Model Reasoning by atleast 70%

Curated tutorials and resources for Large Language Models, AI Painting, and more.

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

DeepSeek-VL: Towards Real-World Vision-Language Understanding

Fengshenbang-LM(封神榜大模型)是IDEA研究院认知计算与自然语言研究中心主导的大模型开源体系，成为中文AIGC和认知智能的基础设施。

OpenMMLab Pre-training Toolbox and Benchmark

中文nlp解决方案(大模型、数据、模型、训练、推理)

🪩 Create Disco Diffusion artworks in one line

MOSS‑TTS Family is an open‑source speech and sound generation model family from MOSI.AI and the OpenMOSS team. It is designed for high‑fidelity, high‑expressiveness, and complex real‑world scenarios, covering stable long‑form speech, multi‑speaker dialogue, voice/character design, environmental sound effects, and real‑time streaming TTS.

A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.

[CVPR'25] Official Implementations for Paper - MagicQuill: An Intelligent Interactive Image Editing System

From Chain-of-Thought prompting to OpenAI o1 and DeepSeek-R1 🍓

Code and models for ICML 2024 paper, NExT-GPT: Any-to-Any Multimodal Large Language Model

SimpleMem: Efficient Lifelong Memory for LLM Agents — Text & Multimodal

Open-source multimodal retrieval engine (Morphik Core). By Morphik — AI back office for skilled nursing & senior living (morphik.ai).

The official repo of MiniMax-Text-01 and MiniMax-VL-01, large-language-model & vision-language-model based on Linear Attention

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"