RAG Playground: A Framework For Systematic Evaluation Of Retrieval Strategies And Prompt Engineering In RAG Systems

Abstract

We present RAG Playground, an open-source framework for systematic evaluation of Retrieval-Augmented Generation (RAG) systems. The framework implements and compares three retrieval approaches: naive vector search, reranking, and hybrid vector-keyword search, combined with ReAct agents using different prompting strategies. We introduce a comprehensive evaluation framework with novel metrics and provide empirical results comparing different language models (Llama 3.1 and Qwen 2.5) across various retrieval configurations. Our experiments demonstrate significant performance improvements through hybrid search methods and structured self-evaluation prompting, achieving up to 72.7% pass rate on our multi-metric evaluation framework. The results also highlight the importance of prompt engineering in RAG systems, with our custom-prompted agents showing consistent improvements in retrieval accuracy and response quality.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypapadimitriou2024rag

Related papers

• Ragsmith: A Framework For Finding The Optimal Composition Of Retrieval-augmented Generation Methods Across Datasets (2025)0.00
• Optimizing Retrieval For RAG Via Reinforcement Learning (2025)0.00
• Ragperf: An End-to-end Benchmarking Framework For Retrieval-augmented Generation Systems (2026)0.00
• Rag-check: Evaluating Multimodal Retrieval Augmented Generation Performance (2025)0.00
• Are We On The Right Way For Assessing Document Retrieval-augmented Generation? (2025)0.00
• Hetarag: Hybrid Deep Retrieval-augmented Generation Across Heterogeneous Data Stores (2025)3.27
• SRAG: RAG With Structured Data Improves Vector Retrieval (2026)0.00
• Neurosymbolic Retrievers For Retrieval-augmented Generation (2026)0.00