Using Llm-as-a-judge/jury To Advance Scalable, Clinically-validated Safety Evaluations Of Model Responses To Users Demonstrating Psychosis

Abstract

General-purpose Large Language Models (LLMs) are becoming widely adopted by people for mental health support. Yet emerging evidence suggests there are significant risks associated with high-frequency use, particularly for individuals suffering from psychosis, as LLMs may reinforce delusions and hallucinations. Existing evaluations of LLMs in mental health contexts are limited by a lack of clinical