You Don't Need A Personality Test To Know These Models Are Unreliable: Assessing The Reliability Of Large Language Models On Psychometric Instruments

Abstract

The versatility of Large Language Models (LLMs) on natural language understanding tasks has made them popular for research in social sciences. To properly understand the properties and innate personas of LLMs, researchers have performed studies that involve using prompts in the form of questions that ask LLMs about particular opinions. In this study, we take a cautionary step back and examine whet