Investigating The Utility Of Surprisal From Large Language Models For Speech Synthesis Prosody

Abstract

This paper investigates the use of word surprisal, a measure of the predictability of a word in a given context, as a feature to aid speech synthesis prosody. We explore how word surprisal extracted from large language models (LLMs) correlates with word prominence, a signal-based measure of the salience of a word in a given discourse. We also examine how context length and LLM size affect the results, and how a speech synthesizer conditioned with surprisal values compares with a baseline system. To evaluate these factors, we conducted experiments using a large corpus of English text and LLMs of varying sizes. Our results show that word surprisal and word prominence are moderately correlated, suggesting that they capture related but distinct aspects of language use. We find that length of context and size of the LLM impact the correlations, but not in the direction anticipated, with longer contexts and larger LLMs generally underpredicting prominent words in a nearly linear manner. We d

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keykakouros2023investigating

Related papers

• Boosting Large Language Model For Speech Synthesis: An Empirical Study (2023)6.77
• An Empirical Study Of Speech Language Models For Prompt-conditioned Speech Synthesis (2024)0.00
• Paralinguistics-aware Speech-empowered Large Language Models For Natural Conversation (2024)3.96
• Context And System Fusion In Post-asr Emotion Recognition With Large Language Models (2024)0.00
• Recent Advances In Speech Language Models: A Survey (2024)14.64
• Speechgen: Unlocking The Generative Power Of Speech Language Models With Prompts (2023)0.00
• Prompting Large Language Models With Audio For General-purpose Speech Summarization (2024)6.34
• A Survey On Speech Large Language Models For Understanding (2024)4.52