Zero-shot Audio Classification Using Image Embeddings

Abstract

Supervised learning methods can solve the given problem in the presence of a large set of labeled data. However, the acquisition of a dataset covering all the target classes typically requires manual labeling which is expensive and time-consuming. Zero-shot learning models are capable of classifying the unseen concepts by utilizing their semantic information. The present study introduces image embeddings as side information on zero-shot audio classification by using a nonlinear acoustic-semantic projection. We extract the semantic image representations from the Open Images dataset and evaluate the performance of the models on an audio subset of AudioSet using semantic information in different domains; image, audio, and textual. We demonstrate that the image embeddings can be used as semantic information to perform zero-shot audio classification. The experimental results show that the image and textual embeddings display similar performance both individually and together. We additionall

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keydogan2022zero

Related papers

• Avgzslnet: Audio-visual Generalized Zero-shot Learning By Reconstructing Label Features From Multi-modal Embeddings (2020)12.10
• On Class Separability Pitfalls In Audio-text Contrastive Zero-shot Learning (2024)3.58
• Improved Zero-shot Audio Tagging & Classification With Patchout Spectrogram Transformers (2022)0.00
• Multilingual Acoustic Word Embedding Models For Processing Zero-resource Languages (2020)8.09
• Improved Acoustic Word Embeddings For Zero-resource Languages Using Multilingual Transfer (2020)7.81
• Speech Enhancement With Zero-shot Model Selection (2020)7.81
• Learning Word Embeddings From Speech (2017)0.00
• Zero-shot Personalized Speech Enhancement Through Speaker-informed Model Selection (2021)7.16