AudioCaps

Overview edit

Variants: AudioCaps

Associated Benchmarks

This dataset is used in 4 benchmarks:

• Audio Generation - Metrics: FD_openl3, FAD, FD, KL_passt, IS, CLAP_LAION, CLAP_MS
• Audio captioning - Metrics: SPIDEr, CIDEr, SPICE, BLEU-4, METEOR, ROUGE-L, FENSE, SPIDEr-FL, #params (M), ROUGE, Sentence-BERT
• Target Sound Extraction - Metrics: SDRi, SI-SDRi
• Text to Audio Retrieval - Metrics: R@1, R@5, R@10

Recent Benchmark Submissions

Research Papers

Recent papers with results on this dataset:

• LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport (2025) -
• TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization (2024) -
• ETTA: Elucidating the Design Space of Text-to-Audio Models (2024) -
• Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning (2024) -
• SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs (2024) -
• EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance (2024) -
• Stable Audio Open (2024) -
• Taming Data and Transformers for Audio Generation (2024) -
• Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding (2024) -
• Improving Text-To-Audio Models with Synthetic Captions (2024) -
• Long-form music generation with latent diffusion (2024) -
• InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024) -
• CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction (2024) -
• Fast Timing-Conditioned Latent Audio Diffusion (2024) -
• EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning (2024) -
• Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation (2024) -
• Audiobox: Unified Audio Generation with Natural Language Prompts (2023) -
• ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation (2023) -
• Retrieval-Augmented Text-to-Audio Generation (2023) -
• AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining (2023) -