Rhapsody: A Dataset for Highlight Detection in Podcasts

[R1] While the paper demonstrates that audio features improve performance, it lacks sufficient justification.

To strengthen our argument that audio features improve performance, we additionally explore HuBERT embedding (https://huggingface.co/facebook/hubert-base-ls960) [1]. Specifically, we take the embeddings at the 10th layer and average pool them over time. Please refer to the performance difference in the following table:

As shown in this table, our approach benefits from both VDA and HuBERT embeddings. Notably, the system with HuBERT embedding performs the best among all the proposed systems, while having a smaller number of predicted highlights than others (3.73). We will add this experimental result to the updated manuscript. Thank you for the suggestion!

[R2] The paper's writing could be improved.

Thank you for your suggestions, we will make edits in the draft accordingly:

• (1) We will switch the order of the first and second items of the contributions list.
• (2) We will strengthen the motivation by adding discussions on use cases.
• (3) We will not use the term 'feature-agnostic' and refer to the methods directly.
• (4) The processing steps in Section 3.1 apply to all systems in Section 3.2, and we thus separate the subsections in the current way.
• (5) We will make the distinction between baselines (random/frequency-based) and non-baselines clearer.

[R3a] Figure 4's purpose and interpretation are unclear - why is it important that models predict similar numbers regardless of |G|?

We pointed out the fact that baseline models predict roughly the same number of highlights regardless of |G| to emphasize the clear upward trend, which means the better alignment of our proposed approach.

[R3b] The transition between discussing quantity (Table 3, Figure 4) and performance metrics (Figure 5) lacks coherence.

We think your suggestion is reasonable. We discuss performance metrics first (Table 2), then quantity metrics (Table 3, Figure 4), and then performance metrics again (Figure 5). We will switch the order of the last two items to have better coherence.

[1] Hsu, Wei-Ning, et al. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM Transactions on Audio, Speech, and Language Processing 2021

The response looks good, thanks for adding the experiment. Looking forward to the revised version.

[R1a] Extractive summarization approach

We agree that extractive summarization methods could generate better text features for detecting highlights. However, we used abstractive summarization rather than extractive summarization due to the noise of raw transcripts. For instance, they don’t provide any delimitation between the interlocutors or the time information between utterances. We used LLM-based abstractive summarization to clean the noise and extract important information at the same time. In summary, the format of the data makes the use of an extractive approach challenging. We will add this discussion to the final paper.

[R1b] Different podcast formats (e.g., interviews, solo speaker, storytelling) may require different definitions and strategies for highlight detection.

As you mentioned, there is a variation in the format of podcasts, and the best strategy for detecting highlights might be different depending on the format of the podcasts. Future work could explore dynamic inference strategies, such as prompt tuning, to better adapt to the podcast format. We will mention your suggestion as one of the future directions in the conclusion section.

[R2] The paper does not evaluate how genre affects model performance.

For your better understanding of the result, we report the hit rate of the models depending on the podcast genre:

Finetuned Llama models perform the best, except for the science & technology genre. Notably, they outperform other models in the comedy genre by a huge gap. It is also interesting that the finetuned Llama model without audio signals performs better than the one with audio signals in the sports genre. In general, LM-based methods don’t perform as well in the news & politics genre as the other genres.

As this analysis provides interesting findings, we will add this table and the discussion in the camera-ready version of the paper.

[R3] Collection and use of creator-written descriptions or summaries

We collect and plan to release YouTube metadata of podcasts in the dataset, and the metadata includes the video’s description on YouTube. We have considered using the video description, but decided not to use them since we noticed that descriptions on YouTube often include unrelated content, such as advertisements and links to the creator’s SNS, unlike descriptions in the Spotify podcast dataset. That being said, we agree that it is unclear in our writing what data was collected for each video and what items are available in the metadata. We will make edits in the final draft to make it clearer.

[R4] Handling of non-content segments (e.g., ads, intros, boilerplate)

We agree that there might exist extraneous content in podcasts. However, we believe that LMs should ideally consider those factors and correctly distinguish between extraneous and non-extraneous content. Furthermore, our approach is capable of handling those segments, as LLMs are asked to summarize raw transcripts while focusing on conveying the overall context without unnecessary detail (Section 3.1 and Appendix C.1). If a segment contains only advertisements, it is unlikely to be one of the ground-truth highlights, as the labels are aggregated over 50k viewers.

[R5] Human evaluation of predicted highlights

A user study about the alignment between human perception and replay peaks has already been done in the literature [1]. To quote their results (Table 5 in [1]), the annotators agreed with 64.8% of the labels. Considering that we even have stricter criteria for highlights, we expect our agreement rate to be higher.

[1] Sul, Jinhwan, et al. Mr. HiSum: A Large-scale Dataset for Video Highlight Detection and Summarization. NeurIPS 2023

Thanks for the reponse. I agree with "preprocessing techniques", and as mentioned in my first comments that technique already exists in the literature [1]. I suggest the authors to add this note to discussion/limitation of the work.

[1] Reddy, S., Yu, Y., Pappu, A., Sivaraman, A., Rezapour, R., & Jones, R. (2021, April). Detecting Extraneous Content in Podcasts. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (pp. 1166-1173).

We sincerely thank the reviewer for their insightful and valuable comments. Please see our responses below:

[R1] The proposed bias correction mechanism could over-correct.

Thanks for raising this point. We were also worried about it at first. We clarify that the first segment can still be identified as the highlight after the correction. As shown in Figure 7, the highlight selection rate of the first segment is adjusted similarly to that of other segments after the correction, indicating the effectiveness of our mechanism.

[R2] Errors in automatic transcription and V/A/D extraction could artificially inflate the complexity of this task.

We agree that errors in automatic transcription and audio feature extraction might affect the model's performance. However, these errors don’t affect the complexity of this task, since we formulate the task as predicting highlights given only the audio track of a podcast episode (Section 2.1). In Section 3.1, we report the best option of what we’ve explored for input feature processing. Future works can explore better, error-free input feature processing techniques, as well as better prompts, or different models.

[R3] It is not clear that the proposed formulation is the best one.

It is certainly possible to solve the proposed task (highlight detection from a long-form podcast audio clip) in many ways, including the ones you propose. We do not claim that our task formulation is the best way to set up the task, but it is a reasonable one. We will discuss alternative formulations (like the one you suggested) in the paper.

[Q1] Did the authors consider using visual information as well?

In this work, we focus on whether the current LMs are capable of finding interesting parts given text or audio. As stated in lines 84-86, we filter out videos in which visual information is important. Although we developed our filtering criteria by manual inspection of our dataset, the dataset may include a few videos where visual information matters, given that our dataset contains 13K podcasts.

[Q2] Did the authors try this task with audio only?

We used Gemini with both text and audio, which is a superset of using just the audio for the task. Therefore, we expect that the audio-only version will underperform the text+audio version, as it’s harder to extract semantic information from the audio signal.

[Q3] What happens if peaks occur at the boundary of segments?

A peak can’t occur at the boundary of segments, since one data point in a replay graph corresponds to a single segment. Please refer to Section 2.1 for detailed information about the replay graph.

[Q4] Why not use the raw transcript segments rather than their summaries?

From our initial experiments, we observed that providing raw transcripts to the models results in worse performance. There might be two major reasons for this. First, raw transcripts don’t provide any delimitation between the interlocutors, which is critical for understanding the content based on text. Second, as shown in Table 1, raw transcripts have ~4.7K words in total, and providing transcripts without any preprocessing results in an extremely long prompt, which can be harmful for the model’s performance.

[Q5] Do the authors have the legal permissions to redistribute the most replayed feature?

Thank you for pointing out an important issue. As stated in the ethics statement, there have been many datasets and benchmarks that include YouTube features, such as video metadata [1,2], transcript [3,4], uploader-generated chapters [4], and replay graphs [5]. To further respect the rights of creators, we plan to release only features that are necessary for reproduction.

[1] Zellers, Rowan, et al. MERLOT: Multimodal Neural Script Knowledge Models. NeurIPS 2021

[2] Zellers, Rowan, et al. MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound. CVPR 2022

[3] Han, Seungju, et al. CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos. ICCV 2023

[4] Yang, Antoine, et al. VidChapters-7M: Video Chapters at Scale. NeurIPS 2023

[5] Sul, Jinhwan, et al. Mr. HiSum: A Large-scale Dataset for Video Highlight Detection and Summarization. NeurIPS 2023

We thank the reviewer for their insightful comments and questions. Please see the answers to your questions below:

[Q1a] How does podcast duration affect model performance?

For your better understanding, we report the performance of the models depending on the podcast duration: https://ibb.co/TBLYRDfj (hit rate), https://ibb.co/Xxjct3j9 (precision)

In this figure, we grouped podcast episodes by rounding up the duration to the nearest 10 minutes. I.e., podcasts with a duration of up to 10 minutes belong to datapoint at $x=10$, podcasts with a duration of 10 to 20 minutes belong to datapoint at $x=20$, and so on. We do not see a clear trend, most systems perform similarly regardless of the podcast duration.

[Q1b] Is there any additional structure in the podcast itself that might point to varying error rates by segment number?

From the left graph in Figure 7, we can observe mainly two tendencies: (1) the first value is significantly higher than the rest, and (2) values at the end (segment 90-100) are a little bit lower than preceding values. We decided not to consider the second tendency, since the trend is way weaker than the first one. However, we agree that investigating the internal structure in replay graphs is an interesting and important future research direction, since it is related to people’s podcast listening behavior.

[Q2] Stronger prompting designs (e.g., informing the model about what the median number of highlights is)

As you pointed out, there might be a stronger prompt than the one we used. The reason why we didn’t provide the model with some statistics on $|G|$ is that we were trying to simulate the real-world inference scenario, where the statistics are not available. That being said, we conducted the initial experiments with a few different prompts and decided to include ‘Include a segment only if there is a highly obvious reason it might be replayed, excluding uncertain segments.’ in the prompt after we found this helps the models to output the appropriate number of predicted highlights.

[Q3] Error bars on Figures 4 and 5

We agree that the figures would be more informative if there were error bars. We will shade the area between $\pm1$ standard deviation around the mean in the camera-ready version of the paper.

[Q4] Update on literature

Thank you for letting us know about the literature that we weren’t aware of. We will revise the writing accordingly in the camera-ready version of the paper.