OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions

Rebuttal to Reviewer tzNS

We appreciate the reviewer for the positive comments and constructive suggestions. We are encouraged that the reviewer agrees that our work introduces a "new task", which is "more advanced than previous work", "new techniques", " a new, high-quality dataset" with "richly annotated conversational videos", "creating more natural and temporally aligned conversational responses."

Q1. “Metrics from SyncNet [R1], which is a more reliable indicator of lip‑sync accuracy, may be a better choice.”

We thank the reviewer for this suggestion. Our evaluation had indeed adopted metrics derived from SyncNet [R1]. In particular, [R1] introduces two metrics: LSE‑D (Lip Sync Error – Distance) and LSE‑C (Lip Sync Error – Confidence). We adopt LSE‑D in Tables 1 and 2 of the main paper, using the official SyncNet evaluation script [R1]. We did not use LSE‑C because it is not well‑suited for our task. LSE‑C, measuring audio‑visual correlation by averaging confidence scores computed based on SyncNet, is effective for talking‑head generation tasks where the subject speaks continuously. However, in our task modeling multimodal conversation, listeners are often silent when the speaker is talking. Although silent segments represent appropriate responses, evaluated confidence values drop sharply, making LSE‑C unreliable for our task. Follow your suggestion, we also provide LSE‑C results below, where our method outperforms baselines.

Similarly, since the listener often remains silent in multi‑turn conversations, our improvement in LSE‑D may seem small compared with LSTM, yet, it actually indicate a large performance gain in OMCRG task.

• Why we emphasised LSE‑D. LSE‑D measures the average temporal offset between the predicted video and the reference audio. In conversational footage, speakers alternate with silent listeners (rather than they are always speaking like talking head); during those silent spans SyncNet’s confidence drops sharply (indicate a low average value of LSE-C), making LSE‑C very noisy. We therefore used LSE‑D as the primary figure of merit. We also have rerun the official SyncNet script and report both metrics:

  Method	LSE‑D ↓	LSE‑C ↑
  LSTM	9.72	0.157
  Audio-visual LLM	10.03	0.269
  Ours	9.56	0.371

The 1.6% LSE‑D reduction confirms a significant improvement, and the gains in LSE‑C further reinforce our method’s effectiveness. We will add these numbers to our revised version and the supplementary materials.

• Context for the “marginal” gap. In a turn‑taking dialogue, perfect lip‑syncing is easier to approach because most frames are either silence or low‑motion listener responses. Even a small absolute reduction in LSE‑D therefore corresponds to noticeably fewer mis‑aligned phoneme–viseme pairs ($\approx$ 200 ms on average).

• Planned revision. We will (i) retain LSE‑D as our headline metric, (ii) include the full LSE‑C results in the revised version, and (iii) clarify in Sec. 5 that both metrics originate from SyncNet. We believe this addresses the reviewer’s concern while demonstrating the practical significance of our improvements.

[R1] Prajwal, K. R., et al. "A lip sync expert is all you need for speech to lip generation in the wild." Proceedings of the 28th ACM international conference on multimedia. 2020.

Q2. Human evaluation.

Thank you for your constructive suggestion. We conducted a user study with 49 participants (28 male, 21 female), all of whom were proficient in English (53.1% reported advanced proficiency or daily‐communication ability). Educational background was high: 95.9% held at least an undergraduate degree, 44.9% held a master’s degree, and 18.4% held a PhD. Age distribution was as follows: 14.3% under 25, 34.7% aged 26–35, 24.5% aged 36–45, and 26.5% aged 46–55. Each participant viewed 16 randomly ordered clips and indicated their preference in pairwise A/B comparisons across four perceptual criteria: speech content appropriateness, audio speech quality, visual quality, and audio–visual synchronization. Our method was preferred in every case, with a minimum preference rate of 67.3% (i.e., speech content appropriateness Ours vs. LSTM) and a maximum of 95.9% (synchronization Ours vs. Audio–Visual LLM). The table below summarizes these preference rates. We will include these human evaluation results in our final revision.

Table: Percentage of times participants preferred our method over each baseline in pairwise A/B tests.

Q3: The qualitative results in the supplementary materials.

Our OMCRG task is much more challenging than related tasks (e.g., talking‑head generation), since it requires simultaneously satisfying multiple requirements, such as (1) online multimodal generation (generate multi-model outputs simultaneously and iteratively, rather than offline single modal generation (generate one modality and the entire sequence at once), and (2) ensuring appropriateness and naturalness of the generated responses.

Moreover, lip–audio synchronization and online audio-video generation remain a challenging problem. For example, talking‑head generation works have devoted much effort to improving lip–audio synchronization, introducing modules/losses such as lip‑mask‑aware training with SyncNet supervision [R2] or pre-trained lip synchronization discriminator [R3].

However, as the first work on the new OMCRG task, we focus on presenting a simple, effective, and scalable framework, instead of introducing overly complex, subproblem‑specific modules for fine‑grained optimization.

Despite this, our approach outperforms baseline methods by a large margin. (e.g., FVD $\downarrow$ 314.9 vs. 681.6, UTMOS $\uparrow$ 1.41 vs. 1.32). Moreover, our framework provides a solid foundation for future improvements such as adding lip-mask-aware-training for improving lip–audio synchronization.

[R2] LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision, arXiv, 2025

[R3] SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis, CVPR 2024.

Q4. inference speed or real-time viability.

We measured the model’s inference speed at 15.6 FPS (64 ms delay) on a single NVIDIA A100 80GB GPU and will report these results in the revised manuscript. Importantly, this benchmark was obtained without any deployment or inference optimizations such as flash‑attention, multi‑GPU parallelism, distillation, or quantization, highlighting substantial potential for further acceleration and more efficient, real‑time deployment.

Q5. Typos

Thanks for your detailed review and catching these mistakes. We apologize for the typos: in line 144, “accumfd udfsalated” should read “accumulated. We have corrected these errors and will conduct a thorough review of the manuscript to eliminate any remaining typos in the final version.

Thank you for the clarifications. However, I have strong reservations about accepting this paper due to the poor lip-sync quality. This is a significant concern, as lip synchronization is a well-established problem with effective solutions developed over the past five years, such as the widely-used lip-sync loss [1].

Although your task is new, your baselines are weak. Your LSE-C value is too low and only surpasses weak baselines. I suggest you compare your LSE-C value with other talking-head papers, such as MakeItTalk (2020) [2] and SadTalker (2023) [3], as well as with real videos. They have well-designed inference codes. You can test only on the talking segments, as silent audio will negatively affect the LSE-C performance.

Additionally, the paper should include a discussion on the potential misuse of this technology and possible countermeasures [4,5,6].

I will reconsider my final score based on your response.

[1] Prajwal, K. R., et al. "A lip sync expert is all you need for speech to lip generation in the wild." Proceedings of the 28th ACM international conference on multimedia. 2020.

[2] Zhou, Yang, et al. "Makelttalk: speaker-aware talking-head animation." ACM Transactions On Graphics (TOG) 39.6 (2020): 1-15.

[3] Zhang, Wenxuan, et al. "Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[4] Salman, Hadi, et al. "Raising the cost of malicious ai-powered image editing." arXiv preprint arXiv:2302.06588 (2023).

[5] Gan, Yuan, et al. "Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[6] Xue, Haotian, et al. "Toward effective protection against diffusion-based mimicry through score distillation." The Twelfth International Conference on Learning Representations 2023.

Thank you for your effort and reply. You have persuaded me that this work serves as a valuable baseline for this new research direction. While the lip-sync quality is relatively low, these initial results provide a solid foundation for future studies. I strongly suggest you include more complete experimental results in your final revision and publish the code to ensure reproducibility and facilitate future work, as promised in your paper and rebuttal. Therefore, I have raised my score and will now recommend acceptance. Good luck!

PS: There are also more passive protection methods [1, 2] for detecting talking-head deepfakes, which would also be a valuable addition to the discussion on ethical considerations.

[1] Pei, Gan, et al. "Deepfake generation and detection: A benchmark and survey." arXiv preprint arXiv:2403.17881 (2024).

[2] Khan, Naseem, et al. "Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems." arXiv preprint arXiv:2507.21157 (2025).

Rebuttal to Reviewer c8bb

We appreciate the reviewer for the positive comments and constructive suggestions. We are encouraged that the reviewer agrees that our work is "a novel task", "the dataset contribution is also worth noting", "two novel components", "interesting methodological approach", and "thorough evaluation".

Q1.1: A more detailed analysis of the dataset's demographic diversity would be beneficial to understand potential biases

A: Thank you for your constructive suggestions. As suggested, we conduct additional analysis of the dataset's demographic diversity. As indicated in the table below, our dataset covers balanced gender, diverse age distributions, and individuals from multiple ethnic backgrounds. The properties of our dataset enable us to model varied conversational behaviors, serving as a valuable benchmark for future research.

Demographic statistics:

Q1.2: While ResponseNet is of high quality, its total duration is relatively small.

A: Due to data availability and costly data processing, building datasets for the OMCRG task is challenging. Unlike typical generation tasks, the OMCRG task requires dyadic high‑quality videos where both speaker and listener appear in the same frame, and they must stay in a static and consistent camera view for a long duration, which is scarce.

Instead of simply increasing data scale, recent studies have shown that high‑quality training data is critical for the performance of generative models. We hence perform costly, human‑curated processing to ensure high-quality, diverse, and representative data. For example, we curated diverse demographics, as well as six conversation genres covering (1) Intimate personal chats, (2) Economic interviews, (3) Interdisciplinary expert panels, (4) Educational tutoring, (5) News commentary, and (6) Casual small‑talk.

This careful curation allows us to introduce the first high-quality dataset ResponseNet for OMCRG. Despite the modest size of ResponseNet, its high quality and diversity, combined with our effective leverage of pre‑trained LLM, which has been pre-trained on large-scale text data (billions of text tokens), enable our method to achieve the best performance. Moreover, our pipeline incorporates powerful, pre-trained audio and visual feature extractors. The audio tokenizer has been trained on over 3,000 hours of diverse speech, supplying generalizable semantic and prosodic encodings. Facial expression and head-pose coefficients are extracted via Google’s MediaPipe framework, providing stable, high-fidelity visual features without additional training cost. These audio–visual–textual models furnish strong priors, eliminating the need for vast amounts of data to learn features for each modality.

Q2: The Chrono-Text Markup uses only two special tokens to align text and vision, which might be too coarse to model highly complex temporal dynamics.

A: Our Chrono‑Text Markup employs only two special tokens to balance temporal fidelity with learnability. Since OMCRG is a highly complex task with multiple requirements, introducing more and fine‑grained special tokens would significantly expand the LLM’s vocabulary and substantially increase training difficulty. An alternative approach is direct audio encoding and decoding with an audio‑based model. However, this introduces format inconsistencies (e.g., sampling rate) and large variations in tone color, pitch, and frequency, making training even more difficult.

In contrast, our Chrono‑Text Markup is simple yet effective, enabling the model to learn utterance start points, durations, and natural pauses without large complexity. We are unable to upload more video results due to the NeurIPS rebuttal policy. Yet, Fig. 5 in the main paper shows our method effectively handles overlapping speech. "Table 2 shows our method reaches a BERTScore of 0.806 and an FVD of 314.94, significantly surpassing the audio‑visual LLM (0.662, 681.55), demonstrating its effectiveness in modeling speech and non‑verbal temporal dynamics. Our method takes a step towards this new task, yet, we believe more elegant solutions could be proposed to better address complex temporal dynamics.

Q3: User study

A: Thanks for your constructive suggestion. We ran a user study with 49 participants (28 male, 21 female). Each subject watched 16 randomly‑ordered clips and assessed speech content appropriateness, audio speech quality, visual quality, and audio-visual synchronisation. All of them were proficient in English (53.1% reported advanced proficiency or daily‐communication ability). Educational background was high: 95.9% held at least an undergraduate degree, 44.9% held a master’s degree, and 18.4% held a PhD. Age distribution was as follows: 14.3% under 25, 34.7% aged 26–35, 24.5% aged 36–45, and 26.5% aged 46–55. In direct A/B preference, a minimum preference rate of 67.3% (i.e., speech content appropriateness Ours vs. LSTM) and a maximum of 95.9% (synchronization Ours vs. Audio–Visual LLM). We will add the full protocol and statistics in the revision.

Table: Percentage of times participants preferred our method over each baseline in pairwise A/B tests.

Q4: Typos and grammar

A: We appreciate you pointing out those spelling and grammar issues. We have corrected such typos and grammatical errors in the final version.

Thank you for your thorough review, positive comments, and insightful suggestions. We appreciate your recognition of OMCRG as a “novel task” that can “generate real‑time synchronized verbal and non‑verbal listener feedback,” and of our OmniResponse framework, which “outperforms baselines in semantic content, audio‑visual synchronization, and generation quality.” We are also grateful for your acknowledgement of ResponseNet as “high‑quality dyadic interactions with synchronized multimodal data.” Your view that this work is “laying the foundation for future human‑agent interaction research” is deeply encouraging, and we will continue refining the manuscript accordingly.

Q1. More detailed analysis of response delay and real-time processing latency

Thank you for your suggestion. As suggested, we conduct a more detailed analysis of response delay and real-time processing latency,

• Response delay. Our raw prototype, which includes preprocessing and postprocessing, runs at 15.6 FPS (≈64ms delay) on a single NVIDIA Tesla A100 80GB GPU. Note that such inference efficiency can be further improved, since it is obtained without any deployment or inference optimizations such as flash‑attention, multi‑GPU parallelism, distillation, or quantization.

• Processing latency. Our method efficiently processes the input stream, taking only 2.94 ms per frame to handle the video stream even on the CPU and achieving an average of 0.6 real‑time factor (RTF) for audio processing on the CPU.

Since our work is the first to explore the new OMCRG task, we mainly focus on building a high-quality dataset and providing a solid baseline framework for future research, rather than extensive latency optimizations. Yet, we believe that efficiency and latency can be substantially improved by incorporating effective techniques and insights from existing online methods, which we leave for future work.

We will includeadd these analysiis in in our final version.

Q2. Detailed elaboration on how the framework handles dynamic input streams, manages processing latency, and ensures responsive timing in natural conversational flows.

We thank the reviewer for this valuable comment and provide a more detailed explanation as follows:

• Handling dynamic input streams.

  • Visual stream. Each incoming frame is processed by MediaPipe, which extracts head‑pose and facial‑expression coefficients in 2.94 ms per frame even on the CPU.
  • Audio stream. Whisper‑large transcribes audio chunks at an average 0.6 real‑time factor (RTF) on the CPU, comfortably within live‑stream requirements.

  These stages compress the raw video and audio into compact token sequences, eliminating redundant frames and long waveforms before they are fed into the generation model.

• Managing processing latency. We constrain the context to a short sliding window, and compress earlier text into static embeddings, enabling incremental updates and avoiding the need to reprocess the full temporal context.

• Auto‑regressive multimodal generation. Our OmniResponse model handles the visual and text tokens in an auto‑regressive sliding window and enables efficient generation:

  • audio tokens is produced at 0.0069 RTF (≈ 145× real time) and converted to waveform at 0.0014 RTF (≈ 712× real time) during de‑tokenization.

  • textual response takes an average per-token latency of 2.2 ms (≈ 455 tokens/sec) on a single A100 80GB GPU.

  • visual response. The facial‑expression coefficients are rendered at >40 FPS even on a single RTX 2080 Ti.

These designs ensure responsive timing in natural conversational flows.

Rebuttal to Reviewer dTxZ

We sincerely thank the reviewers for their generous feedback. We are encouraged that the reviewer agrees that our work introduces "a novel task", "considers a real-world scenario: Online Multimodal Conversational Response Generation", proposes "a new high-quality dataset " to support OMCRG research, "the first online model", and the model "significantly outperforms baseline models".

Q1. The baselines in the main experiment could be more comprehensive (just see the GPT-series models)

A: Thanks for your suggestions. We have included the most powerful language-based conversation models (GPT-4o, GPT-4, and GPT-o1), and we also supplement more baselines:

• Qwen-7B-Chat,
• Gemini-2.5-Flash,
• Claude-Sonnet-4-20250514,
• DeepSeek-R1-200528,

and the results are listed below:

We will add these results in the revised version.

Q2. The auxiliary experiments lack depth.

A: Thank you for this valuable suggestion. In addition to the ablation study in Table 3, we carried out a comprehensive failure-case analysis to pinpoint where our model underperforms. Specifically, we found that our system seldom generates head rotations beyond ±40°, which can introduce noticeable distortions in side-view frames. To quantify this effect, we extracted side-face and front-face frames from our generated videos and measured realism via Fréchet Video Distance (FVD):

We also conducted a user study (with 49 participants) to assess human preferences in pairwise A/B tests:

In addition, we had built a baseline, namely, Audio-visual LLM, that omits the immediate text modality from our method in Tab. 2 of the main paper, to validate the effectiveness of adding text as an intermediate modality. The Audio-visual LLM baseline directly fuses visual and audio embeddings from both the speaker's and listener's past behavior to predict the listener’s next visual and audio tokens. Table 2 ( also shown below) demonstrates that introducing text as an intermediary modality significantly enhances the naturalness and realism of non-verbal responses, as indicated by the FD and FVD scores.

We will include these auxiliary experimental results into the final version by adding a latency study, a user study, error‑case analysis, and ablations across different LLM variants. We would also highlight that our primary contribution is task formulation, a first, end‑to‑end online generation model, an evaluation benchmark, and a high‑value dataset for this novel task.

Rebuttal to Reviewer bqe1

We sincerely thank the reviewer for the positive comments and constructive suggestions. In particular, we are delighted that you recognize our work as “a novel task”, that it “advances multimodal conversational AI for real-time human-computer interactions”, that it “positions it as a promising candidate for future human-computer interactions”, that the “proposed dataset proves to be highly beneficial”, and that our model “generates high-quality multimodal listener responses.” Your encouragement is greatly appreciated, and we will refine our manuscript carefully in accordance with your suggestions.

Q1: Dependency on text modality.

A: Thank you for your comments. We have conducted experiments to evaluate whether relying on text would constrain the richness of non-verbal communication (see Table 2 of the main paper and table below). That is, we build a baseline, namely w.o Text (i.e., Audio-visual LLM), which removes text modality from our method. We also provide another baseline, namely, LSTM-based method, employing widely-used network, i.e, a recurrent neural network to learn from only audio and visual modality.

Our OmniResponse significantly outperforms the w.o Text baseline across all evaluated metrics, including non-verbal ones. These results demonstrate that introducing text as an intermediary modality significantly enhances the naturalness and realistic of non-verbal responses, as indicated by the FD and FVD scores. Similarly, our method outperforms the LSTM-based approach by a considerable margin.

This superior performance is due to the combination of the intermediate text modality and our method's design:

• Text removes speaker-specific pitch and timbre, avoiding the negative effects by noise or variability in the audio while enabling the LLM to focus on intent understanding.
• Unlike raw speech, text does not affected by sampling rate or background audio noise, reducing the training difficulties for large-scale pre-training.
• We introduce the Vision Projection module and Vision Decoder, and conduct Multimodal Context Modeling, allowing our model to learn from both visual modality itself and cross-modal correlations.

We will add more discussions in the final version of our paper.

Q2: How to efficiently handle visual and textual history when processing long sequences and how to minimize latency in such scenarios.

A: Thank you for your comments. Our method handles long sequences and minimizes latency as follows:

• We limit temporal inputs to a sliding window spanning the most recent $\Delta T$ = 2.2s (from time $\tau$ to $t-1$). All tokens preceding $\tau$ are distilled into a static textual summary, far cheaper than keeping them as full temporal tokens, shrinking the context length.

• For the visual stream, only a short sequence of speaker‑listener face tokens is retained.

• With these strategies, our system sustains 15.6 FPS (≈ 64 ms end-to-end delay for audio plus facial animation) on a single NVIDIA A100 80 GB GPU.

To our knowledge, this is the first online multimodal conversational response generator. We outline additional efficiency improvements as future work, and the revised manuscript will detail the mechanisms that enable our pipeline to approach real-time operation.

Q3: Discussion on Failure cases.

A: Thank you for your suggestion. Follow your suggestion, we conduct a detailed analysis of failure cases. First, we observe that when the speaker’s speech contains large background noise, our method would wrongly generate listener responses. Second, our model rarely produces head rotations beyond approximately ±40°, and the resulting side-view frames can exhibit noticeable distortions. To quantify these effects, we extracted side-face and front-face frames from our generated videos and performed the following realism evaluation:

We will supplement these analyses and more failure case discussions in our revised version.

Q4. Typo.

A: Thanks for your suggestion. We apologize for the typos and have corrected these typos.

Dear Reviewer bqe1,

Thank you for your careful and thoughtful review. We sincerely appreciate the time and effort you have devoted to reviewing our paper. Your insightful feedback has been invaluable in sharpening our arguments, broadening our experiments, and refining the clarity of our manuscript.

We hope that our detailed explanations, additional results, failure cases provided, and expanded discussion have addressed your concerns fully. If you feel further experiments or information would be beneficial, we would be delighted to accommodate your suggestions.

Should our revisions meet your expectations, we would be most grateful if you could reflect this in your final evaluation. In any case, we deeply appreciate the time and expertise you have dedicated to improving our work.

Wishing you all the best and a pleasant remainder of your day.

With sincere thanks,

The Authors

Hi Reviewer bqe1,

The author–reviewer discussion phase will close soon. If you have not yet participated, please take this opportunity to review the rebuttal, check how your comments have been addressed, and share any remaining concerns with the authors. Your engagement in these final days is key to ensuring a thorough review process.

AC