Hear you are: Teaching LLMs Spatial Reasoning with Vision and Spatial Sound

We sincerely appreciate your suggestion to analyze the question answering setup. This encouraged us to closely examine our framework and provide a clearer breakdown of its components.

W1

We acknowledge your concern about potential confounding factors in QA-based evaluation, such as failures stemming from misinterpreting inputs or cross-modal reasoning.

To address these concerns, we provide two diagnostic tables.

Modality Understanding:

The first table demonstrates that the model is able to process each modality independently and align them within the same latent space as the LLM, ensuring that the encoders generate meaningful and well-structured representations.

As we mentioned in the paper, to ensure the effectiveness of each modality-specific representation, the audio and visual encoders, along with their respective projectors, are pretrained in a unimodal setting using a large language model. For details, please refer to line 208 in the paper.

Performance metrics of Audio and Vision modalities

After the warm start of the encoders, each modality encoder performs reasonably well on detection and localization within its own modality.

LLM Reasoning: The second table evaluates the LLM’s ability to answer questions when given perfect modality representations. This isolates its reasoning and language capabilities from perception-related errors. We follow Oracle experiment of BAT[52].

All metrics, except the 3-field classification in Q3, the DoA accuracy in Q4, and the DoA accuracy in Q7,8, show almost perfect results. The relatively lower performance on Q3 and Q4 seems to stem from the limited mathematical ability of the LLM we used, as these questions require calculating spatial relationships between two objects and the receiver. In Q7,8, the DoA accuracy is affected by the presence of a non-sounding object that belongs to the same semantic category as the sounding object in prompt.

Oracle performance (O+Q)

These analyses help validate whether the modality understanding and question-answering components function as intended. Most of the time, reasoning is not the bottleneck. Errors tend to arise either from perception or from failures in aligning modality-specific representations with the LLM’s reasoning capabilities.

Large language models and question answering datasets allows new question types or answer categories to be added with minimal cost, often without the need to retrain the entire model. In contrast, classifier-based approaches require predefined categories and typically demand full retraining to accommodate new types. By leveraging the generative nature of language, we can explore broader and more realistic evaluation scenarios beyond the constraints of closed-form classification.

We will incorporate the aforementioned experiments into the paper to encourage more systematic analysis of each component involved in multimodal question answering.

W2

We appreciate the suggestion to include this relevant work. In our paper, we discuss an earlier pioneer work on sound localization [37], as well as several more recent approaches [21,7,23,42,29,39,43]. Nonetheless, we will include this reference in the revised version for completeness.

W3

We agree that reporting standard errors adds confidence to the findings. Due to limited time and GPU resources, we repeated the two most important experiments, R+B+Q and R+M+Q. As shown, the performance differences are minimal. However, we would like to highlight that metrics which rely on spatial audio consistently show clear differences between the two settings across trials (e.g., Q3-inv 3-field, Q4-inv DoA, Q6-sound, Q8-DoA).

W4

We use SigLIP2 with the NaFLEX setting (Line 225), which allows flexible image resolutions and aspect ratios. This is crucial for our use of 360-degree panoramic images, which have a 4:1 width-to-height ratio. DINO assume square inputs, which distort our panoramic images. While we agree that DINO offers strong features, it is not compatible with our setting. SigLIP2 provides a better fit due to its geometric flexibility.

Q1 Answer

Please read rebuttal to yWdi, From the reviewers

Q2 Answer

We finetuned VideoLLaMA2 on our dataset, but did not finetune ISSL or ACL-SSL. They are originally trained on datasets which is source of audio clip of our dataset (VGGSound). Additionally, since ISSL and ACL-SSL use different visual encoders (ResNet-18 and CLIP, respectively), a perfectly fair comparison with our model (which uses SigLIP2) is not feasible. More details are provided in Section A.1.1 of the supplementary material.

We sincerely appreciate the insightful comments on real-world scenarios, multi-source spatial reasoning, and stitching artifacts. In particular, the suggestion regarding multi-source settings inspired an interesting analysis that enriched our understanding of the model's behavior.

Real-world Scenario Discussion

Please read rebuttal to yWdi

Multi-source Spatial Reasoning

Real environments often include overlapping sound sources, whereas our dataset focuses on single-source localization. BAT[52], which introduced the Spatial AST model, used the encoder for multi-source spatial reasoning. Since our model also adopts the same encoder, we expect it could handle multi-source settings if such examples are included during training. Due to time constraints, we only conducted a zero-shot evaluation, which showed some degradation but generally reasonable behavior. To investigate this further, we conducted zero-shot evaluations under three different multi-source scenarios:

1. Adding a second sound source with a different sound class at a different location from the original source. (v,v)
2. Duplicating an original sound source and putting the copy at a different location from the original. (x,v)
3. Adding a second sound source with a different sound class at the same location as the original source. (v,x)

Among these results, we would like to highlight the metrics that show significant differences across experimental settings.

Classification accuracies (e.g., Q1, Q7, Q8, Q9): These results are relatively trivial and do not reveal substantial variation.

Q1 – Coming-from Accuracy:

The “Same Sound, Different Location (x,v)” setting shows the highest performance. This is because the model does not fail in sound classification, and the spatial alignment between the original sound source and the corresponding visual object is preserved without contamination. The better performance of the “Different Sound, Same Location (v,x)” setting compared to the “Different Sound, Different Location (v,v)” setting is also reasonable. Even when classification is incorrect, the model can still localize the sound effectively, and the visual encoder can correctly recognize the object at that location.

Q4 – DoA Accuracy:

DoA performance unsurprisingly degrades under “Different Location” settings, (x,v), (v,v). The performance gap becomes larger when the visual modality is unavailable. This highlights the importance of visual input in supporting spatial localization.

Q6 – Sounding Accuracy:

This metric is relatively straightforward. The question requires spatial audio understanding to identify which visual object is producing the sound. “Different location (x,v), (v,v)” introduces interference, and the presence of a second sound from the same category (x,v) adds further confusion.

Q7 vs. Q8: In Q7, the settings with both “Same sound, Different location” or “Different sound, same location” outperform the “Different sound, different location” setting. In contrast, in Q8, the “Same sound, Different location” condition performs worse, even though it achieves the highest classification accuracy. This discrepancy stems from differences in scene composition. Q7 contains only one visual object from the target category. Therefore, even with two audio sources from the same category, the level of confusion remains limited. Q8 includes two visual objects from the same category and two audio sources from the same category, which increases the complexity of spatial reasoning and makes disambiguation more difficult.

Q9 – Visibility Accuracy: As long as the model correctly classifies the object, visibility accuracy remains unaffected.

We are grateful for the insightful suggestion, which significantly improved the quality of our work.

Stitching Artifacts

While we could not conduct the blurring experiment within the rebuttal period, we respectfully refer you to Figures 4 and 5 of the supplementary material. These examples illustrate that spatial audio is still required for accurate localization, even in the presence of stitching artifacts. In both scenes, the candidate objects (e.g., two chickens or two electric blenders) belong to the same visual category and exhibit similar stitching cues (though these are not particularly salient to the human eye), making it difficult for the model to rely on low-level visual artifacts alone. This suggests that the model needs spatial audio cues to correctly disambiguate the true source. Additionally, the cat next to the chicken on the right side appears to have similar stitching cues, and semantic understanding is still required to distinguish the cat from the chicken despite the presence of such visual seams.

Thank you for addressing most of my concerns and including the additional experiments. The multi-source experiment is a great demonstration. Please make sure to include it in the revised version.

After reading the rebuttals, I would like to keep my original rating. What I really wish is that you could provide real-world evaluations, which I think are crucial for evaluating generalization. You may consider evaluating in-the-wild datasets like StereoWalks [1] in the future.

[1] Min et al. Supervising Sound Localization by In-the-wild Egomotion. CVPR 2025.

We truly appreciate your thoughtful questions on the real-world scenario, the precision of spatial localization from binaural audio alone, comparisons with stronger LLMs, and the projector design. In particular, your question on stronger LLMs inspired us to explore future opportunities.

Real-world Scenario Discussion

Please read rebuttal to yWdi

How the model produces such fine spatial coordinates from binaural audio alone?

We apologize for the confusion. Our model does not achieve such high precision; in our case, both training and evaluation are performed with 0.1-meter precision. As we are building a benchmark, we annotated the dataset with coordinates up to three decimal places to support more precise localization in future models.

Comparison with stronger LLMs

Regarding the stronger baseline, we appreciate your suggestion. While we were unable to implement the Qwen-Omni-based Hear You Are LLM due to time constraints, we appreciate your understanding. Nevertheless, we would like to outline the expected implementation of a Qwen-Omni-based Hear You Are LLM and discuss its feasibility.

First, Qwen2.5-Omni explicitly supports only mono audio, as stated in its official GitHub repository:

qwen-omni-utils/src/qwen_omni_utils/v2_5/audio_process.py, Line 50: raise ValueError("Support only mono audio")

In addition, its ViT-based vision architecture does not explicitly support the 4:1 image aspect ratio required for our panoramic image input. Therefore, to make this implementation compatible with our setup, we would need to replace both the audio encoder (e.g., with Spatial-AST) and the vision encoder (e.g., with SigLIP2 NaFLEX).

However, with these two major components substituted, the distinctive element remaining from the original baseline would be the large language model (LLM). Accordingly, we instead chose to provide a focused discussion on the potential performance gains from using a stronger LLM.

We would like to refer to the oracle performance provided in the rebuttal to TUUu, where Q3 and Q4, which require mathematical reasoning, show relatively low accuracy. We hypothesize that this is due to the limited mathematical capability of Qwen2-7B.

Notably, Qwen2.5 demonstrates significant improvements in arithmetic tasks:

This suggests that using Qwen2.5-Omni’s LLM could raise the upper bound for performance on questions such as Q3 and Q4, which involve numerical and spatial reasoning.

Additionally, to the best of our knowledge, Gemini 2.5-Flash/Pro currently does not support spatial or binaural audio understanding.

Gemini 2.5 Pro does not mention any capability related to spatial reasoning with audio in its technical report. To evaluate whether it possesses spatial understanding, we conducted a simple experiment. We used audio samples from our supplementary material, excluding those labeled as directly in front or directly behind, and prompted Gemini to classify whether the sound was coming from the left or the right. Our empirical testing showed that its performance was no better than random guessing. Moreover, when we repeated the same input three times, we observed inconsistent responses across trials. This suggests a lack of consistent spatial audio understanding.

prompt: Given the binaural audio, answer which direction is the sound is coming from. Please answer left or right.

Prediction Comparison Table

As we are introducing a new task, we acknowledge the lack of well-established baselines. Even existing sound source localization models are not directly comparable to our setup. To provide a meaningful point of reference, we fine-tuned VideoLLaMA2 on our dataset and carefully aligned the experimental settings to serve as a fair and transparent baseline. We appreciate your consideration of our efforts to build a fair point of comparison in the absence of existing baselines.

Design choice of Q-Former

We appreciate the point. We followed the official Spatial-AST design, which uses a Q-Former for audio, and applied it to SigLIP2 for consistency.

We agree that an MLP is a promising alternative. However, we also want to emphasize that specific architecture of the audio and video encoder themselves is not the main contribution of work. There will be many good choices. We are presenting the dataset and benchmark and functioning model, which is non trivial. There will be a lot of opportunity of future work to improve.

We truly appreciate your positive assessment of our work, especially acknowledging the novelty and clarity of our proposed task and dataset. While our study is not without limitations, we are grateful that you recognized the challenge and complexity of real-world data collection.

In response to the common concern regarding real-world generalization, we have compiled a structured discussion. It’s not a complete solution, but we hope it shows that we have seriously considered the key issues and are thinking critically about how to bridge the sim-to-real gap. We hope this structured discussion is helpful in addressing the concern.

Regarding Real-World Applicability

We fully agree with the importance of evaluating real-world generalization. Given the time and resource constraints, we were unable to conduct real-world experiments during this rebuttal.

As yWdi noted, “itself would be a pretty significant task to design real world data collection and having the synthetic environments mimic that.” We agree that real-world data collection is undeniably costly and complex. Nevertheless, we believe it is essential for future work. A small-scale evaluation set could serve as a starting point to assess how well the model generalizes from simulation to reality (sim2real). Depending on the size of the gap, this would help determine how much real-world training data would be needed for fine-tuning.

As seen in prior work such as BAT[52], defining new simulation scenarios that challenge existing models and evaluating their capabilities in those settings can serve as meaningful contributions. In particular, it allows researchers to demonstrate the feasibility of applying a model to a new task before making a heavy investment in dataset collection. Our work follows this philosophy and aims to encourage future research in spatial audio-visual understanding.

Instead, we would like to respond to Reviewer WvgB’s request “can the authors comment more concretely on how they foresee bridging the simulation-to-real gap?” by providing a structured discussion on what it would take to achieve this, incorporating insights from existing literature, and reviewer suggestions.

From the literature

We highlight two notable efforts. SONICVERSE[1*] introduced three practical strategies: (1) mixing ego-noise from real robot recordings during training, (2) varying source gain to address loudness mismatch, and (3) calibrating depth sensor range to match real-world hardware. Chen et al. [2*] addressed the issue that ray-tracing-based acoustic simulators degrade at low frequencies, and proposed a frequency-adaptive strategy that selects optimal sub-bands based on spectral energy and prior error trends, improving sim-to-real performance in acoustic field prediction.

[1*] Gao et al., Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear, ICRA 2023.
[2*] Chen et al., Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction, IROS 2024.

From the reviewers

TUUu noted that real robots often have physical offsets between visual and audio sensors, such as a microphone located below a camera. While our setup assumes co-located sensors for simplicity, we agree that such offsets can be incorporated during simulation. Moreover, from the perspective of the object rather than the sensor, applying augmentations such as assigning sound sources to varied positions within the object volume could help improve spatial generalization.

doKX pointed out that real environments often include overlapping sound sources, whereas our dataset focuses on single-source localization. BAT, which introduced the Spatial AST model, used the encoder for multi-source spatial reasoning. Since our model also adopts the same encoder, we expect it could handle multi-source settings if such examples are included during training. Due to time constraints, we only conducted a zero-shot evaluation (see Table of the rebuttal to doKX), which showed some degradation but generally reasonable behavior. We would appreciate the reviewers’ understanding that our current focus is on ‘exploring audio-visual spatial alignment beyond semantic alignment’, and we consider multi-source learning as an important direction for future work.

In summary, while our current study focuses on audio-visual spatial alignment in a controlled setting, we view it as a first step toward audio-visual understanding in the real world. We appreciate the reviewers’ insightful comments, which have helped us identify important future directions.