Stitch and Tell: A Structured Data Augmentation Method for Spatial Understanding

Dear Reviewer 3Q8z, we truly appreciate your thorough review, positive feedback, and the time and effort you devoted to strengthening our paper. We are grateful that you highlighted the simplicity and practical value of our approach, as well as the importance of spatial understanding for real-world applications. We are also encouraged by your recognition of the effectiveness of SiTe. Your thoughtful remarks are highly motivating for us.

Q1:Did the authors evaluate if, as a side effect, also high-resolution image understanding of LMMs trained with SiTe data is improved?

R1: Thanks and we really like this insightful comment! We conducted additional experiments on the HR-Bench 8K [a] and V-Star [b] high-resolution datasets. As shown in the table below, models enhanced with SiTe consistently achieve better performance on these high-resolution benchmarks. This result motivates us to further explore the potential of SiTe for more aspects in visual understanding task!

[a] Wang, Wenbin, et al. "Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 8. 2025.

[b] Wu, Penghao, and Saining Xie. "V*: Guided visual search as a core mechanism in multimodal llms." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q2: Did the authors consider the resizing or grouping the images of similar size?

R2：Thank you for your thoughtful suggestion! We found your idea very interesting and have incorporated it by improving the SiTe stitching strategy.

$\mathbf{SiTe_{resize}}$: For left-right stitching, we resize the shorter image so that both images have the same height; for top-down stitching, we resize the narrower image so that both images have the same width.

$\mathbf{SiTe_{ratio}}$: We first categorize images based on their aspect ratios. Images with width/height > 1.2 are grouped for top-down stitching, and images with height/width > 1.2 are grouped for left-right stitching.

$\mathbf{SiTe_{default}}$ refers to the default random stitching method described in the paper.

We conducted experiments using these variants in the pretraining stage of LLaVA with the blip_laion_cc_sbu_558K(558K) dataset. The results are shown below:

As shown, $\mathbf{SiTe_{ratio}}$ achieves better on most benchmarks. We suppose this is because grouping images by ratio ensures that the stitched images have more realistic proportions, avoiding unnatural shapes that are too wide or too tall. $\mathbf{SiTe_{resize}}$ may create stitched images with resolutions that differ from the testset, which could negatively affect performance.

Q3: Did SiTe hurt or benefits spatial understanding when it comes to a notion of depth?

R3：Thank you for raising this concern! We addressed it by further evaluating our models on CV-Bench[a] and on RealWorldQA[b], which focuses on basic real-world spatial understanding in multimodal models. The results are presented below.

As shown, the injection of spatial supervision via SiTe leads to improved model performance on out-of-domain spatial relation tasks and demonstrates strong generalization to more complex spatial reasoning scenarios in CV-Bench and RealWorldQA. A reasonable hypothesis is that the model benefits from its improved understanding of fundamental directional relationships.

[a] Tong, Peter, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." NeurIPS 2024.

[b] X.AI. Realworldqa. Blog post, July 2025. Accessed: 2025-07-30.

Q4：Could the authors please report the standard deviation as well?

R4: Of course! We compute the standard deviation in the below table. Due to space limitations, we show a subset of the results here.

As indicated by the standard deviation results, the performance gains are consistent and robust, and we will add the whole results in the revised paper.

Q5：Are you considering comparing other data augmentation methods, even they are more heavier?

R5：Thanks for the suggestion! Following approaches similar to [a] and [b], we performed data augmentation by using a multimodal large language model to rewrite the original captions and inject spatial information into the training data. We used the following prompt for GPT-4o to rewrite captions based on the 558K.

Based on the image and its original caption, Your task is to generate a new caption that prioritizes spatial relationships, no more than 20 words.

To evaluate the rewritten captions[c], we used Qwen2.5VL-7B as a judge with the following prompt:

Your task is to compare two captions for an image and determine which better meets the evaluation criteria. (Print ONLY 1, -1, or 0 as the result) Evaluation Criteria:

1. Accuracy: How accurately the caption describes the actual content of the image.
2. Positional Details: The quantity and quality of positional/directional descriptions. Result: Return 1 if Caption1 is better, -1 if Caption2 is better, and 0 if equally good. Caption1: {new_caption} Caption2: {old_caption}

The results are as follows:

Score=1: 94.36%, Score=0: 2.97%, Score=-1: 2.68%

These results confirm that the rewritten captions significantly improve spatial semantic richness and descriptive accuracy.

We then used the rewritten captions to train LLaVA in the pretraining stage. The results are shown below:

As shown, rewriting captions leads to significant performance improvements, demonstrating the importance of explicitly insert spatial information in captions. This approach also produces captions that are more coherent and natural, injecting richer supervisory signals and more detailed information.

However, it is worth noting that caption rewriting for the 558K dataset cost approximately $560 using the GPT-4o API, and this expense would increase significantly for even larger datasets (not including the additional cost of caption quality evaluation).

As described in our paper, SiTe focuses on providing a lightweight, effective, and unambiguous way to inject spatial supervision. In future work, we plan to explore combining SiTe with caption rewriting to further enhance spatial semantic richness in the training data.

[a] Chen G H, et al. Allava: Harnessing gpt4v-synthesized data for lite vision-language models[J]. arXiv preprint arXiv:2402.11684, 2024.

[b] Chen L, et al. Sharegpt4v: Improving large multi-modal models with better captions[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 370-387.

[c] Lee, Yebin, et al. Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model. arXiv preprint arXiv:2406.06004 (2024).

We thank the review again for this insightful and interesting comment. We will include the above analysis in the revised paper to make the paper more complete. Thanks again!

Dear Reviewer htxQ, we thank for your careful and helpful comments on our work. We appreciate the reviewer’s recognition of the simplicity and effectiveness of our approach, as well as the scalability benefits of SiTe. Below are our responses to all the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.

Q1: How many total samples were used for training? Do you use 1K/5K augmented data along with original data? What is the total training size, composition, and diversity?

R1：Thank you for your question! SiTe applies data augmentation in both the pretraining and fine-tuning stages.

In the pretraining stage, we experimented with two datasets. The first is blip_laion_cc_sbu_558K: $\mathbf{LLaVA_{Baseline}}$ was trained on 558K image–caption pairs, while $\mathbf{LLaVA_{SiTe}}$ used 458K pairs, since one-third of the data was stitched and duplicate pairs were removed (as shown in Table 3 of the manuscript). The second dataset is flickr-30K: $\mathbf{LLaVA^{Flickr}}$ was trained on 30K image–caption pairs, while $\mathbf{LLaVA_{SiTe}^{Flickr}}$ used 24K pairs, with one-third of the data being stitched and deduplicated.

The number of data in pretraining stage is shown in the figure below (taking LLaVA as an example):

In the supervised fine-tuning stage, we performed partial stitching on the original llava_v1_5_mix665k dataset. Specifically, we randomly sampled 1K or 5K augmented samples and added them to the original fine-tuning dataset llava_v1_5_mix665k(665K) for full-parameter fine-tuning. This design helps maintain model generalization and avoid overfitting.

The number of data in supervised fine-tuning stage is shown in the figure below :

Regarding template diversity, SiTe employs a variety of directional stitching templates, as detailed in Appendix Tables 5, 6, 7, and 8. During augmentation, the stitching format is randomly sampled from these templates, so each template is selected with equal probability. As for the diversity of the training set, since SiTe does not introduce any additional supervision beyond the templates, the diversity of the augmented data remains consistent with that of the original dataset.

Q2: What is the distribution of different stitched image types and question types?

R2：Thank you for your question. As stated in the paper, the proportion of horizontally and vertically stitched images, as well as question types, is set to 1:1 by design:

We apply both horizontal and vertical stitching in a 1:1 ratio.

And we specify the default SiTe setting:

By default, the stitched data make up one-third of the total set.

These details can be found in Section 4.2 (Pretraining Stage) of our paper.

Q3: Would the method benefit from testing on SOTA MLLMs?

R3：Thank you for your suggestion! Given the architectural similarities between QwenVL and InternVL, and considering that we have already included experiments with LLaVA-Qwen2 in the main text, we further conducted preliminary experiments on InternVL2-4B. We modified the training process to follow a Pretrain + SFT paradigm and applied SiTe augmentation during the pretraining stage. Specifically, we used both the blip_laion_cc_sbu_558K dataset and its SiTe-augmented version (using the default setting).

As shown, InternVL-SiTe consistently outperforms the baseline InternVL on most spatial understanding tasks, while also maintaining strong generalization on general benchmarks. This demonstrates that SiTe can also benefit SOTA MLLMs. These results will be incorporated into the updated version to further enhance the quality of our submission.

Q4: Is the proposed method able to handle more complex reasoning about occlusion, depth, and partial visibility?

R4: Thank you for your insightful comment! We acknowledge that the current stitching paradigm does not fully capture the complexity of natural spatial relationships. However, SiTe is designed to provide a lightweight and effective way to inject explicit spatial supervision at minimal cost, without requiring complex data collection or annotation.

To evaluate SiTe's generalization, we selected out-of-distribution spatial relation samples from COCOQA (COCOQA-ood), which include spatial relation keywords such as "behind", "in front of", "close to", "far away from", "inside", "outside", "within", "internal", "external", "ahead of", and "adjacent to". We further evaluated our models on CV-Bench[a], which specifically tests complex spatial reasoning, including depth and occlusion, and on RealWorldQA[b], which is designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The results are shown in the table below:

As shown, the injection of spatial supervision via SiTe leads to improved model performance on out-of-domain spatial relation tasks and demonstrates strong generalization to more complex spatial reasoning scenarios in CV-Bench and RealWorldQA. A reasonable hypothesis is that the model benefits from its improved understanding of fundamental directional relationships. We agree that more realistic and comprehensive spatial augmentation is an important direction for future work.

In summary, although SiTe does not directly address occlusion or 3D reasoning, our results show that it improves generalization to these more complex spatial tasks.

[a] Tong, Peter, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." NeurIPS 2024.

[b] X.AI. Realworldqa. Blog post, July 2025. Accessed: 2025-07-30

We thank the reviewer again for these careful comments and hope these responses address your concerns!

As the discussion period is nearing its conclusion, we sincerely hope that our clarifications have sufficiently addressed your concerns, and would welcome any additional feedback you may have. If you find our responses satisfactory, we would greatly appreciate it if you could consider updating your score to reflect the outcome of the discussion.

Thank you for your consideration. :)

Dear Reviewer eNJr, we are thrilled that you found the idea is interesting and provides an effective way to introduce spatial supervision without requiring manual annotations and expensive computational resources. We are especially pleased that you recognize our extensive experimental results. Your acknowledgment of the SiTe consistent improvements in spatial understanding tasks is greatly appreciated. We will address your concerns and questions below.

Q1: Can you provide the complexity of the methodology?

R1: SiTe is designed to be both computationally and operationally. The core augmentation procedure consists of two simple steps:

Image Stitching: This step involves copying and pasting two images onto a new canvas. Its computational complexity is linear in the number of pixels (O(H*W)).

Template-based Caption and QA Generation: Both operations are either constant (O(1)) or linear in the length of the caption (O(L), where L is the number of words).

If SiTe is implemented as an online augmentation, stitched images and captions are generated in memory on-the-fly for each training batch, eliminating the need to store them on disk. This results in negligible storage overhead, with space complexity reduced to O(B × H × W × C), where B is the batch size. Only the original dataset needs to be stored. We will clarify this in the revision.

Q2: Where is the limitations section?

R2: We would like to clarify that the limitations of our approach are explicitly discussed in Appendix A. In this section, we note that the current work focuses on horizontal (left–right) and vertical (top–down) spatial configurations. We also discuss that future work may explore more comprehensive spatial relations by incorporating depth maps, multi-view images, and other settings that enable 3D perception.

Q3: Does this method lead to a significant increase in the size of the training dataset?

R3: Thank you for raising this concern. In fact, with SiTe, the total number of training samples is reduced rather than increased. For each stitched sample, the two original image-caption pairs are removed from the raw dataset and replaced by a single new sample. As a result, the final training set contains fewer samples than the original dataset. This design ensures that SiTe does not pose scalability issues, even for large-scale applications. Furthermore, SiTe can be implemented as an online augmentation, generating stitched samples dynamically during training without increasing storage requirements. We will clarify this point in the revised manuscript.

In summary, SiTe is both storage-efficient and scalable, reducing the size of the training data and making the overall pipeline more efficient.

Q4: This method may not be sufficient to capture all nuances of spatial relationships in images

R4: Thank you for raising this important point. We agree that our approach is a simplified model of spatial reasoning and does not attempt to replicate all the complexities of human perception. The main goal of SiTe is to inject clear and explicit spatial supervision, which is often lacking in existing datasets.

To explore the generalization of our method, we selected out-of-distribution spatial relation samples from COCOQA (COCOQA-ood), which include spatial relation keywords such as "behind", "in front of", "close to", "far away from", "inside", "outside", "within", "internal", "external", "ahead of", and "adjacent to". We further evaluated our models on these as well as on the CV-Bench[a], which specifically test for complex spatial reasoning including depth and occlusion, and on RealWorldQA[b], which is designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The results are shown in the table below:

As shown, the injection of spatial supervision via SiTe leads to improved model performance on out-of-domain spatial relation tasks and demonstrates strong generalization to more complex spatial reasoning scenarios in CV-Bench. A reasonable hypothesis is that the model benefits from its improved understanding of fundamental directional relationships.

While our method focuses on horizontal and vertical relations, we observe in our experiments that the enhanced spatial representations learned through SiTe also generalize to a broader range of spatial reasoning tasks, including cases beyond these basic relations. This demonstrates that SiTe provides a practical and effective foundation for spatial understanding. We believe that SiTe offers a practical and scalable first step, and we explicitly discuss in the limitations section that extending to more complex and realistic spatial relations is an important direction for future work.

Overall, although SiTe only augments basic directional relations, it demonstrates strong generalization to more complex spatial relationships in practice.

[a] Tong, Peter, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." NeurIPS 2024.

[b] X.AI. Realworldqa. Blog post, July 2025. Accessed: 2025-07-30.

Q5: The caption of Figure 1: spatially-aware->Spatially-aware.

R5: Thanks for pointing this out. We have double-checked and polished our manuscript to fix the typos.

We hope this response clarifies the remaining concerns and highlights the contributions of our work. Thank you again for your positive comment.

As the discussion period is nearing its conclusion, we sincerely hope that our clarifications have sufficiently addressed your concerns, and would welcome any additional feedback you may have. If you find our responses satisfactory, we would greatly appreciate it if you could consider updating your score to reflect the outcome of the discussion.

Thank you for your consideration. :)

Dear Reviewer 3L35, thank you for recognizing the strengths of our paper. We are pleased that you approve the motivation of our work, appreciate the novelty and extensibility of the SiTe method, and acknowledge the comprehensive experiments. We will address your concerns and questions below.

Q1: Can you describe the process for selecting image pairs, and whether object co-occurrence within image pairs influences the model’s learning dynamics?

R1: Thank you for your question. The process for selecting image pairs is thoroughly described in Section 3 (“Image-Spatial Stitching”) and Section 4.2 (“Pretraining Stage”) of our paper. During the image-spatial stitching, we perform random sampling as follows:

Given two randomly sampled image–caption pairs (I₁, T₁) and (I₂, T₂) from the dataset, we first construct a stitched image by spatially stitching the two input images along a specific axis

Algorithm 1 provides detailed pseudocode for the image stitching procedure, including how the canvas is created and how the two images are merged.

To avoid duplicating image-text information, we describe in Section 4.2 a non-overlapping sampling strategy.

We use the raw dataset to create both horizontally and vertically stitched samples. In the final training set, there are three components: horizontally stitched samples, vertically stitched samples, and the remaining raw data. Each image–caption pair used for stitching is removed from the raw dataset and does not appear as a standalone example.

To avoid duplicate supervision, image–caption pairs used for stitching are removed from the raw set, ensuring each image appears only once.

This ensures that the model sees exactly the same underlying image-text information as the baseline, making comparisons fair. We also maintain a 1:1 ratio of horizontal to vertical stitching in the training set:

We apply both horizontal and vertical stitching in a 1:1 ratio.

And we specify the default setting:

By default, the stitched data make up one-third of the total set.

These steps support the full reproducibility of our experiments.

Additionally, we implement an object co-occurrence filter. As described in Section 3 (“Tell the Question and Answer”), we remove overlapping objects when constructing spatially-aware QA pairs:

To reduce ambiguity, we remove overlapping nouns shared by both sides.

This design ensures that SiTe produces unambiguous spatial questions.

Q2：The authors do not discuss limitations and the potential for the method to generalize to a broader range of spatial relations

R2: We would like to clarify that the limitations are discussed in Appendix A. Our work currently focuses on horizontal and vertical spatial configurations and we will exploring broader spatial relations, including 3D perception in future work.

To further address your concern, we evaluated our models on the CV-Bench[a], which specifically test for complex spatial reasoning including a broader set of spatial relations.

The results are shown in the table below (Owing to space constraints, further results on additional datasets are provided in our response to R4 for Reviewer eNJr.):

As shown, the injection of spatial supervision via SiTe leads to demonstrate strong generalization to more complex spatial reasoning scenarios. A reasonable hypothesis is that the model benefits from its improved understanding of fundamental directional relationships.

[a] Tong, Peter, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." NeurIPS 2024.

Q3: It is unclear whether the evaluation datasets are specifically constructed or filtered for explicit spatial reasoning. While the VSR dataset contains implicit spatial relations, is it fair to compare?

R3: Thank you for your comment. We provide a clear description of the evaluation benchmarks in Section 4.4, where we introduce each dataset and its characteristics. The pretraining datasets including those for stitching, are described in detail as follows:

To evaluate the effectiveness of SiTe across different training sets, we experiment with two image–caption datasets: blip_laion_cc_sbu_558K(558K) and Flickr30K.

SiTe uses horizontal and vertical spatial relations, with templates listed in Appendix Tables 5–8. Nevertheless, the evaluation benchmark are public that cover a much broader range of spatial relations and were not filtered for explicit spatial reasoning.

It is fair to compare the $\mathbf{LLAVA\_{SiTe}}$ and $\mathbf{LLAVA\_{VSR}}$. The VSR dataset covers many spatial relation types. We use it as a strong baseline in our comparisons. As shown in the table of the top 10 relations below, VSR has not only basic directional relations but also includes other types. Our results show that SiTe outperforms models trained with VSR data ($\mathbf{LLaVA_{VSR}}$), highlighting the effectiveness of our approach.

Q4: What is the difference between LLAVA\_SiTe and LLAVA\_{SiTe}^{Flickr}? And why does the MME score exceed 100?

R4: Sorry for the confusion! We respectfully clarify that, $\mathbf{LLaVA_{SiTe}}$ is pretrained on the original LLaVA training set (558K) with 1/3 data by SiTe, while $\mathbf{LLAVA\_{SiTe}^{Flickr}}$ is pretrained only on Flickr30K with 1/3 data by SiTe. The motivation for including both settings is to demonstrate the generalizability of SiTe in different datasets.

Regarding the MME benchmark [a], each image is associated with two questions: one for "accuracy" (max score 100), and another for "acc-plus" (max score 100, whether both questions for the same image are correct). The total MME score is the sum of two, so the maximum possible score is 200. Thus, there is no need to normalize the score to 100 on this benchmark.

[a] Chaoyou Fu, et al. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394, 2023.

Q5: Dose the classical data augmentation methods have Unexpected Competitiveness?

R5: Thank you for your comment. In this work, we evaluated two classical data augmentation methods: crop and rotate. For each method, we averaged the results across different benchmark. The performance gains (Δ) over the baseline are summarized below:

As shown above, classical methods offer limited gains and often introduce factual conflicts between images and captions, leading to performance drops on some benchmarks (e.g., Spatial-MM, MME_Pos, COCO-QA_Spat). In contrast, SiTe consistently achieves stronger and stable improvements on spatial tasks.

Q6: Can you explain the effectiveness and necessity of SFT in the overall training pipeline?

R6: We are sorry for the confusion. To validate the effectiveness of SiTe, we conducted data augmentation separately in the pretraining and SFT stages, and the the SFT results are not further fine-tuning from the pretrained models. Thus, the two stages are not directly comparable.

To further explore the effectiveness of applying SiTe in both pretraining and SFT stages, we conducted the following experiment:

As shown, applying SiTe-based augmentation in both stages yields further improvements across multiple benchmarks, so these stage are both effective and necessary.

We hope the above response can further address the concerns and misunderstanding. Thank you again for your feedback!

Thank you for your thoughtful response and for engaging with our explanations. We are glad that our clarifications have helped you better understand our work. Below we address your remaining concerns in detail:

Object co-occurrence：We appreciate the suggestion of leveraging semantic co-occurrence in constructing image pairs. However, recent research has shown that over-reliance on frequent object co-occurrence can induce spurious correlations and hallucination in multimodal language models [a, b, c]. Nevertheless, we recognize the potential value of semantic-based pairing and will conduct experiments to further investigate its impact.

[a] Wu, Mingrui, et al. "Evaluating and analyzing relationship hallucinations in large vision-language models." arXiv preprint arXiv:2406.16449 (2024).

[b] Li, Yifan, et al. "Evaluating object hallucination in large vision-language models." arXiv preprint arXiv:2305.10355 (2023).

[c] Zhou, Yiyang, et al. "Analyzing and mitigating object hallucination in large vision-language models." arXiv preprint arXiv:2310.00754 (2023).

More detailed discussion in limitation: While our current approach focuses on binary spatial relations such as left/right and up/down, it does not explicitly address depth-aware relations like “in front of” or “behind.” To handle these more complex spatial relationships, a promising strategy would be to simulate depth by scaling and partially occluding objects. This would allow us to create training examples that represent “front” and “back” spatial relations, further enriching the model’s spatial understanding. For more relationship enhancements, we will continue to explore in our future work.

Why Does Training on Vertical/Horizontal Relations Improve Generalization to Depth: Although our training focuses solely on vertical and horizontal relations, these tasks may share a fundamental spatial reasoning mechanism with depth-related relations[a,b]. In spatial understanding task, the model needs to develop consistent representations of object positions and their relative layouts in space. Directional augmentation, such as vertical/horizontal relations, strengthens the model’s ability to anchor and compare features within a structured spatial framework. This process enhances the model’s overall spatial awareness. As a result, the learned representations become more transferable across different spatial dimensions, including depth. This leads to improved performance on “closer/further” evaluations, even though there is no explicit supervision for those relations.

Performance of Rotate and Crop on CV-Bench:

The results above demonstrate that SiTe outperforms both crop and rotate methods on more complex spatial reasoning tasks. We will continue to explore ways to further enhance the model’s 3D spatial reasoning abilities in future work.

[a] Zhang, Bo, et al. "Distinct networks coupled with parietal cortex for spatial representations inside and outside the visual field." Neuroimage 252 (2022): 119041.

[b] Wu, Charley M., et al. "Similarities and differences in spatial and non-spatial cognitive maps." PLoS computational biology 16.9 (2020): e1008149.

Thank you again for your careful consideration and for increasing your score.

If our responses have fully addressed your comments, we would be sincerely grateful if you could consider a further positive adjustment to your score. If you have any additional questions or requests, please do not hesitate to reach out, we are more than willing to provide further clarifications or conduct additional experiments as necessary. Your feedback is very important to us :)