TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data

We really appreciate the reviewer’s thoughtful feedback on our work. Based on that feedback:

1. We have run or plan to run additional experiments, including:

• Measuring the variance in results from sampling 8 images on fMoW. We find minimal (0.1-0.2%) variance.
• Investigating the impact of image references on bitemporal tasks. We find it is the delineation of visual tokens between images that explains most of the improvement.
• Evaluating open-source multi-image models (Qwen2VL, InternVL2) on the temporal EO tasks. Results TBD.
• Evaluating TEOChat on single image grounding tasks. Results TBD.
• Analyzing failure cases of GPT4o and Gemini-1.5 Pro. Results TBD.

We will share results from the remaining experiments as they finish.

2. We have also made changes to the writing in the revised paper, including:

• Softening the claims about comparisons to specialist models.
• Clarifying the comparison to GeoChat and Video-LLaVA.
• Clarifying the amount of TEOChatlas which includes temporal tasks.
• Explaining differences between our work and the concurrent CDChat work.

Our responses to your individual comments are below.

Weaknesses

Similar to video models like Video-LLaVa, the authors limit the sequences to a maximum of 8 images. For longer sequences, how this restriction can affect the model results?

The reviewer makes a valid observation about sequence length. The vast majority of EO tasks use sequences with fewer than 8 images due to infrequent high resolution captures of specific locations, and have variable time intervals between captures. Consequently, EO temporal sequences exhibit much less redundancy compared to natural videos, so TEOChat’s ability to perform well on 8 image sequences addresses the practical needs of most EO applications. We note that across all the datasets we explore in this work, fMoW is the only dataset with sequence lengths longer than 8 images, with 87% of the dataset having sequences with 8 or fewer images. Furthermore, our hardware prevents us from running inference with sequence lengths longer than 8 images, but if there is another experiment we could run that would answer your question, please let us know.

TEOchat randomly sample 8 images without replacement from longer sequences, the choice of samples selected can significantly change the results. Can the authors report variance in results due to different random samplings?

This is an interesting point. We have run an experiment to measure the variance in performance due to the random samples on fMoW, which is the only dataset in our work with sequences containing more than 8 images. Please see the table below.

We find very little variance (0.1 - 0.2%) from different random samplings. We have added this to the paper (Table 12).

The statement "TEOChat also performs comparably to or outperforms a specialist model trained to perform specific tasks" seems quite a strong claim given the proposed model performs quite low in certain cases: e.g., xBD Loc. (IoU) and QFabric [2 and 5 images].

This is a fair point, and we have softened the claims throughout the paper accordingly. For example, for that specific statement we have changed it to:

“TEOChat also performs comparably to or outperforms several specialist models trained to perform specific tasks.”

Table 1, I believe the GeoChat model is evaluated zero-shot given it cannot be trained with multiple images? This should be clarified clearly in Table 1 that the comparison is between a zero-shot model evaluation (not built for multiple images) and TEOchat that is specifically trained with multiple images. Similarly, as Table 2 shows, the VideoLLava is also evaluated zero-shot in Table 1, and compared with the version that is finetuned on the TEOchatlas dataset, I do not think its a fair comparison and authors should clearly mention that they are comparing zero-shot models with a supervised model.

This is another fair point. We have made this more clear in the Table 1 caption in the revised version. Still, we believe it is a meaningful comparison, as the construction of a fine-tuning dataset is one of our contributions and is key to the capabilities demonstrated by TEOChat.

Contribution 2 gives the impression that the 554k instruction set is this paper's contribution while over 55% of it is included from previous work GeoChat. It would be reasonable to clarify this and change the claim accordingly.

We agree that this would benefit from clarification, and have changed it accordingly in the revised version:

“TEOChatlas contains 245,210 temporal EO examples spanning dozens of temporal instruction-following tasks as well as 308,861 single EO examples from GeoChat_Instruct, totalling 554,071 examples and 1,244,393 images.”

I want to thank the authors for their responses to my comments. I am positive about the paper overall and would vote for acceptance. The data and analyses presented in this work are interesting, as this is the first effort in temporal sequence processing with LMMs. I would suggest the authors to include new analysis and results in the revised version when they are available.

We sincerely appreciate the reviewer’s feedback on our work. Based on that feedback:

1. We have added a comparison to SOTA specialists on the datasets.
2. We will be including several more qualitative examples demonstrating TEOChat’s capabilities more specifically.
3. We have fixed a few typos and improved some language throughout the paper.

Our responses to your individual comments are below.

While contributing a novel application and a new dataset is encouraging, the technical contribution of this paper is insufficient. This paper primarily focuses on employing Low-Rank Adaptation (LoRA) to fine-tune a LLaVA-1.5-like architecture, which allows the model to handle both temporal and Earth Observation (EO) data. This paper could benefit from a more in-depth analysis of why such a design for the framework was chosen.

We appreciate the reviewer’s feedback. We would like to emphasize that our work is the first to develop a vision and language assistant for temporal EO data, enabling the model to handle many more tasks and datasets than previous foundation models for remote sensing data all within a single framework. The model is high performing across these tasks, which is nontrivial to accomplish as demonstrated by our experimental ablations. We believe this has important implications on future research in foundation models for remote sensing, and these findings could also be valuable to researchers developing VLMs for image sequences in other domains.

We agree that motivation for the design of the framework is necessary. We include multiple paragraphs motivating the design of TEOChat in Section 4.1 and we report several ablations justifying design decisions empirically in Section 5.2. Specifically, we motivate the design choices for:

• VLM architecture: an adapted Video-LLaVA [1] architecture, extending LLaVA-1.5 [2].
• Weight initialization: Video-LLaVA outperforms LLaVA-1.5 in our ablations.
• Fine-tuning strategy: LoRA for memory/retaining pretrained knowledge, frozen vision encoder for memory as is common among EO VLMs [3-6], projector tuning which outperforms freezing in our ablations.
• Prompt design: including image references, which outperforms their exclusion at train-time and test-time in our ablations.

If the reviewer still has concerns about how we motivate our design of the approach, please let us know.

[1] Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. EMNLP 2024.

[2] Improved Baselines with Visual Instruction Tuning. CVPR 2024 Highlight.

[3] RSGPT: A Remote Sensing Vision Language Model and Benchmark. arXiv 2023.

[4] GeoChat: Grounded Large Vision-Language Model for Remote Sensing. CVPR 2024.

[5] LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model. ECCV 2024.

[6] VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis. arXiv 2024.

Although the authors mention that their approach achieves impressive performance on various datasets (refer to Table 1), the analysis of these results lacks specificity. It is recommended to present relevant visualizations to demonstrate the hypothesis.

We will be adding several more qualitative examples of TEOChat’s capabilities across different tasks to show specific examples of its strong performance. If the reviewer has any other suggestions for more specificity or visualizations, please let us know.

While the study provides a valuable analysis of the proposed approach against existing benchmarks, it would be enhanced by incorporating more recent specialist methods into the comparative analysis.

Thank you for the suggestion. We include the most recent specialist results for QFabric and a comparable, recent specialist for fMoW RGB in Table 1. In the revised version, we now include a comparison to the state-of-the-art (SOTA) performance on fMoW RGB in the results text [8], which TEOChat attains close performance to (75.1% vs. 79.2%). We also now include a comparison to SOTA on S2Looking detection [9] and xBD localization [10] in the results text, and find that TEOChat considerably underperforms them in pixel-based F1 score. This is largely because these two bitemporal change detection models output segmentation masks, whereas TEOChat is constrained to output boxes. As we state in the Conclusion, further improving object localization is an important direction for future work.

[8] ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts. arXiv 2024.

[9] A New Learning Paradigm for Foundation Model-based Remote Sensing Change Detection. IEEE TGRS 2024.

[10] ChangeMamba: Remote Sensing Change Detection with Spatio-Temporal State Space Model. IEEE TGRS 2024.

There are some typos in the paper. For example, Line 106: “To align the the EO imagery”, double “the”.

Thank you for pointing out this issue. We have fixed this typo and a few others.

Based on your feedback, in the next revision we have added more qualitative examples to show TEOChat’s generalization capabilities which are not present in the baselines. We now highlight them in Figure 4 in the main text, and moved the canonical task examples previously in Figure 4 to the Appendix (Figure 7 in Appendix I.5). The new examples show that TEOChat exhibits generalization to tasks not explicitly represented in its training dataset. Notably, it can compose its spatial and temporal reasoning capabilities, for example identifying objects only regionally annotated in the single image tasks (like airplanes and cars) within different images in the sequence.

If you could please let us know if we have addressed your concerns, we would be very grateful.

We are very grateful for the reviewer’s insightful comments on our work. Based on the comments:

1. We have split off a separate related works section from the introduction.
2. We have added more discussion on applicability and scalability of TEOChat.
3. We will be adding more qualitative examples demonstrating TEOChat’s generalization.

Our responses to your individual comments are below.

W1. The introduction often reads as a related work section. Consider moving some of the contextualization of the field in the related work section and focus a bit more on the core story and contributions in the intro.

Thank you for your feedback on the introduction. As you can see in the revised paper, we have moved many references into a separate related work section (placed in Appendix Section A due to space limitations) and now focus on the core story and contributions in the intro.

W2. While there is a long history of geo-spatial data in computer vision, the paper contribution does strike me a little narrow in scope.

W3. Methodological novelty. The paper applies a fairly straightforward fine-tuning recipe. While the paper does not oversell the method, this combined with the narrow scope (W2) do make me a bit concerned.

We appreciate the reviewer’s feedback. We would like to emphasize that our work is the first to develop a vision and language assistant for temporal EO data, enabling the model to handle many more tasks and datasets than previous foundation models for remote sensing data all within a single framework. The model is high performing across these tasks, which is nontrivial to accomplish as demonstrated by our experimental ablations. We believe this has important implications on future research in foundation models for remote sensing, and these findings could also be valuable to researchers developing VLMs for image sequences in other domains.

W4. What are some broader applications of these agents? Is it possible to deploy these agents at scale to understand patterns in data that would be intractable for humans to otherwise deduce themselves?

This is an interesting question which we believe deserves discussion in the paper. We include two paragraphs on broader impacts in the Appendix, and have added more about these points there:

“Automated, generalist methods for processing temporal EO data like TEOChat have the potential to lead to significant, positive societal impacts. For example, we believe that such methods could be deployed at scale on sequences of Earth observation data to automatically derive information, ranging from disaster response to urban development monitoring. This analysis is typically conducted by experts acquiring data on the ground or by manually inspecting UAV/satellite imagery, which becomes impractical to do in large areas which may require several thousands of high resolution images to capture. Such technology could, for example, aid government agencies and humanitarian organizations to acquire information much more rapidly, which is key for disaster response [1].”

[1] A systematic review of disaster management systems: approaches, challenges, and future directions. Land 2023.

W5. The task presented seem like important capabilities but not particularly difficult for specialized models to solve. How do baseline classifiers, referring expression, change detection, etc. algorithms perform on the tasks. What generalization capabilities can be expected in the proposed model that are not present in these baselines?

This is a great question. We have compared to specialist baseline models which can only perform a single task (classification, change detection, and region-based question answering), and have found that TEOChat performed comparably to or outperforms many of them.

Furthermore, TEOChat can perform tasks specified through natural language, making its uses much more broad than any specialist. We have seen TEOChat exhibit interesting generalization to single and temporal image tasks not explicitly represented in the dataset because of this natural language interface. We will add qualitative examples of this to the next version of the paper and let you know when we do.

In the next revision, based on your feedback we have added more qualitative examples to show TEOChat’s generalization capabilities which are not present in the baselines. We now highlight them in Figure 4 in the main text, and moved the canonical task examples previously in Figure 4 to the Appendix (Figure 7 in Appendix I.5). The new examples demonstrate that TEOChat exhibits generalization capabilities not explicitly represented in its training dataset. Notably, TEOChat can compose its spatial and temporal reasoning abilities, for example identifying objects only regionally annotated in the single image tasks (like airplanes and cars) within different images in the sequence.

If you could please let us know if we have addressed your concerns, we would be very thankful.

We sincerely thank the reviewer for providing helpful feedback on our work. Based on that feedback, we have added the following to our revised paper:

1. A comparison to works besides GeoChat. TEOChat demonstrates similar or better single image performance compared to other single image only approaches on six of the seven tasks.
2. RS captioning results on UCM-captions, Sydney-Captions, and RSICD. TEOChat outperforms specialists and performs competitively with generalist single image EO models on the larger two of the captioning datasets (UCM-captions and RSICD).

Point-by-point responses follow.

It would be more convincing if the authors could provide more evidence on the diversity and representativeness of the dataset, as well as fine-tuning results on supervised datasets.

We appreciate the reviewer's point. In the Appendix, Figure 5 plots the geographic distribution of TEOChatlas, showing it is highly diverse and representative of many areas around the world, and Table 7 shows it spans several sensors/spatial resolutions and tasks. Is there any other information we could include to further prove its diversity/representativeness?

Regarding fine-tuning, we now include results from fine-tuning on 3 captioning datasets (as done in RSGPT), described below. While we agree fine-tuning on other datasets would be interesting, we feel this is out of scope of our study, which aims to develop a model that used for many different tasks without the need for fine-tuning. However, we will include code in our open-source repo to make it easier for users to fine-tune TEOChat on other datasets.

Why only a comparison with GeoChat is done, whereas many other similar works are available?

We compared TEOChat to GeoChat as it is the most directly comparable model: it uses a similar architecture and is trained on a subset of the TEOChatlas dataset. However, we agree with the reviewer that comparing to other works would still be beneficial, so we have added results from other models to Table 5. See the table below.

TEOChat shows strong performance across the single image tasks, achieving the highest performance on LRBEN presence (+0.5%) and composition (+0.7%), close to highest on LRBEN Rural/Urban (-1.0%), and highest zero-shot on HRBEN Presence (+3%). It notably underperforms LHRS-Bot and VHM on AID (-10.4-10.8%), which we believe is likely due to the inclusion of several more LULC classification datasets in their training sets. We reemphasize that both of those models were only designed for single image tasks, whereas TEOChat can also perform temporal tasks. We added these results to the paper (Table 5), pasting for convenience:

We italicize results from fine-tuning on the corresponding training set. All other results are zero-shot. We bold the highest zero-shot score, except LRBEN which all models but Video-LLaVA are fine-tuned on.

What is the performance for other RS captioning datasets such as UCM-CAPTIONS and SYDNEY-CAPTIONS datasets which are mentioned in RSGPT[1].

This is an interesting question. Prior works evaluating on these captioning datasets include them in the training dataset, so we have run experiments fine-tuning TEOChat on UCM-Captions, Sydney-Captions, and RSICD following RSGPT’s procedure. Please see Tables 9-11 in the revised Appendix, which are not pasted here for brevity.

Compared to both single image EO specialist and generalist models, we find that TEOChat achieves the highest performance across multiple metrics on UCM-captions and comparable performance to RSGPT on RSICD. On Sydney-captions, TEOChat underperforms RSGPT, but qualitatively we find TEOChat’s captions are high quality. We emphasize that in addition to these single image capabilities, TEOChat has temporal reasoning capabilities that none of these other models have.

It will be also important to showcase the performance w.r.t other performance measures such as recall etc

We appreciate the reviewer’s sentiment about performance measures. We used the standard, commonly reported metrics for each dataset, and tried to include a diverse set of metrics across classification and detection tasks including accuracy, F1, and IoU. We can report F1 performance broken down by precision and recall. Does the reviewer have suggestions for other metrics that would help holistically measure performance?

Thanks for the revision.

1. the results for the proposed method on the caption datasets (RSICD and Sydney) are not the best. some explanation and showcasing average results on all datasets might add value. It will also be important to showcase the zero short results.

If any of these analyses can exhibit the superiority of the proposed method, I will be in favour of the paper and raise my rating.

2. Ok about this
3. Might add the results of GeoChat on 224x224 as another row to the table.

I can not appreciate the last reply from the authors:

1. if some experiments are requested that exhibit poor results does that showcase the novelty of the proposed method?
2. I Would request to go through GitHub and other sources of a paper before criticizing,

3: ** -For this comment "There is also evidence to believe at least some of the metrics reported in the GeoChat supplement are in fact incorrect"**- How do you conclude this? I will request -ACs, SACs, and PCs. for your attention on this. It is very illegitimate comment made by the author to prove something. Is this the purpose of a discussion?

We acknowledge the concerns raised by the reviewer and respectfully offer the following clarifications:

1. As we have stated, these newly added single image captioning results are not core to the contributions or claims of our paper, and there are known issues with these benchmarks that necessitate caution when inferring from results. Still, TEOChat achieves results reasonably competitive with other single image methods on average in both the fine-tuning and zero-shot settings according to our results (Table 14, Table 18, Figure 8).
2. We have gone extensively through the GeoChat GitHub repository, and did not find the supplemental material, only available separately in the CVPR repository, until the reviewer shared the newly uploaded pdf on GitHub. However, as we stated before, these supplemental results are only relevant to the new captioning experiments this reviewer requested during the discussion period. We will resolve any discrepancies in these results to the best of our ability.
3. We have not concluded that the presented results are certainly incorrect, just that the drastic improvement in METEOR score puts the results into question. To be more specific, on RSICD, recent methods achieve METEOR scores ranging from 23.0 to 30.1 (see RSGPT), GeoChat reports a METEOR score of 49.3, and our computed numbers are 28.9 for TEOChat and 28.0 for GeoChat. On UCM captions, recent methods achieve scores ranging from 36.5 to 42.2, GeoChat reports a METEOR score of 75.4, and our computed numbers are 43.9 for TEOChat and 44.4 for GeoChat. The numbers we compute are much more similar to the ranges reported in RSGPT. Should the GeoChat fine-tuning, inference, and evaluation code for the captions datasets be released in the future, we would be happy to conduct a more thorough investigation.

As our discussion has shifted away from the core aspects of our work, we remain open to discussion about aspects more related to our central claims and contributions. We still greatly appreciate the reviewer’s time and effort to review and discuss our paper, so we thank you again.

We have made another revision of the paper to address the remaining reviewer comments. Specifically, as we stated we would do in our previous post, we have added:

1. A comparison to two open-source multi-image VLMs, namely Qwen2-VL and InternVL2 (Table 4). TEOChat outperforms both Qwen2-VL and InternVL2 by a large margin on xBD damage classification (+38.2 F1, +32.7 F1 respectively) and S2Looking change detection (+25.8 F1, +23.7 F1 respectively).
2. An evaluation of TEOChat on single image grounding tasks (Table 12). TEOChat outperforms MiniGPTv2 across all grounding tasks (+5.3% overall) and outperforms GeoChat in detecting large objects (+2.2%) but underperforms GeoChat across the other grounding tasks, which we largely attribute to the lower spatial resolution required to handle the extra temporal inputs. We emphasize, however, that TEOChat has additional temporal capabilities that GeoChat does not.
3. An analysis of the failure cases of GPT4o and Gemini-1.5 Pro (Section 5.4). Gemini-1.5 Pro and GPT4o tend to underpredict change in S2Looking change detection and underestimate building damage in xBD damage classification.
4. More qualitative examples of TEOChat outputs, including instances of generalization (Figure 4 in Results) and example predictions on the captioning datasets (Figure 8 in Appendix Section I.6). Interestingly, the examples show that TEOChat can compose its spatial and temporal capabilities in ways not explicitly represented in the training dataset.

We would be very grateful if the reviewers could examine these changes (all marked in red in the revised paper) and let us know if your concerns have been addressed.

Thank you again.

We have made a few more changes to the revised paper based on the recent comments, including:

1. Average results on the three captioning datasets (Table 14). TEOChat outperforms all approaches (including GeoChat and two specialists) besides RSGPT. We add explanation for this in the paper, described in detail to reviewer 7dgE below.
2. Zero-shot results on the three captioning datasets and average performance (Tables 15-17). TEOChat performs similarly to or outperforms GeoChat on average across all metrics, with notable improvements on the largest, more difficult captioning dataset (RSICD).
3. Adding more motivation to Section 1. We have trimmed extraneous language in the dataset construction and experiments section in order to make room for this additional motivation in Section 1.

We believe our paper has substantially improved because of the feedback, so we are very grateful to the reviewers for their valuable help to make the work stronger. If there are any remaining concerns, we would be happy to continue discussing.