InstructSAM: A Training-free Framework for Instruction-Oriented Remote Sensing Object Recognition

Weaknesses

Tested only on two datasets covering two classes categories. Only high-res data is evaluated most public RS data is medium-res (e.g. Sentinel 2 with 10m).

Thank you for this observation. Most object recognition datasets in remote sensing are with high-resolution. DIOR was chosen for its wide GSD range (0.3m to 30m), which includes medium-resolution images. Many categories, such as "ship," "airport," "expressway service area," and "ground track field," frequently occur in medium-resolution images (as shown in Figure 2(d)). Additionally, EarthInstruct is a multimodal (NIR+RGB) and multi-resolution (0.08m–30m) benchmark, making it representative of diverse remote sensing scenarios.

Task novelty overlaps earlier instruction-based RS works; “first” claim overstated.

We appreciate your feedback. While there are overlaps with earlier works like EarthDial [1] and TEOChat, we believe InstructCDS and EarthInstruct introduce significant novelties:

• Broader instruction scope: Existing works are limited to counting or detecting one object type per query. In contrast, EarthInstruct benchmark is the first benchmark that supports open-vocabulary,​ open-ended, and open-subclass settings in EO, enabling much more complex and diverse instructions.
• Structured outputs: Earlier works fail to generate structured outputs for multi-class tasks. InstructSAM provides structured, interpretable results, aligning with complex user-defined instructions.

Section headers do not follow the style guidelines.

Thank you for pointing this out. We will ensure all section headers follow the style guidelines, with only the first word and proper nouns capitalized.

Real world applicability is limited due to very specific architecture design and limited generalization experiments.

We acknowledge this concern and have added model generalization and scaling experiments to address it. InstructSAM demonstrates good generalization across various open LVLMs, as shown below.

Additionally, InstructSAM's "out-of-box" design allows it to benefit from any improved open-source or proprietary foundation models, enhancing its applicability. The use of LVLM allows InstructSAM to interpret diverse instructions.

Questions

RS domain: Why did you focus your experiments on RS when you are using models from the natural domain? Why not evaluate natural images as well?

Our focus is on remote sensing (RS) due to its unique challenges, such as the lack of semantically diverse training data. In contrast, the natural domain benefits from abundant datasets like Object365, LVIS, and V3Det, making it relatively straightforward to achieve InstructCDS using models like GPT4o+OWLv2 or by training a generative model such as Ins-DetCLIP [1].

Also, we observed that SAM struggles to segment complex objects (e.g., person, car) as a single entity in natural images. However, this issue is less pronounced in satellite images due to the different texture features with natural images.

That said, we conducted qualitative experiments on natural images, such as electronic components and dice detection (Fig. 13). In these cases, InstructSAM outperformed generic detectors like OWLv2 and Qwen2.5-VL. Quantitative evaluations on natural images remain an interesting direction for future work.

Broader evaluation: As your paper focus on RS, additional experiments with other datasets would improve the paper, specifically, low-res datasets (e.g. S-2) or dataset with more diverse class types (e.g. from land-use or agriculture).

Thank you for this suggestion. While DIOR already includes medium-resolution images, we agree that additional experiments on low-resolution datasets or those with diverse class types (e.g., land-use or agriculture) would strengthen the paper. Here we demonstrate InstructSAM's superior zero-shot ability in land cover classification on FLAIR dataset compared to the recent open-vocabulary segmentation method. Broader evaluation is an exciting direction for future work.

Out-of-distrubution: You are testing InstructSAM on two further datasets in the appendix. Why are they called “out-of-distribution”? You are using generic models (SAM2 and CLIP) that are most likely trained on these categories as well.

We apologize for the confusion. You are correct that these datasets are not truly "out-of-distribution" for InstructSAM. The term "out-of-distribution" refers to datasets outside the training distribution of open-vocabulary detection models like OWLv2, CASTDet, and LAE-DINO. These models showed significant performance degradation on these datasets, whereas InstructSAM, equipped with more capable foundation models, performed better.

Task novelty claim: As you clarify in the related work, InstructCDS does not fundamentally differs from existing instruction-oriented tasks and your claim “new suite of tasks” might be a bit overstated.

• Instruction-Oriented Object Counting and Segmentation: While we followed the settings of previous work on Instruction-Oriented Object Detection [1]. We add object counting and object segmentation to form a new suite of tasks, in other words we are the first to implement the comprehensive tasks altogether. This is a significant advancement, particularly for object counting, as current benchmarks like CountBenchVQA [2] are limited to single-class counting with explicit class names in the query. In contrast, InstructCDS requires counting based on implicit instructions.

• Novel evaluation framework: We defined F1-score for multi-class object counting, providing a more informative evaluation metric (line 141).

References

[1] Pi, Renjie, et al. "INS-DetCLIP: Aligning detection model to follow human-language instruction." ​The Twelfth International Conference on Learning Representations​. 2024.

[2] Paiss, Roni, et al. "Teaching clip to count to ten." ​Proceedings of the IEEE/CVF International Conference on Computer Vision​. 2023.

We thank you for great comments and hope our response clarified the paper's contribution to this community.

Strengths

The mask-label ablation is impressive. If I'm reading this right, then if you just filter boxes by confidence score the version with the BIP matcher outperforms all operating points.

Thank you for recognizing this. Indeed, Figure 6 compares the per-class F1-score of InstructSAM with filtering solely by a single confidence score across all images and categories. Each category requires a different optimal threshold, making a single threshold ineffective across classes. In contrast, the counting-constrained matching process dynamically assigns labels without relying on a fixed threshold, ensuring optimal assignments for each image.

Weakness

The prompt design section of the ablation study is unclear. I don't understand what is being varied

We appreciate your feedback. The "Add Instr" column in Table 4 refers to additional instructions, specifically dataset-specific annotation rules (e.g., NWPU’s “harbor” as distinct piers, excluding small vehicles). See lines 645–651 and 678–693 for details. These instructions guide the LVLM counter to better align with user intent, significantly reducing inherent biases.

While the mask-label ablation is very interesting, I think the reason that BIP matching is able to outperform all operating points is that SAM2 is prompted with a regular grid of points, producing many more candidates than needed. The process isn't trained to produce the correct boxes around all objects, its being asked to produce boxes where there may not be a valid object.

This is an insightful observation. Indeed, SAM2 often generates more mask proposals than the actual number of objects in the image. Constraint Equation (4) at Line 204 guarantees that all mask proposals are assigned when they are fewer than the predicted object counts.

Being confidence free isn't necessarily a good thing. If the final boxes were assigned a score, the ability to filter / prioritize them could be beneficial for some applications. Can this method be imbued with a confidence score?

Great point again! Each matched mask-label pair in InstructSAM already has a semantic similarity score, $s_{ij}$, which is automatically saved in the final predictions. This score can be used for filtering or prioritization, similar to conventional detectors. For example, in the open-subclass setting, threshold filtering based on $s_{ij}$ improved box mF1 by up to 8.4%.

Comments and Questions

A recent paper: Vision Language Models are Biased (Vo et al. 2025) showed a strong bias and reliance on context cues in counting VLMs. Followup work should consider this.

Thank you for pointing this out. This paper focuses on counterfactual scenarios (e.g., counting legs on a three-legged chicken). In remote sensing, annotation rules may vary across datasets but rarely contradict common sense. Adding annotation rules significantly improves counting accuracy. For example, when distinguishing "bridge" from "overpass," the model often recognizes overpasses as bridges due to their hierarchical relationship. However, annotation rules improved bridge accuracy and overpass recall by large margins:

[Line 66] Slow Qwen bounding box generation. Why your runtime is constant (wrt to number of boxes), but others are not. Why does inference time increase with the number of objects in instruction-oriented models?

Representing every bounding boxes as text requires more tokens. Experimental results show that inference time is proportional to the number of output tokens. In contrast, counting requires far fewer tokens, making InstructSAM-Qwen faster than Qwen2.5-VL when the number of predicted objects exceeds five.

[Line 77] Is confidence free actually a pro? Could that also be a con due to no ability to filter based on a score?

Confidence-free methods are advantageous in zero-shot or OOD scenarios, where bounding box quality is highly sensitive to threshold selection. However, if a validation dataset is available for tuning thresholds, confidence scores can be useful.

[Line 163] Using the maximum F1 might not be fair. It is a "best possible" view and cannot be used to say a candidate model is "better" than another.

We agree. A fairer approach is to tune thresholds on training data. InstructSAM can also be evaluated for "best possible" performance by enabling confidence filtering, as shown in the table above.

The impact of threshold filtering varies across different settings. For tasks like OVD and OED, which involve a large number of categories, a single threshold is ineffective at balancing performance across all classes. However, in scenarios with fewer classes, such as subclass-specific tasks, threshold filtering becomes more effective and can significantly improve performance.

[Line 160] Strength: The method for computing AP without confidence values is correct.

Thanks, we hope this method can raise the awareness of fair comparison bewteen confidence-based and confidence-free methods in generic object detection domain.

[[Line 171] Performing a random sample on the selected pairs at cosine similarity > 0.95 and then quantifying the error rate would be valuable to justify this threshold choice.

Thank you for the suggestion. We manually evaluated open-ended counting results on NWPU using Qwen2.5-VL and GPT-4o. The error rate was 3% (1 out of 32), with the only error being "sports_field" mapped to "baseball_field." Most synonyms were correctly mapped, but some misses (e.g., "tank" vs. "storage tank") highlight the need for better RS-specific CLIP models.

[Line 201] The 1.2x scale factor seems arbitrary, was it tuned? Not not a big deal if it wasn't.

It was not tuned initially. We hypothesized that adding context would improve accuracy since CLIP is trained on full images rather than object crops. Experiments on NWPU in the OVD setting confirmed this, with 1.3x yielding the best results:

[Line 222] Clarify, I find this wording hard to follow: For open subclass setting, we set two parent class "means of transport" and "sports field".

The subclasses for "means of transport" and "sports field" are listed in Table 7 (Appendix B). We will add a reference to this table in the main text for clarity.

[Figure 6] Is the Yellow line adding to the key takeaway of this plot? I'm also confused how mAPnc of InstructSAM w/o matching has different values at different thresholds. Am I missing something here? Please double check these numbers and if the takeaway is that BIP matching outperforms all operating points, state that explicitly or add clarifying text so readers don't make that mistake.

The yellow line represents InstructSAM without matching, where predictions are filtered by a similarity score threshold before evaluation. That's the reason the value differs. We will clarify this in the revised manuscript. The key takeaway is that BIP matching outperforms all operating points, and we will explicitly state this to avoid confusion.

Have you tried using the predicted object classes from the LVLM in your SAM2 prompt? I'm wondering what the difference in scores would be if you primed SAM based on the VLM output rather than just using a grid of prompt points.

The official implementations of SAM and SAM2 do not support text prompts, though this capability was discussed in the original paper [1]. Additionally, prior work on text-prompted segmentation for remote sensing (e.g., GeoPixel [2]) has shown limited generalization to classes outside the training set (see Figure 10, Appendix G.2). For these reasons, we use gridded point prompts in our experiments.

What is the maximum size of the inputs to the binary integer program? You state that it is solvable efficiently, but this is wrong in general (unless P=NP). I think it is important to justify that the problem size won't become pathological - or if it does, state the limit at which that will happen. The PuLP solver is good, but it doesn't scale that far.

Thank you for this insightful question. You are correct that BIP is NP-hard in general; our claim refers to it being tractable in practice for the problem scales encountered in our framework.

In our experiments, the problem size was modest. For example, on NWPU, the maximum number of mask proposals (N) was 200 and categories (M) was 8, with the solver finding an optimal solution in under 0.07s.

To further justify this, a multiple linear regression of the solution time against N and M yielded the equation: Time = -0.0052 + 0.00013*N + 0.00535*M. This model's high coefficient of determination (R² = 0.84) confirms that the solution time scales predictably and near-linearly within our operational range, ensuring practical efficiency.

References

[1] Kirillov, Alexander, et al. "Segment anything." CVPR. 2023.

[2] Shabbir, Akashah, et al. "Geopixel: Pixel grounding large multimodal model in remote sensing." ICML. 2025.

We thank you for great comments and hope our response clarified the paper's contribution to this community.

Weekness

Reliance on pre-trained models (SAM2, CLIP, LVLMs) introduces inherited biases. Sometimes SAM2 struggles with complex geometries and only provide patches in these complex geometries.

We acknowledge this limitation, as it is an inherent drawback of using a generic SAM2 for segmenting EO images. Recent research has developed SAM variants tailored for EO imagery [1, 2], though their pre-trained weights are not yet publicly available. We believe InstructSAM will benefit from any pre-trained models (SAM2, CLIP, LVLMs) models for EO images, thanks to its training-free design. This adaptability ensures that future advancements in EO-specific foundation models can be seamlessly integrated into our framework.

Limited validation on some goal-oriented instruction prompts. For example, given the prompts "detect flooded buildings", it's hard to find flooded building validation dataset. Therefore, it's hard to validate model performance for such kind of prompt. We agree the evalution of goal-oriented instruction prompts would be valuable. Here we further tested InstructSAM on FloodNet [3] (0.02m GSD, incorporating 9 classes covering diverse object of interests in remote sensing community). The LVLM is the key to understanding goal-oriented instructions. With structured prompt design and given the annotation rules, InstructSAM counts objects at a very high accuracy, while SegEarth-OV and OWLv2 failed to comprehend the subtle difference between flooded buildings and non-flooded buildings. However, given the clear texture presented in such high-definition drone images, SAM2 may segment an object into patches, reducing the recall of mask proposals and detection performance. This is consistent with the limitations we found in Appendix E.

Questions

Scalability with Larger LVLMs: InstructSAM uses GPT-4o (proprietary) and Qwen2.5-VL (open). How does it affect the performance with other large open LVLMs (e.g., LLaMA-3-70B, Gemini 1.5)?

We evaluated InstructSAM with other large open LVLMs, such as Gemma 3 and Mistral-Small, to assess scalability of InstructSAM. These results show that other open LVLMs, such as Gemma 3 and Mistral-Small, also achieve competitive performance. Notably, Qwen2.5-VL-72B significantly outperformed its 7B counterpart, demonstrating good scalability with larger models.

SAM2 Alternatives: SAM2 is central to mask proposals. Did you explore other lightweight alternatives (e.g., MobileSAM, FastSAM) for edge deployment? Also quantify trade-offs (speed vs. recall)?

Thank you for this valuable suggestion. We evaluated several variants of SAM2 to analyze the trade-offs between model size, inference time per image, and mask proposal recall. SAM2 Small strikes a good balance between model size and recall, making it a viable option for resource-constrained scenarios.

References

[1] Yan, Zhiyuan, et al. "RingMo-SAM: A foundation model for segment anything in multimodal remote-sensing images." IEEE Transactions on Geoscience and Remote Sensing. 2023.

[2] Shan, Zhe, et al. "ROS-SAM: High-quality interactive segmentation for remote sensing moving object." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[3] Maryam, Rahnemoonfar et al. "FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding" IEEE Access. 2021.

We thank you for great comments and hope our response clarified the paper's contribution to this community.

Minor Concerns:

Class legends in figures are very small. One has to zoom in a lot to actually read them.

Thank you for pointing this out. We will improve the display of legends in the final version.

Major Concerns and questions:

The included classes are very specific and do not cover major areas of interest in the Earth Observation community e.g land use land cover, crops, floods, burnt areas, tree species. Many of these categories have hierarchical structures e.g forest type, tree genus, tree species etc often resulting to finegrained classification tasks. Solving them would typically require varying sensors and Ground Sampling Distances as input. Including more diverse data could make the benchmark more representative for EO. Nevertheless, this work is an important first step towards this direction.

Thank you for acknowledging our contribution as an important first step. Our primary focus is on object recognition tasks, but we agree that extending InstructSAM to segmentation datasets is valuable for addressing broader EO challenges. To this end, we have conducted additional experiments on FloodNet and FLAIR datasets to explore its potential for land cover and flood segmentation tasks.

How were these datasets selected? The explanation given in L129 is a bit vague.

We appreciate the opportunity to clarify our dataset selection process:

• Diversity in resolution and modality: Current object detection datasets in remote sensing are predominantly high-resolution and RGB. NWPU-VHR-10 includes NIR bands, while DIOR spans a wide range of resolutions (0.3m to 30m), representing diverse GSDs.
• Annotation rules and user requirements: NWPU-VHR-10 is a subset of DIOR in terms of classes, but their annotation rules differ significantly, reflecting varying user requirements. For example, NWPU excludes vehicles in low-resolution images, while DIOR includes them. This makes these datasets ideal for evaluating a model's ability to follow user-specific instructions beyond simple category matching.
• Comparability with prior work: Both datasets have been widely used in previous zero-shot object detection and segmentation studies, such as ZoRI (AAAI'25) and DesReg (AAAI'24). We provide a detailed comparison with these methods in Appendix D1, Table 10.

It is not very clear why the benchmark was limited to these two datasets. Obviously, there are not many detection datasets for Earth Observation but there are definitely more than two.

We agree that there are other datasets available. However, we have found that NWPU-VHR-10 and DIOR are sufficient to demonstrate both the strengths and limitations of InstructSAM in optical object recognition:

• Strengths: InstructSAM excels at following diverse instructions and operates in a training-free paradigm.
• Limitations: Its performance is constrained by the capabilities of SAM and CLIP models, as detailed in Appendix E.

That said, we acknowledge the value of extending the benchmark to other modalities and tasks. This is an exciting direction for future work.

How can the benchmark be (potentially) extended?

The benchmark can be extended in several ways:

• Increase in class diversity: Adding more fine-grained categories, such as tree species or specific aircraft types.
• Incorporate additional modalities: Including SAR data to evaluate cross-modality generalization.
• Expand task scope: Extending to land cover classification and change detection tasks.

Extending the evaluation of InstructSAM to standard RS segmentation datasets outside of the benchmark and comparing its performance with standard baselines would be interesting and could help us identify its current limits. For example, how would it perform on pixel based land cover land use classification e.g on FLAIR.

Thank you for this suggestion. To adapt InstructSAM for land cover tasks, we made the following key modifications:

1. Area-based prediction: Since segmentation tasks involve "uncountable" objects, we used the LVLM to predict the proportion of each class in the image (e.g., 30% building, 70% brushwood).
2. Modified objective function: We added a penalty term to minimize the difference between the predicted and assigned areas for each class. The updated objective function is:

$$\min_{X} \frac{1}{N} \sum_{i=1}^N \sum_{j=1}^M (1 - s_{ij}) x_{ij} + \lambda \sum_{j=1}^M \left| \sum_{i=1}^N a_i x_{ij} - t_j \right|$$ $$s.t. \sum_{j=1}^M x_{ij} = 1, \quad \forall i \in {1, ..., N}$$

where $t_j$ is the target area for class $j$ predicted by LVLM, $a_i$ is the area of mask $i$, and $\lambda$ is the weight of the penalty term.

The zero-shot performance on FLAIR dataset was promising. InstructSAM-GPT4.1 outperformed the SegEarthOV baseline. The penalty significantly improved the accuracy of mask-label assignments.

The impact of the granularity of the target classes?

InstructSAM performs well for coarse-grained categories (e.g., "means of transport") but struggles with fine-grained distinctions (e.g., Boeing 777 vs. Airbus A320, or hockey field vs. soccer field). This limitation arises because current RS-specific CLIP models (e.g., RemoteCLIP, GeoRSCLIP, SkyCLIP) are not trained on semantically rich datasets. Incorporating high-quality, diverse training data (e.g., from OpenStreetMap tags) could significantly enhance their capabilities.

How does it adapt to varying environmental conditions?

This is an excellent point. The appearance of objects can vary significantly across geolocations and seasons. Adding geolocation and seasonal information to the prompt could improve instruction-oriented object counting. We plan to explore this in future work.

Extending the evaluation of InstructSAM to standard RS segmentation datasets outside of the benchmark and comparing its performance with standard baselines would be interesting and could help us identify its current limits.

We conducted additional experiments on the SARDet-100k [2] dataset to evaluate InstructSAM's performance on SAR imagery. These results highlight the challenges of applying current LVLMs and SAM2 to SAR data. Generic LVLMs struggle to interpret SAR imagery, and SAM2 fails to generate meaningful masks. This underscores the need for modality-specific adaptations in future work.

References:

[1] Li, Kaiyu, et al. "SegEarth-OV: Towards training-free open-vocabulary segmentation for remote sensing images." ​Proceedings of the Computer Vision and Pattern Recognition Conference​. 2025.

[2] Li, Yuxuan, et al. "SARDet-100k: Towards open-source benchmark and toolkit for large-scale sar object detection." Advances in Neural Information Processing Systems. 2024.

We thank you for great comments and hope our response clarified the paper's contribution to this community.