Segment Any Change

To Reviewer n4V1

W1: I doubt whether SAM has the ability to detect some very minor changes.

As you suggested, we demonstrate the case of tiny/minor changes, e.g., small vehicle changes. Please check Figure 4 in the rebuttal PDF. The main observation is that directly applying AnyChange to the original image overlooked these subtle changes (see the first row of Figure 4). After we bilinearly upsampled red box region by 2x and then applied AnyChange to it, we found some tiny/minor changes can be detected (see the second row of Figure 4). This observation shows that our method has the ability on tiny object change detection, while it is not perfect. Future works can take our method as a strong baseline and improve this point further.

W2: the robustness to the illumination/color changes and viewpoint changes

• illumination/color changes have been simulated by randomly applying color jitter to the pre and post-event images, respectively. We used ViT-B as the backbone for fast experiments. The results are presented as follows. The performance jitter of mask AR is less than 2% (-1.9%,+0.1%, -1.9%, -1.4%). We think this sensitivity to color variance is acceptable.

• viewpoint change cannot be simply simulated in image space, therefore, the above experiments cannot be conducted. Sorry for this. However, we indeed have considered evaluating the viewpoint robustness of our method since S2Looking is exactly an off-nadir/side-looking (can be seen as different viewpoints) building change detection dataset. Compared with our baselines, AnyChange performs better under viewpoint changes.

W3: Concerns about Table 3.

We believe there is a misunderstanding in Table 3. Sorry for the confusion. Table 3 presents a data-centric experiment to explore the possibility of AnyChange as a change data engine. The network architecture is fixed ChangeStar (1x96). The only difference in the comparison is the training labels used by the network. Our model (last row) is the network trained on the predictions produced by our AnyChange. The compared models (5th, 9th, 10th rows) are the same network trained with 100%, 1%, and 0.1% of ground truth labels (GT).

W3.1: in Table 3, why did not authors report results of using 10% GT and 100% GT?

In our original submission, we have reported the performance of models trained on 100% GT (see 5th row of Table 3). The quantitative gap between the labels generated by our AnyChange and 100% GT has been revealed. When reducing the number of GT to 1% and 0.1%, our result is better than both of them, although 0.1% (4.7x10$^6$ pixels) still greatly exceeded the number of labels we used for prompting (3.5x10$^3$ pixels). Therefore, we decided that testing the 10% GT case was unnecessary.

W3.2/Q1: the proposed method did not demonstrate the advantage over existing algorithms even though I know they used fewer annotations. The accuracy is far below the existing results, which is not acceptable.

• 100% GT comes from manual labeling, our pseudo-label is generated by AnyChange with point prompts in a zero-shot way. Therefore, it is reasonable and acceptable that 100% GT trained model has superior accuracy than ours. This is because pure manual labeling is the upper bound of our data engine based on AnyChange.

• The advantage of our method is 1) training-free. 2) enabling human-in-loop. 3) better zero-shot performance. 4) SOTA performance under the unsupervised setting. 5) better performance under the supervised setting with fewer annotations. All of these are not achieved by existing algorithms.

W3.3: upper bound of the proposed method when conducted in the same experimental setting.

We conducted this experiment by training a pure pixel-based ViT-B-based AnyChange model in a fully supervised manner, exactly as done with the other methods in Table 3. The F1, precision, and recall are 68.2, 71.8, and 64.9, respectively. This is the upper bound of AnyChange as a change detection network architecture.

W3.4: the meaning of "This confirms the potential of AnyChange as a change data engine for supervised object change detection."

Here we claimed that AnyChange can be utilized as a more efficient and interactive labeling tool, functioning as a change data engine, capable of providing superior change pseudo-labels with fewer manual labels as prompts. This claim is supported by Table 3 (9-11 th rows).

W4.1: The gap between AnyChange (Oracle)[F1: 62.2] and previous best result [F1: 67.9].

• AnyChange (Oracle) is an instance-level, promptable change detector, not specialized in pixel-level change detection. In Table 1, we need to evaluate it at both pixel and instance levels. Therefore, its trainable modules consist only of LoRA layers and a change score network. This setup allows us to determine the upper bound of embedding dissimilarity while preserving its promptability and zero-shot instance prediction capabilities. If we drop this setup and train a pixel-level specialist, it achieves a better F1 of 68.2%.

• Previous best is only a pixel-level change detection specialist with extra syntheic data pre-training and 100% GT fine-tuning.

W4.2: doubt the ability of the proposed method as authors said "Oracles obtained via supervised learning have superior precision"

We only claimed this point in Table 1. AnyChange (Oracle) with supervised learning has higher precision (metric) than its zero-shot counterparts.

To Reviewer eUYB

W1: How is the framework able to compute pixel level features for the mask proposals? Is there an interpolation step?

There is a bilinear interpolation to upsample the feature map back to the original image size. Then we compute the mask embedding by averaging each position's embedding of the mask proposal. This can avoid quantized errors caused by changing mask geometry to adapt the feature map.

W2.1: The model may be easily impacted by radiometric changes.

We simulate the radiometric changes by randomly applying color jitter to the pre and post-event images, respectively. The results are listed as follows:

W2.2:The experiments presented in the paper are on benchmark datasets and may not reflect practical applicability of the proposed framework.

The data of these four benchmark datasets were almost collected for real-world applications, e.g., urbanization, disaster damage assessment, natural resource monitoring.

• LEVIR-CD is designed for ordinary building change.
• S2Looking is designed for building change under side-looking / off-nadir / different viewpoint observation conditions. The data were collected from GaoFen (GF), SuperView (SV), and BeiJing-2 (BJ-2) satellites.
• xView2 aims to assess building damage changes, which includes 19 real disaster events with six disaster types (wildfire, earthquake, tsunami, hurricane, volcano, flooding). The data were collected from WorldView-2, WorldView-3, GeoEye-1.
• SECOND is designed for land-use/land-cover changes, including 30 change types. The object changes involve the categories including non-vegetated ground surface, tree, low vegetation, water, buildings and playgrounds.

We have demonstrated our method on practical apllications in Figure 1. The Disaster Damage Assessment cases of Figure 1 include 2023 Kalehe DRC Flooding (first row) and 2023 Turkey-Syria Earthquake (second row) events, which are not included in any public benchmark dataset we used.

W3: How does the matching algorithm handle overlapping objects with an image? Such as a tree canopy covering a part of the road. Averaging image embeddings in such cases might lead to erroneous results.

We have experimented the mentioned case. Thanks to SAM's strong generalization on object segmentation. The result shows the tree canopy and road segments are segmented as three independent parts, please check Figure 2 in the rebuttal PDF we newly uploaded. The tree canopy or road's embedding only encodes information belonging to themselves. At least in this case, there is no erroneous result. This implies that our matching algorithm does not need to deal explicitly with overlapping objects once the visual foundation model is strong enough in our framework.

Q1: Is the framework generalizable? Is SAM able to detect non-building related changes? All the experiments on the datasets shown are related to building change detection.

Yes, our design is general for change types. The SECOND dataset we have used in experiments is a land use/land cover change dataset, including 6 land-cover classes (non-vegetated ground surface, tree, low vegetation, water, buildings, and playgrounds) and up to 30 change types, extending beyond just building-centric changes.

To Reviewer SmiM

W1: ablation study for matching direction.

We have added this ablation study. The results are as follows:

We can find that the performance of single-directional matching is sensitive to temporal order, e.g., mask AR of two single-directional matching on LEVIR-CD are 1.3% and 35.9%, respectively. This is because the class-agnostic change is naturally temporal symmetric which is exactly the motivation of our bidirectional design. This result also confirms generally higher and more robust zero-shot change proposal capability.

W2: Figure 4 does not effectively demonstrate the model's efficacy because it is unclear if the model is segmenting all buildings or just the changed buildings.

Thanks for this good suggestion. We have updated Figure 4 as you suggested, which includes unchanged and changed buildings simultaneously. Please check Figure 3 in the rebuttal PDF we newly uploaded. It is more clear to show that AnyChange can more accurately detect building changes with the help of the point query.

minor grammatical issues: we will proofread thoroughly to fix them.

We are happy to see our responses addressed your previous W1 and resolved W2 to some extent. Thank you for your willingness to feedback on our updated figure.

For the observation From 1 to 3 points, false positives become more.

[potential reason] This is because it achieved a stable trade-off between precision and recall. When we move 1 to 3 points, we can find more building changes are recalled, along with the recall of other changes, e.g., vegetation to bare land, near buildings (right side of the figure).

[quantitative results: overall performance obtained improvement] While the emergence of false positives leads to a precision drop, the recall rate gains. This operation (from 1 to 3 point queries) resulted in improved overall performances measured by the F$_1$ score. These improvements were observed on three datasets (supported by Table 2, 3.6% on LEVIR-CD, 2.2% on S2Looking, and 3.0% on SECOND). Therefore, we think this positive trade-off is acceptable.

For a subjective concern about the "degree of recognized significance" raised by Reviewer MDhn

Anyway, we sincerely thank you for keeping your positive rating now, though you are impacted by that subjective opinion "degree of recognized significance" by Reviewer MDhn.

[objective facts: our model has a novel zero-shot change detection capability and superior performance on conventional benchmarks] Although we demonstrated our model with a novel capability of zero-shot change detection, put our zero-shot model into two conventional settings (unsupervised and supervised), and achieved new SOTA results, these objective facts seem not to persuade reviewers that our work achieved significant improvements in terms of new capability and new performance.

Objectively, as authors, we cannot and should not respond to this subjective opinion. We totally understand the difficulty of judging a work without any reference. This is why Reviewer MDhn held reservations even though Reviewer MDhn acknowledged the value of our work and his/her concerns have been addressed.

If subjective opinion can outweigh objective results and facts, we believe this is frustrating enough for all. Each of us is both a reviewer and an author.

Sincerely thank you again for your helpful comments, feedback, and warm support. Best wishes to you.

To Reviewer MDhn

W1.1: The novelty and uniqueness of this approach compared to existing cosine similarity-based methods (e.g., RaVAEn) are questionable.

Our AnyChange is fundamentally different from existing methods.

• Zero-shot vs. unsupervised. Existing unsupervised methods need to train a model to obtain visual representation, e.g., RaVAEn needs to train a VAE on specific data distribution to extract features, while ours belongs to zero-shot method without any training. Thus, our method is more resource-efficient, which is also appreciated by Reviewer SmiM.

• Interactive. Existing methods (including RaVAEn) do not support interactive mode, while our method can be interactive and used as an efficient label tool (supported by Table 3), thanks to our point query mechanism which is based on our bitemporal latent matching and SAM's point prompt mechanism. This implies our method supports human-in-the-loop. This point is also appreciated by Reviewer eUYB.

• Instance-level vs. pixel-level. Existing methods (including RaVAEn) outputs pixelwise change maps (raster), while ours produces instance change masks (polygon).

Our bitemporal latent matching differs from existing cosine similarity-based methods in three major points:

• Computational unit (instance vs. pixel). Existing methods, e.g., RaVAEn, adopt pixel embedding, while we use instance mask embedding, leveraging SAM's advantage. Our ablation study (Table 1, SAM+CVA match vs. AnyChange) indicates that this factor is key to the performance difference.
• Matching direction (bidirectional vs. none). Existing methods are pixel-based and do not consider matching directions. Ours is instance-based, and matching direction is a key design for instance-based methods. We reveal bidirectional matching performs better and more robust via the following ablation study (see also response to W1 of Reviewer SmiM).
• Integration with SAM's promptability. Integration with SAM's promptability has not been explored or even made possible by existing approaches, while we demonstrated a valuable integration with point prompt by our point query mechanism.

To the best of our knowledge, we did not claim the use of cosine similarity as our contribution. On the contrary, we explicitly stated that cosine similarity is a suitable choice for measuring similarity (Lines 174-175).

W1.2: The primary difference appears to be the use of SAM’s latent representations.

We anticipated that reviewers might be concerned about this point. Therefore, we included two baselines: SAM + CVA Match and SAM + Mask Match to ablate the impact of SAM’s latent. Our AnyChange and two baselines use the same SAM's latent representations, however, due to the different matching strategies, our AnyChange is superior (see Table 1 and Ablation: Matching Strategy, Line 251-261).

W2: The foundational observations Q1 and Q2 are only demonstrated with electro-optical images in the satellite domain, limiting the broader applicability of the proposed method to other domains or modalities.

• Our method can be also used in natural image domain. Please check Figure 1 in the rebuttal PDF.

• The idea of utilizing intra-image and inter-image similarities in SAM's latent space can potentially aid in solving any vision tasks that require multi-temporal inputs, which is appreciated by Reviewer SmiM.

• Modality limitation comes from SAM itself rather than our bitemporal latent matching and point query mechanism. Our design itself is modality-agnostic.

• Although our method is evaluated solely on optical satellite images, it still has a broad range of applications. The following five NeurIPS main conference papers all only focus on electro-optical images in the satellite domain.

  • [NeurIPS 2023] Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing
  • [NeurIPS 2023] SAMRS: Scaling-up remote sensing segmentation dataset with segment anything model
  • [NeurIPS 2022] SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery
  • [NeurIPS 2021] Spatial-Temporal Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis
  • [NeurIPS 2021] LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation

Q1: How did you set up the dataset and model to prove Q2?

The images were chosen from the LEVIR-CD dataset. We manually label all buildings of $t_2$ images as ground truth. The model is SAM (ViT-H) with default configurations. For $t_1$ image, we randomly select one building point as a point prompt and run SAM inference to obtain the building mask. For $t_2$ image, we adopt grid points (segment anything mode) to run SAM inference to obtain mask proposals. We compute all mask embeddings and then match $t_2$'s all mask embeddings by $t_1$'s building mask embedding, thus obtaining $t_2$'s building masks. Similarity is measured by cosine distance. Thresholding adopts the OTSU algorithm. The F$_1$ score and recall rate as metrics are reported to confirm Q2.

1. I acknowledge the points raised in your rebuttal regarding W1.1 and appreciate the clarification on the differences between AnyChange and existing methods. I agree that the aspects of "zero-shot" and "interactive" change detection, along with the shift to instance-level analysis enabled by SAM, are valuable contributions to the field. However, I still hold reservations about whether the novelty associated with utilizing SAM as a feature extractor, essentially inheriting these advantages, is sufficient to meet the bar for a NeurIPS publication. While the enhancements stemming from bitemporal latent matching and its integration with SAM's promptability are valuable, the core methodological contribution seems to heavily rely on the inherent capabilities of a pre-trained SAM model, a significant portion of which appears to be explored in existing works like [Chen et al., 2024] and [Oh et al., 2023].

• [Chen, et al., 2024] Chen, Hongruixuan, Jian Song, and Naoto Yokoya. "Change Detection Between Optical Remote Sensing Imagery and Map Data via Segment Anything Model (SAM)." In IGARSS. 2024.
• [Oh, et al., 2023] Oh, Youngtack, et al. "Prototype-oriented Unsupervised Change Detection for Disaster Management." In NeurIPS Workshop. 2023.

2. Regarding W2, I agree that focusing solely on optical satellite images doesn't inherently diminish a paper's contribution. As you pointed out, the papers you listed as examples demonstrate substantial novelty by leveraging specific characteristics of satellite imagery to overcome limitations in existing research. For instance, the adaptations made for self-supervised learning and the development of a novel super-resolution model directly address challenges and exploit opportunities presented by satellite data characteristics. My concern is that the current manuscript doesn't present a similar level of contribution in advancing the state-of-the-art. While bitemporal latent matching offers a novel post-processing approach for change detection using pre-trained SAM outputs, its novelty and impact, when compared to other NeurIPS publications, seem insufficient to warrant acceptance. The manuscript’s reliance on pre-trained SAM without significantly addressing the limitations or exploring new frontiers within the context of change detection, weakens its position as a strong NeurIPS publication.

We sincerely thank you for your helpful feedback and acknowledge our work's value. Appreciate! We have provided further clarification to your remaining concerns, hope it helps.

Point 1: The core methodological contribution seems to heavily rely on the inherent capabilities of a pre-trained SAM model, a significant portion of which appears to be explored in existing works like [Chen et al., 2024] and [Oh et al., 2023].

• For a training-free method, relying on a foundation model is completely acceptable, which is widely acknowledged by our community, e.g., there are many training-free image editing methods based on Stable Diffusion. Our method depends on SAM, however, our contributions are orthogonal to this visual foundation model itself. It is through the emergence of SAM that we can formally explore zero-shot change detection, which focuses on the generalization of unseen change types and data distribution. This is one of the most important problems in change detection.

• The two mentioned good papers indeed used SAM for change detection. However, it is completely different from our method from the task, motivation, core methodology, experimental design, and conclusion.

  • [Chen, et al., 2024] used OSM map to prompt SAM for image-map change detection.
  • [Oh et al., 2023] used SAM to extract object masks on a single pre-event image as a basic unit of voting post-processing (see their paper Sec 2.2, Refinement with SAM Model). This operation with their DINOv2-based change map is similar to what our baseline SAM + CVA match did. We will cite their paper for credit.
  • major difference 1: Neither of them exploited intra-image and inter-image semantic similarities in SAM's latent space, however, this is one of our main claimed contributions.
  • major difference 2: Neither of them achieved zero-shot instance change detection and benchmarked both pixel and instance-level performance.
  • major difference 3: Neither of them is interactive.

For Point 2, our motivation is just to clarify optical images have broader applications by giving five NeurIPS papers as evidence. Thank you for letting us know your previous concern W2 about "limiting the broader applicability" has been addressed.

For your further concern (this round) about the "current manuscript doesn't present a similar level of contribution in advancing the state-of-the-art."

• [SOTA results] Our zero-shot performance (Table 1), unsupervised performance (Table 4), and supervised performance (see W3.3 of Reviewer n4V1) all achieved state-of-the-art.

• To judge the contribution of research work, there is a fundamental difference between a so-far unexplored task (zero-shot change detection) and those well-established tasks (MAE-like self-supervised learning, and super-resolution).

• For the mentioned self-supervised learning and superresolution in the satellite domain, they all have well-established baselines and benchmarks, e.g., SatMAE improved over MAE (baseline) with their design. However, MAE has given a successful roadmap, i.e., masked image modeling and benchmark method (linear probing, fully fine-tuning).

• [new frontier] Zero-shot change detection remains an unexplored yet very important problem in the satellite domain without any well-established baselines and benchmarks. Our work provides the first zero-shot change detection roadmap, including problem formulation, baselines, state-of-the-art models (supported by Tables 1 and 4), and benchmarks.

• Last but not least, also as a NeurIPS reviewer, I subjectively think it is impossible to judge the contributions between publications on different topics. In the topic of change detection, we can confidently claim our work is groundbreaking.

Thank you once again for your time and effort. If you have any concerns, please feel free to share them here.

Thank you for your detailed and thoughtful responses to my concerns. I appreciate the further clarification provided, particularly regarding the distinction between your work and the cited papers, as well as the emphasis on the unexplored nature of zero-shot change detection in the satellite domain.

While I still have some reservations regarding the overall extent of contribution significance as I pointed out, I recognize the value of exploring and establishing a baseline for zero-shot change detection in the specific context of remote sensing. Please understand that this is not a dichotomous issue of whether there is a unique contribution or not, but rather a question of degree of recognized significance. Therefore, I will not raise further objections and am willing to conclude this discussion. I believe the points raised in both the review and the rebuttal will contribute to a balanced and informed discussion during the reviewer-AC discussion phase.

Thank you again for your efforts in addressing my concerns.

We also thought our discussion had finished with a good status. Hope you understand we have to respond to other reviewers when they have remaining concerns.

We have no intention of considering what the general standard of NeurIPS is. This is always a relative concept.

[relative contributions]. We anticipated that without a reference/baseline for zero-shot change detection, it would be difficult for reviewers to make judgments. Therefore, apart from our custom zero-shot baselines, we also put our zero-shot model into a common context (unsupervised and supervised setting) to evaluate its effectiveness. Tables 3 and 4 support this point.

[community contribution]. Our model will be the anchor point in zero-shot change detection, which can effectively avoid such difficult review processes. This will benefit the change detection community.

Our focus is still on nudging the door of zero-shot change detection, one of the most important problems in the multi-temporal remote sensing community. We identified this problem and resolved this problem via an extremely clean method we recognized (SAM with our training-free adaptation). This is an important baseline in zero-shot change detection, which should not be ignored.

Is there any good case to show a study's contribution though their model performance stems from a foundation model?

Exactly, countless outstanding works follow this paradigm in the generative AI community. For example,

• ControlNet (ICCV 2023 best paper) is a training-based adapter (essentially zero convolutional layers) for Stable Diffusion. Their performance stems from the decision to employ Stable Diffusion for conditional image generation. However, ControlNet exhibited outstanding and unprecedented control ability in their community. Please note that adding control-itself is also not a unique technical proposal in the image generation community.

Likewise,

• Our bitemporal latent matching and point query is a training-free adapter for SAM. Our performance stems from the decision to employ SAM for change detection. However, our work exhibited superior and unprecedented zero-shot change detection and interactive capability in our community. Our unique technical insight is to induce a novel change detection capability from SAM, which is not common sense in the remote sensing community. Without our method, there is no bridge between change detection (multi-temporal task) with SAM's interactive, zero-shot, instance-level capability (demonstrated on a single image task).

We fully respect the reviewer's opinion and hold our opinion with our evidence.

Finally, good luck to everyone.

While I intended to conclude my discussion on this paper in my previous comment, I have realized, after observing other reviewers' discussions, that my review and subsequent discussions with the authors have been frequently referenced. Therefore, I believe it is necessary to clarify my perspective further. I also feel compelled to add this comment for the sake of transparency and clarity, as I will maintain the following stance at the beginning of the reviewer-AC discussion phase.

Firstly, I acknowledge the authors' formulation of zero-shot change detection, the benchmarks and robustness analysis, and the technical soundness of the proposed "Bitemporal Latent Matching". Moreover, reading the discussions with other reviewers, I have solidified my opinion that the authors' contribution in this point is clear.

Secondly, despite this, and despite the discussions I have had with the authors, I still believe that the majority of the advantages of the 'interactive', 'automatic', 'zero-shot', and 'instance-level' change detection approach, which constitutes a significant portion of this paper's strengths, stem from the decision to employ SAM for the change detection task, rather than being unique technical proposals of this research. This decision itself is unprecedented (though related works exist) and holds novelty not found in existing literature. However, the authors and I have a fundamental disagreement on whether this aspect can be considered a sufficient technical contribution to the field of change detection that meets the general standards of NeurIPS.