Bringing SAM to new heights: leveraging elevation data for tree crown segmentation from drone imagery

Dear reviewer,

Thank you for your thoughtful review and helpful comments. We are pleased to hear that you appreciate the quality of this work and the importance of the problem we are tackling and that you highlighted that you “[like] the paper for reporting solid, honest, not exactly ground-breaking but ultimately useful results on a difficult and overlooked task.”

We respond to your comments and questions below.

W1- Splits: We will make the explanation on splits clearer in the paper. First, we would like to clarify that there are neither image pixels, nor annotations that belong to two different splits. Furthermore, while some sites are represented in both the testing and training sets in the Quebec Plantations dataset, we tried as much as possible to assign sites fully to splits and limit spatial autocorrelation. The only reason why some orthomosaics were divided into both the training and test sets is the distribution of species across sites. Indeed, we wanted to make sure that species of interest were both in the train and test set. This is the only time when we manually drew AOIs. However, as much as possible, the train/val/test regions were kept as spatially separate “blocks” ensuring no overlap between the splits. The table below shows in which splits the sites are assigned. Only the sites afcagauthmelpin and afcagauthier were broken up into the train and test sets. We will add a visualization in the Appendix of the splits for both the training and testing sets.

W3/Q1-Q2 - re an RGB-only version of BalSAM and explaining the photogrammetry pipeline to recover the DSM : We understand your suggestion of running an RGB-only version of BalSAM as a suggestion to use canopy height estimation from RGB images instead of the DSM as input to the model. We would like to clarify that the DSM is already a product derived from RGB imagery only, and is always available at no additional cost if drone RGB imagery is available. As a consequence, it is also much higher resolution than the imagery used by the models in the papers linked in your review. For example, the DSM on the Plantations dataset is 0.5cm/pixel resolution, while HRV canopy height maps are typically ~0.3m.

Structure-from-motion (Sfm) photogrammetry, which is used to create RGB orthomosaics from high-resolution RGB drone imagery, generates 3D photogrammetry dense point clouds, from which the DSM is derived. In essence, the approach is to:

1. Find key points in individual images (SIFT).
2. Match keypoints in image pairs to estimate camera locations and parameters (SfM).
3. Densify the 3D point cloud (MVS).
4. Project that point cloud to 2D to get the DSM.

We refer the reviewer to [1] for further details on the photogrammetry pipeline.

Fig 1. of this paper shows that the DSM is just a 2D projection of the dense point cloud, with interpolation to fill missing values when projecting the 3D points to a 2D grid (raster). Therefore, in some sense, BalSAM is already based only on RGB imagery. Please let us know if that answers your question.

As for the comment that “model comparisons are somewhat unfair, as not all baselines use the DSM”. Could you kindly clarify your concern? One of the axes of this study is to determine when and how the DSM is useful, which is why we explicitly compared the same methods with and without DSM input, as shown in our results tables 1, 2 and 3 with the checkmarks in the column “DSM”.

W2. “The DSM modality only brings marginal improvements and does not enable the model to consistently outperform simpler alternatives like RSPrompter.” A main goal of this study was actually to investigate whether the DSM helps for certain forest contexts and architectures. Our study shows 1/ that using the DSM alongside RGB imagery improves segmentation and classification mean results in the case of tree plantations, which have a specific structure, 2/ in mature forests, the DSM either improves results or keeps them comparable within error bars, 3/ while SAM out-of-the-box does not perform well, SAM-prompt based methods (RSPrompter, BalSAM), which have so far not been used for tree crown segmentation, reach best performance overall on all the datasets. This work is a first step towards guiding practitioners in their choices of models for forest monitoring, where different use cases may call for different methods depending on the nature of the data. BalSAM only has an additional 10M parameters compared to RSPrompter, which is a fraction of the total model size (the SAM image encoder alone has 632M parameters).

Q3 – re the focal loss: Upon your suggestion, we ran an experiment with Focal loss with Mask R-CNN on the SBL dataset (to be consistent with the setup of results reported in table 14 Appendix D.4.) The gamma parameter of the loss was set to 2. Results are reported below:

The model performs better than with the Weighted Loss, but not as well as when using Cross-entropy loss. We are currently running more experiments with other values of the gamma parameter.

Remarks: “[...] clarify the terms of "pretrained" across tables and models [...]”: For R-CNN based models, we explain in the text and tables captions that “pre-trained“ models refer to backbones initialized with ImageNet weights as opposed to trained from scratch. For SAM-based models (SAM, RSPrompter, BalSAM), we indeed use the weights from SAM but the image encoder and mask decoder are kept frozen in all configurations. We’ll make sure to highlight this better in the paper.

Formatting: Thank you for the very helpful feedback, we took this into account and will change figures to vector format and format the tables in a clearer way.

Many thanks for taking the time to review our paper and please let us know if you have further questions.

[1] Iglhaut J, Cabo C, Puliti S, Piermattei L, O’Connor J, Rosette J (2019) Structure from Motion Photogrammetry in Forestry: a Review. Current Forestry Reports 5: 155–168.

Dear reviewer,

Thank you very much for your insightful comments and valuing the importance of this application in forest monitoring and the potential for impact of our work.

We respond to your questions below.

Q1–Local maxima prompts: While there are indeed many ways to manually refine rules for filtering local maxima, we did not do any filtering. We originally experimented with setting thresholds on the DSM or tuning further the size of the neighbourhood to define local maxima. However, there is tall herbaceous vegetation in some sites, and well-tuned parameters on a site would not necessarily transfer well to other sites within the same dataset. Also, the DSM does not provide height with respect to the ground but rather relative height between objects in the image, and does not account for differences in terrain elevation. While it could be possible to normalize the DSM with respect to the lowest point in a site, if we have imagery from a site on a very angled slope, filtering local maxima based on a threshold would not necessarily bring much improvement. Moreover, having too many DSM prompts on tree crowns even if they are not in the center is not the main limitation of this method because each point is fed independently to SAM. If segmented objects overlap, we use NMS and only the object with highest confidence score is kept.

Q2–Discussing challenging cases and directions for future research: Thank you for your suggestions. Indeed, the BCI dataset is especially challenging because of the structure of the forest, including overlapping crowns; the lower resolution and quality of the imagery; and the long-tailed class distribution as highlighted in 5.1 and figure 4). We will provide further details and illustrative visualizations of predictions on the different datasets in the paper. Please see our response to Q3 for more on this topic, which is related to analysing errors of the models. Thank you also for your suggestions for improving the discussion section. We will provide a more comprehensive discussion on directions for future research, expanding on the ideas presented in the conclusion (L359-372). In particular, regarding exploring other ways of using the DSM, for the R-CNN based models, one could consider models using DSM prediction as an auxiliary task rather than as an input to the model. A different avenue is making use directly of the 3D point clouds that are used to obtain both the DSM and the RGB orthomosaics.

We also believe that adding contextual metadata into our models in the form of spatial information, spectral information, and the type of forest could improve the performance of tree crown instance segmentation models. In fact, while we have so far trained models separately on each dataset, as more high-resolution drone imagery data becomes available, it could be very interesting from an application perspective to learn representations on combined datasets at different resolutions, and location metadata could prove helpful, as it has been shown to in species distribution modelling contexts. Spectral information, available through e.g. satellite open-source programs, could also be added as input into the models even if it is not available at a resolution as high as that of the drone imagery, as additional signal.

Q3- “...what is causing error in the model for the different datasets” We already provided a number of elements for analysis on the Quebec plantations dataset but will make sure that we include more details and additional visualizations on other datasets as well. Regarding the Quebec plantations dataset, as illustrated in row 1 of Fig. 3, a case that is consistently missed across models is presented when trees crowns are very close to each other or don’t stand out much from the background. We have also observed errors in sites with tall herbaceous vegetation. As highlighted in the discussion, some classification errors align with human errors (e.g. distinguishing white and black spruces)
Across datasets, overlapping crowns are indeed challenging. As an ablation, we trained a Mask R-CNN with DSM inputs only (no RGB image) to assess its informativeness on the Plantations dataset vs the SBL dataset.

When looking at the drop in performance compared to the Mask R-CNN models using RGB images as inputs, we observe a much larger drop on the SBL dataset compared to the Plantations dataset. In fact, single-class mIoU remains high, which is another way to see that tree crowns are distinguishable from the background and from each other using the DSM only, and points to the usefulness of the DSM in plantation contexts.

Q4- application-readiness – Our study shows that using the DSM along the RGB imagery consistently improves segmentation and classification results for the plantation use case, therefore, we would recommend that practitioners looking to quantify the carbon stored in boreal plantations include the DSM information in their models. We leave it to practitioners to decide, based on their application and available data which models to use. For example, if one only has access to bounding boxes as annotations, we showed that Faster R-CNN+SAM is a reasonable baseline to consider and that including the DSM helped (on both the SBL and Plantations datasets). We will improve the discussion section with your feedback and outline more clearly future work directions with elements from our responses to Q2 and Q4 as well.

Thank you again for your time and reviewing our paper, and please let us know if you have any other questions.

Dear reviewer,

We thank you for your time and detailed review, and recognizing the importance of this forest monitoring application.

The goal of this work is two-fold: investigating the task of tree crown instance segmentation with respect to (i) whether the DSM helps for certain forest types and architectures, and (ii) when and how SAM is useful. Both questions have not been explored extensively before and our work is the first to compare models across those two axes of analysis, even though SAM is the most readily accessible model to many practitioners and the DSM is a product that is always readily available at no additional cost from the RGB imagery.

We show 1/ that using the DSM alongside RGB imagery improves segmentation and classification mean results in the case of tree plantations, which have a specific structure, 2/ in mature forests, the DSM either improves results or keeps them comparable within error bars, 3/ while SAM out-of-the-box does not perform well, SAM-prompt based methods (RSPrompter, BalSAM), which have so far not been used for tree crown segmentation, reach best performance overall on all the datasets.

Our work is a first step towards guiding practitioners in their choices of models for forest monitoring, where different use cases may call for different methods depending on the nature of the data.

We would also like to highlight that what might seem like small average differences in evaluation scores can translate to large differences in downstream applications. Consider the equations used in [3] to determine the amount of carbon stored in trees of different species of the form y_i = beta1_i* (diameter^beta2_i)*(height^beta3_i) where y_i is the biomass of part i of a tree (i = foliage, bark, …). The total biomass of the tree is the sum of the y_i. If one considers y_foliage, the coefficients (beta1, beta2, beta3) are (2.5, 2.45, -2.3) for sugar maple trees and (0.2, 2.38, -1.11) for white spruce - thus, making classification mistakes in an instance segmentation task can lead to huge differences in carbon estimates, especially for small trees (which is what we have in the Plantations dataset).

We respond to your questions and concerns below.

Weakness1/Q1/Limitation: While it is true that BalSAM does not always offer a significant advantage over RSPrompter, our results show that BalSAM either outperforms or performs similarly (within error bars) to RSPrompter across use cases. As outlined in the discussion (5.1), a reason for the lesser informativeness of the DSM on the BCI dataset could be the complex structure of canopies in tropical forest in which there is high overlap between tree crowns and species diversity, than in the Quebec Plantations dataset.

Regarding assessing the overall usefulness of the DSM, our study shows that the DSM proves useful in most of the experiments when added to RGB information. When the DSM is not helpful (e.g. Mask R-CNN baseline on the SBL dataset), this seems to be due to the structure of the forest which practitioners can assess (see examples of DSMs in figure 1).
We recommend practitioners to use the DSM along RGB images for plantation use cases where we showed that it consistently improves performance for all methods in terms of the multi-class mAP and wmAP scores. We then leave it to users to decide which method is more suited for other use cases, based on which data they have available. For example, if one only has access to bounding box annotations to train their model, the Faster R-CNN+ SAM method could be used and we showed that adding the DSM in this context could bring significant improvements compared to using RGB only.

Relatedly, we would like to respond to your concern in the “limitations section” about “limited availability of high-resolution DSM”, and clarify that the DSM is in fact always readily available at no additional cost when one acquires drone imagery. Structure-from-motion (Sfm) photogrammetry, which is used to create RGB orthomosaics from high-resolution RGB drone imagery, generates 3D photogrammetry dense point clouds, from which the DSM is derived. The 3D point cloud is actually needed for orthorectification to obtain RGB orthomosaic. In other words, the DSM is a by-product of the photogrammetry process and is thus available at no additional cost from RGB drone data processed with Sfm. This is a central point and a key motivation for our study, which we will further emphasize in the paper.

Q2. “... analysis of trade-offs between accuracy, inference speed, and training cost.” The table below shows average inference speed per sample for different models using the same V100-SXM2-32GB GPU. While we did not control for the type of GPU used across experiments and therefore training cost figures would be misleading, we report the number of trainable parameters in each model (this includes the final layer for the Plantations dataset, which has 9 classes). For models that can also take the DSM as a 4th channel along the RGB images as input, the addition of the DSM does not lead to a significant increase in number of parameters.

We would like to note that during inference time in real world scenarios, we do not have significant constraints (apart from using a single GPU) on the size of the models and their inference speed since the operation is mode offline. Typically, organisations looking into assessing the carbon stored in their plantations do not need embedded system models.

Weakness2/Q3. We thank the reviewer for raising the valid concern of comparing models with different capacities and suggesting to add experiments using more recent ViT-based models. We ran Mask2Former with Swin-base and Swin-L backbones on the Quebec Plantations dataset using RGB or RGB+DSM as input; we report results in the table below. The models have 106.9M and 215.5M parameters, respectively. They were initialized with COCO-pre-trained weights, trained for a maximum of 100 epochs (as were the other models presented in the paper) and took ~1.5 days to train on a single GPU. Given the timeline of the rebuttal, we performed only minimal HP tuning on the learning rate and learning rate scheduler. We find that the DSM improves performance for Mask2Former Swin-base. It is possible that results could be further improved and that Swin-L models require longer training. We are currently conducting further HP tuning with Mask2Former and will add results on multiple seeds in the main paper. That being said, several previous works have highlighted limitations of Mask2Former in the context of forest monitoring from remote sensing imagery. For example, in [1], the authors highlight: “There are several transformer-based successors to Mask-RCNN, from the DETRfamily of models to Mask2Former. While we experimented with these architectures, [...], we were unable to reliably outperform Mask-RCNN and found poorer convergence behaviour”. [2] also find that Mask2Former models’ “performance is not comparable to the CNN-based architectures” in the context of semantic segmentation on the SBL dataset.

Weakness3: “...encourage the authors to include more discussions around the useful signals to perform tree crown segmentation and provide more insights into the failure points of existing methods” Thank you for your helpful feedback. We will expand on the discussion and future work to cover these topics more comprehensively, as follows:

The usefulness of the DSM is (unsurprisingly) linked with the structure of the canopy. Models also tend to perform less well in contexts in which we have overlapping crowns. As an ablation, we ran a Mask R-CNN with only the DSM as input (no RGB imagery) on the Plantations and SBL datasets and report results below. We see a much larger drop in mIoU on the SBL dataset when comparing to Mask R-CNN models using image inputs, while on the Plantations dataset, mIoU remains high, which highlights the informativeness of the DSM in the plantations case.

We also believe that a promising avenue for future work is adding contextual metadata into our models in the form of spatial information, spectral information, and the type of forest, to improve the performance of tree crown instance segmentation models. While we have so far trained models separately on each dataset, as more high-resolution drone imagery data becomes available, it could be very interesting from an application perspective to learn representations on combined datasets at different resolutions. Location metadata could also prove helpful, as it has been shown to in species distribution modelling contexts. Spectral information, available through e.g. satellite open-source programs, could also be added as additional input signal into the models even if it is not available at a resolution as high as that of the drone imagery.

We hope this addresses your concerns and we are happy to answer any further questions you may have.

[1] Veitch-Michaelis, J., et al. "OAM-TCD: A globally diverse dataset of high-resolution tree cover maps."(2024)

[2] Ramesh, V., et al. "Tree semantic segmentation from aerial image time series." (2024)

[3] Lambert, M.-C., et al. "Canadian national tree aboveground biomass equations." (2005)

Dear reviewer,

Thank you for your helpful review and for valuing our contribution to the field of forest monitoring with this first study on applying SAM and integrating DSM information for tree crown instance segmentation.

We respond to your questions below.

W1/Q1 Indeed, the classification accuracy remains low overall on the BCI dataset. As we briefly mention in the discussion, we think this is due to the higher diversity of trees in this tropical context and to the lower resolution and quality of the images from this dataset compared to the other datasets. Additionally, annotations of the BCI dataset are noisy (as can be seen in Figures 1 and 3) and some tree crowns are not annotated. This reflects the complexity of the task of delineating their crowns and classifying them, even for experts in biology. The examples in Figure 3 (rows 3 and 4) show that our models sometimes segment trees that were not labeled. This may also lead to an underestimation of the performance of the methods. A potential avenue for tackling this challenge is to continue the work on using different losses that we present in D.3, as well as to explore more losses suited for long-tailed distributions.

W2/Q2 Upon your suggestion, we conducted experiments with Mask2former using the Swin-base and Swin-L backbones on the Quebec Plantations dataset using RGB or RGB+DSM as input. We report results in the table below. The models were initialized with COCO-pre-trained weights, trained for a maximum of 100 epochs (as were the other models presented in the paper), with batch size 8 and took ~1.5 days to train on a single GPU. Given the timeline of the rebuttal, we performed only minimal HP tuning on the learning rate and learning rate scheduler. We find that the DSM improves performance for the Mask2Former Swin-base model. It is possible that results could be further improved, and that models using the larger Swin-L backbone require longer training. Therefore, we are currently conducting further HP tuning with Mask2Former and will add results on multiple seeds in the main paper. That being said, several previous works have highlighted limitations of Mask2Former in the context of forest monitoring from remote sensing imagery.

For example, in [1], the authors highlight: “There are several transformer-based successors to Mask-RCNN, from the DETR family of models to Mask2Former. While we experimented with these architectures, [...], we were unable to reliably outperform Mask-RCNN and found poorer convergence behaviour”. [2] also find that Mask2Former models’ “performance is not comparable to the CNN-based architectures” in the context of semantic segmentation on the SBL dataset.

Regarding the suggestion to fine-tune SAM, this would unfortunately not solve the problem of instance segmentation - in which we also care about classifying the segmented instances, since SAM predicts masks without classes. Moreover, SAM’s image encoder has 632M parameters and fine-tuning SAM would require considerable compute resources, rendering its training process inaccessible to most forest-monitoring practitioners. In order to train RSPrompter and BalSAM on a single GPU, we used a batch size of 2, while fine-tuning SAM with limited computational resources would simply not be feasible.

W3: “The standard error of the result is relatively large, further robustification might be necessary in actual deployment.” We acknowledge that this is a limitation of RSPrompter and BalSAM, as we wrote in L360-361, and agree that further robustification is a priority for future work. This could be done by stabilising the training process through regularization and augmentations, especially on hard classes. We will also further analyse the errors and success cases in the best and worst runs from different seeds of the same model to better understand where issues arise and inform the choice of regularization and augmentations.

We hope that this answers your questions, and please let us know if you need any further clarifications.

[1] Veitch-Michaelis, J., et al. "OAM-TCD: A globally diverse dataset of high-resolution tree cover maps." Advances in neural information processing systems 37 (2024)

[2] Ramesh, V., et al. "Tree semantic segmentation from aerial image time series." (2024)