Question 4: Could you further articulate the scientific value of DroneBird, especially in terms of the new challenges it introduces for video-based research? Specifically, what novel research questions or difficulties does it bring?

Thank for the valuable question. We summarize the contributes and new challenges brought by DroneBird into the following four points:

1. Small target：We compared the number of foreground pixels occupied by independent targets in the DroneBird data and the existing crowd data at the same resolution, and the independent targets in DroneBird occupy fewer pixels (less than 20 pixels hight) than the independent targets in the existing dataset (more than 160 pixels hight). Note that although the counting target in the crowd counting task is the human head, the human body still provides important information and the human body occupies more pixels compared to birds, so our dataset provides data on small targets for counting. On the other hand, in order to avoid drone interference with bird activity, DroneBird's drone was photographed farther away from the flock, resulting in a further reduction of pixels occupied by the photographed individual birds.

2. Sparse distribution：Unlike humans, birds in nature need enough space to move around, which means birds are more sparsely distributed in space. Coupled with the smaller target size of birds, the sparseness of the foreground portion is even more pronounced in DroneBird compared to the already existing dataset.

3. Fast movement: Unlike existing video crowd-sourced data, the unique flight movements of birds allow them to do fast movements, posing a challenge for better temporal modeling.

4. New counting category:

   DroneBird fills the gap of missing video bird data in the field. In fact, the performance of the existing method on the bird counting dataset is much weaker than its performance on the crowd counting dataset. As shown in Table 1 in the main paper, the MAE and RMSE scores of most recent methods are commonly in single digits on crowd counting dataset, while they are mainly much larger than 30 in the bird counting task. For example, the MAE and RMSE of the Gramformer [3] on the FDST dataset are 5.15 and 6.32, respectively, while the MAE and RMSE on the DroneBird dataset are 49.11 and 65.50, which is a significant gap between the two counting tasks. It further verified the advantages of our proposed method over the existing methods. Our method achieves considerable performance on video crowd datasets while also leading existing methods on the more challenging DroneBird dataset.

   In addition, the models trained on crowd data fail to conduct bird counting directly (See Table below). We evaluated the performance of our method trained on the VSCrowd dataset (Exp. 1) and the DroneBird training set (Exp. 2) on the DroneBird testing set. The results show that the model trained on the VSCrowd dataset failed to perform well on the bird data, which indicates the large gap between crowd and bird data.

We believe that DroneBird is qualified to alleviate the current scarcity of bird data in the field, supporting further bird conservation activities.

Question 3: How is Exp. III in Table 2 implemented? Given my understanding, SAM's application without DEMO may lack practical relevance—how does it contribute in the absence of DEMO?

Thanks for the valuable question. Please kindly note that DEMO indicates the density-embedded masked modeling, which introduces the density map as an auxiliary modality to guide the masked patches reconstruction. SAM improves the masked modeling strategy by focusing on the foreground regions to overcome the sparse distribution of targets. Without DEMO, the model degrades to the ViT framework instead of Multi-MAE framework. Although it does not explicitly use density map as an auxiliary modality to instruct the patch reconstruction, we can also use the density map to perform adaptive and asymmetric sampling for the sparse fore- and back-ground.

Therefore, the masking guidance of the image provided by the density map in our SAM is an independent process and does not rely on the density map as an auxiliary modal input to the encoder (DEMO). After removing the DEMO, the token with only the image modality is fed into the encoder, and the token with the image modality is still masked according to its corresponding density map (SAM). Briefly, in Exp.III, the input of the E-MAC w/o DEMO is a partially masked image. The input of the E-MAC w/ DEMO is a partially masked image with a partially masked density map.

Question 2: Optical flow is typically more effective for capturing dynamic objects in scenes with a static camera. In the drone viewpoint of DroneBird, how does the model address the challenge where all objects in the field of view are in motion? Particularly, TCF's improvements are minor according to the ablation study—how might this be explained?

Thanks for the valuable question.

1. Depending on the actual requirements of the counting task and avoiding any impact on the bird's activities while capturing, our data requires the drones to remain stationary or move only slowly during capturing, so that there is no large motion impact for the camera.
2. In the ablated experiments, TCF is performed based on the baseline [4]. Since the baseline is simply performed on the pre-trained Multi-MAE weights, the single-frame estimation performance is limited. Thus, the multi-frame integration of TCF is hard to achieve a significant improvement. To fully validate the effect of TCF, we have added a new experiment by removing the TCF from the complete E-MAC (Exp. V in Table 2 of Sec. 5.3). In this experiment, the single-frame performance is promising. As shown in Table below, TCF improves the performance by 15%, which verifies the effectiveness of TCF.

Question 1: In Table 1, video-based methods underperform image-based methods on the DroneBird dataset. What might be the reasons for this discrepancy, and could this be further elaborated?

Thanks for the valuable question. We believe this may be due to the small proportion of birds in the images.

The data in DroneBird is captured from a drone's perspective, and to ensure that the birds' activities are not disturbed by the drone, the drone typically maintains a considerable distance from the bird flock, resulting in a small pixel proportion for individual birds.

1. Existing video methods focus more on how to leverage temporal information to improve model performance, not specifically designed for small-scale targets.
2. Single-frame methods often have special designs for small targets. For example, Gramformer [3] proposed a graph-modulated attention mechanism, which allows the model to focus on complementary information and effectively predict densities in smaller crowds, thus performing relatively better in counting small-scale targets.
3. Our method, considering the foreground-background imbalance issue, focuses more on foreground targets, thus copes to focus on a smaller proportion of prospect information and achieve superior performance.

Weakness 2: Although DroneBird introduces a new category, the dataset's difficulty and uniqueness are not strongly evident, and it lacks clear differentiation in research value compared to existing datasets beyond adding a bird-counting scenario.

Thanks for the valuable comment. Please kindly note that our DroneBird dataset is the first dynamic video counting dataset for small targets. The dynamic flying bird is significant smaller than traditional crowd counting objects. We calculated the number of pixels occupied by individual birds in the DroneBird dataset and the number of pixels occupied by individual humans in the existing crowd data. The number of pixels occupied by individual birds in DroneBird is significantly smaller than that occupied by individual humans in the crowd data (20 vs. 160), with the former being one-eighth of the latter.

In addition, the proposed dataset does not just add a new counting category. Compared to video crowd counting, birds move faster and tend to be more sparsely distributed in space, and these characteristics pose greater challenges. In fact, the performance of the present method on bird counting scenarios is much weaker than its performance on crowd counting datasets. As shown in Table 1 in the main paper, the MAE and RMSE of the existing method on crowd counting are only in single digits, whereas they are much higher than those on the bird counting task, which is a significant gap. For example, the MAE and RMSE of the Gramformer [3] on the FDST dataset are 5.15 and 6.32, respectively, while the MAE and RMSE on the DroneBird dataset are 49.11 and 65.50. Moreover, this further validates the advantages of our proposed method over the existing methods: our method is ahead of the existing methods in both crowd and bird counting tasks.

In addition, DroneBird indeed offers a new counting category. One of the significant contributions of our new dataset is providing new target categories for the counting domain. We observed that methods trained on other categories (mostly crowds) without fine-tuning or training directly failed to apply to bird counting tasks. As shown in the Table below, E-MAC trained on VSCrowd (Exp. 1) failed to perform well on the DroneBird testing set, while E-MAC trained on the DroneBird (Exp. 2) training set achieved a considerable performance. Therefore, an open bird counting dataset available for model training can provide additional data support for counting task research. The issue of bird conservation has been mentioned for a long time [1,2], and the proposal of DroneBird helps to assist in the analysis of bird activity, which in turn helps to conserve birds.

We sincerely thank the reviewer for your valuable comments and appreciate your recognizing on our novel bird counting dataset, accuracy-enhanced, efficiency, and feasible model. We provide detailed responses as follows:

Weakness 1: The methodology's components (such as DEMO, SAM, and TCF) lack cohesive integration, appearing independently implemented without clearly demonstrated necessity for crowd counting.

Thanks for the valuable comments. The proposed E-MAC is the first framework, which introduces the pre-trained vision foundation model to video object counting. We provide more explanations for the key components.

1. Aligning with foundation model (DEMO): The proposed DEMO first takes the density map as an auxiliary modality to guide the token reconstruction, aligning the video object counting model with the multi-modal structure of Multi-MAE.
2. Improve the masking strategy of DEMO by SAM: The proposed SAM is a key component of DEMO in our framework. We statistically analyze the sparsity in the video data and present the results in Fig. 12 of Sec. A.7. We found that there is typical sparsity in the existing video data, which leads to a significant imbalance of the fore-background information. Therefore, the masking strategy of DEMO that balanced selects the fore- and back-ground is inefficient. Instead, we improve DEMO with SAM to adaptively perform masking according to the density map, assigning high mask probability for the sparse foreground regions, which effectively improves the counting performance.
3. Multi-frame Fusion (TCF) for DEMO: TCF is proposed to avoid potential uncertainty in some frames and improve the overall counting performance. It extends DEMO to cope with video counting tasks, using optical flow to temporally correlate neighboring frames and fusing the front and back frame information by calculating the residuals of the neighboring frames.

References

[1] Bairlein, Franz. "Migratory birds under threat." Science 354.6312 (2016): 547-548.

[2] Lees, Alexander C., et al. "State of the world's birds." Annual Review of Environment and Resources 47.1 (2022): 231-260.

[3] Lin, Hui, et al. "Gramformer: Learning Crowd Counting via Graph-Modulated Transformer." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 4. 2024.

[4] Bachmann, Roman, et al. "Multimae: Multi-modal multi-task masked autoencoders." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

References

[1] Bachmann, Roman, et al. "Multimae: Multi-modal multi-task masked autoencoders." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[2] Zhang, Yingying, et al. "Single-image crowd counting via multi-column convolutional neural network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[3] Li, Yuhong, Xiaofan Zhang, and Deming Chen. "Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[4] Liu, Weizhe, Mathieu Salzmann, and Pascal Fua. "Context-aware crowd counting." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[5] Lin, Hui, et al. "Boosting crowd counting via multifaceted attention." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[6] Lin, Hui, et al. "Gramformer: Learning Crowd Counting via Graph-Modulated Transformer." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 4. 2024.

[7] Wu, Zhe, et al. "Spatial-temporal graph network for video crowd counting." IEEE Transactions on Circuits and Systems for Video Technology 33.1 (2022): 228-241.

[8] Hou, Yi, et al. "Frame-recurrent video crowd counting." IEEE Transactions on Circuits and Systems for Video Technology 33.9 (2023): 5186-5199.

[9] Fang, Yanyan, et al. "Locality-constrained spatial transformer network for video crowd counting." 2019 IEEE international conference on multimedia and expo (ICME). IEEE, 2019.

[10] Li, Haopeng, et al. "Video crowd localization with multifocus gaussian neighborhood attention and a large-scale benchmark." IEEE Transactions on Image Processing 31 (2022): 6032-6047.

Question 2: Why does the proposed method show higher RMSE on the VSCrowd dataset compared to GNANet in Table 1?

Thanks for the valuable question. Please kindly note that GNANet [10] adopts more frames than ours during prediction. Specifically, GNANet uses both the previous, current and next frames for temporal modeling (offline), while our method only uses the previous frame and the current frame (online). As a comparison, our method still maintains competitive performance in RMSE and performs better on most images with lower MAE compared to GNANet (27% improvement).

Question 1: In Table 1, do the compared methods use the same backbone network (ViT-B) as this paper's method? If not, please specify the reasons.

Thanks for the constrcutive question. Please kindly note that we are the first work to introduce the pre-trained vision foundation model to video object counting task. Although few of existing methods use the same backbone as ours, we have added experiments on introducing the pre-trained model to some existing methods, such as STGN [7] and Gramformer [6]. As shown in the Tabel below, STGN and Gramformer using the pre-trained MultiMAE weights only achieve 4.04, 4.19 (Exp. 1) and 5.15, 6.32 (Exp. 2) on MAE and RMSE, respectively. While our method achieves 1.29 and 1.69 of MAE and RMSE (Exp. 3), which is a significant improvement. In Exp. 5, although using the pre-trained vanilla ViT weights, our method still achieves 1.49 and 1.93 of MAE and RMSE.

Specifically, most of existing methods can be typically classified into three categories: extracting multi-scale features by improving the structural designs [2,3,4], focusing on the scale-variation problem by introducing attention-based methods [5,6], and temporal information ensemble methods [7,8,9]. These works focuses on addressing scale variations as well as temporal issues. Unlike these methods, we for the first time explore the impact of the founditional model on the counting task and present the first framework based on the pre-trained vision foundation model. We use the density map as an auxiliary modality for efficient sampling in the form of asymmetric sampling to achieve efficient learning of image-to-density map regression. We conduct a series of experiments (Sec. 5) to demonstrate the effectiveness of our method. Please kindly note that although the pretrained ViT model provides comparable performance, our E-MAC even further improve more than 40% (Exp. 4 vs. Exp. 5). Our approach extends the great potential of the visual foudation model for counting tasks and provides a new perspective for the community.

Weakness 2: It is recommended to provide code or critical sections of the code for review to enable thorough evaluation of the methodology and results.

Thanks for the valuable comment. All the code will be definitely released, and in fact our code is already ready. We have provided the code for the proposed method in the anonymous URL for review. The code includes the implementation details of the proposed method, pretrained weights and the evaluation process. We hope this will help reviewers better understand the methodology and results.

We'd like to thank the reviewer for the valuable comments, and acknowledgment of our method tackling the imbalance of fore-background effectively, versatility, and our dataset significantly broadens the application domain. We believe the constructive feedback will improve the paper and increase its potential impact on the community.

Weakness 1: The manuscript lacks visual analyses of the three key components, which would illustrate the targeted effectiveness of each module.

Thanks for the valuable comment. We visualized some of the predicted results of our key components (TCF, DEMO, SAM) and our baseline [1], as shown in Fig. 11 in Line 978 of the Appendix. We sequentially added our key components (TCF, DEMO, SAM) to the baseline, and observed that the visualization effects were continuously improving.

Additionally, we have also included some visual effects of the TCF module in Fig. 9 of Sec.A.5 in the Appendix. Additionally, we provided the difference of predicted density map w/ and w/o using SAM and the comparison of the number of foreground and background image patches entering the encoder w/ and w/o using SAM in Fig. 10 of Sec.A.6 in Appendix. We hope these visual analysis can help reviewers to better understand the effectiveness of each module.

1. In Fig. 9, we show the visual effects of the TCF module. Obviously, the TCF module effectively eliminates some key background interference.
2. In Fig. 10, we compare the prediction difference of the same image when trained with different masking strategies. The results show that the model trained with SAM has a more accurate prediction of the targets, while the model trained w/o SAM failed to predict some targets.
3. In Fig. 10, we also compare the number of foreground and background image patches entering the encoder w/ and w/o using SAM. It is evident that w/ SAM, there was a significant imbalance in the background image patches entering the encoder. Using only the foreground part resulted in slightly more foreground image patches than the background entering the encoder. Therefore, we introduced the background retention probability (BRP) to adjust the proportion of foreground image patches feeding to the encoder, ensuring a more balanced input of foreground and background image patches for model learning.

References

[1] Wu, Zhe, et al. "Spatial-temporal graph network for video crowd counting." IEEE Transactions on Circuits and Systems for Video Technology 33.1 (2022): 228-241.

[2] Hou, Yi, et al. "Frame-recurrent video crowd counting." IEEE Transactions on Circuits and Systems for Video Technology 33.9 (2023): 5186-5199.

Question 5: During training, $S_t=I_t,D_t$ contains the input image and GT labels. How does the model work in inference, without GT as input?

Thanks for the question. Since the training process requires masking operations on both images and density maps, the masking rates for images and density maps are obtained by a Dirichlet distribution. It is conceivable that in some cases, all image tokens are retained while all density map tokens are discarded. When all image tokens are retained, there is no need for GT to perform dynamic masking for images, which is consistent with the data input during the inference phase.

Please kindly note that, during the inference phase, we only input the images from the previous and current frames, indicating all the density map patches are masked. For the decoder, we use random tokens to complete masked tokens in $T_D$. We will emphasize this in the revision to avoid confusing.

Question 4: The description of SAM is also different from that in Fig. 2. The description takes the ground truth $D_t$ to mask tokens, while SAM in Fig. 2 involves $I_{t-1}$, $D_{t-1}$, and a random mask, which does not appear in Sec. 4.3.

Thanks for the valuable question. For the density map modality, at any stage of training, we use a Dirichlet distribution to determine the number of tokens to retain and apply random masking to the density map, as we mentioned in Line 243 of Sec. 4.2. To avoid confusing, we have provided more explanations in Sec. 4.3. We use the density map to guide the foreground masking probability of the corresponding image modality, thereby achieving adaptive masking. For the time steps t and t-1 in Figure 2, since t-1 is located in the first layer, we use the variables from time step t-1 to describe the SAM process in the figure. In fact, for time step t, the subscript of the variables in the SAM module should be changed from t-1 to t. To avoid confusing, we have standardized all the descriptions in figure and the main text. Thanks again for the constructive comments.

Question 3: in Eq.(2), $T_D \in B^{B\times N_D\times C}$ is a token set including vectors. How are these vectors sorted?

Thanks for the valuable question. We have also provided a detailed Figure (Fig. 8) in Sec.A.2 of the Appendix. Assuming we have got the number of tokens to be retained ($N_I^{ret}, N_D^{ret}$), we further explain our sorting operation on $T_D$ based on Fig. 8 as follows:

1. For an image-density map pair, the image and density map are divided into patches (for easier understanding, we divided the image and density map into 25 patches in Fig. 8).
2. Patches from the image and density map in the same position are paired. Patch pairs are then processed into token pairs ($T_I$, $T_D$).
3. For each patch pair, we use the integral value of the density map patch, which represents the number of objects in the image patch, to represent the weight of the corresponding token pair.
4. We sort the token pairs based on the weight calculated in the step. 3 and obtain the original token indices before sorting.
5. For $T_I$, we retain the top $N_I^{ret}$ tokens in the sorted $T_I$.
6. For $T_D$, we shuffle the order of $T_D$ and retain the first $N_D^{ret}$ tokens.
7. We then concatenate the retained tokens from $T_I$ and $T_D$ and shuffle its order to form the final token set.

Question 2: The TCF module presented in Fig. 2 is inconsistent with the corresponding description. In Eq. (1), the product of the "warp" operator is added to $\hat{D}\_{t}$ to generate $\hat{D}\_{\text{fuse}}$, and the "warp" operation involves $$. In the description, a cross-attention is applied to generate $\hat{D}\_t^{\text{res}}$, which is then combined with $\hat{D}\_t$ to produce $\hat{D}\_{\text{fuse}}$. In Fig. 2, the generation of $\hat{D}\_t^{\text{res}}$ does not involve $\hat{D}\_t$ according to the arrows that demonstrate the data flow.

Thanks for the valuable question. In Figure 2 and Eq. 1, we use a ''W '' to simultaneously represent the warp and cross-attention operations. In Figure 2, $\hat{D}_t$, $\hat{D}\_{t-1}$, and optical flow are all connected to ''W '', indicating the use of optical flow, $\hat{D}\_t$, and $\hat{D}\_{t-1}$ to generate $\hat{D}\_{res}$, which is consistent with Eq. 1. We have also provided the complete training pseudo code in the appendix for detailed explanation. The step-by-step operations of the TCF module are summarized as follows:

1. $O = \phi\_{opticalflow}(I\_t, I\_{t-1})$
2. $\hat{D}\_{t-1}^{warp} = \phi\_{warp}(O, \hat{D}\_{t-1})$
3. $\hat{D}\_{res} = \phi\_{ca}(\hat{D}\_t, \hat{D}\_{t-1}^{warp})$
4. $\hat{D}\_{fuse} = \hat{D}\_{res} + \hat{D}\_t$

We will emphasize this in the revision to avoid potential confusing. Thanks again for the constructive comment.

We would like to thank the reviewer for the thoughtful and thorough comments on our paper as well as for recognizing our new proposed counting dataset and well performance. We will also make an effort to increase clarity throughout. Below, we provide the point-to-point responses:

Question 1: In video counting, how is the count of a video defined? Is it the average count of objects, or the number of different instances that appear in the video?

Thanks for the valuable question. According to the previous works [1,2], the target of video counting is to predict the number of objects in each frame of the video. We follow this setting and define the target of video counting as the number of objects in each frame. For the evaluation of counting results, video counting calculates the difference between the predicted results and the ground truth for each frame, and then computes the mean absolute error (MAE) and mean squared error (RMSE) across all frames. For example, given a sequence of 100 video frames, the goal of video counting on this sequence of video frames is to predict the number of targets on each frame. For this sequence, the result of video counting will be an array of length 100, where each element represents the number of targets on the corresponding frame. Assuming the predicted result for each frame is $\hat{D}\_i$ and the corresponding ground truth is $D_i$, then the evaluation metric MAE (Mean Absolute Error) is calculated as $MAE = 1/100*\sum_{i=1}^{100} |\hat{D}\_i-D_i|$, RMSE (Root Mean Squared Error) is calculated as $RMSE = \sqrt{1/100*\sum_{i=1}^{100} (\hat{D}\_i-D_i)^2}$.

RQ1. The explanation is clear but it would be better to describe it in the main paper.

RQ2. It is confusing to use $\phi_{(\cdots)}$ to denote the operation in the pseudo code, but a "W in a circle" to represent these operations in Fig. 2. It would be better to replace "W" with $\phi$ in Fig. 2 or replace $\phi$ with $W$ in the pseudo code.

RQ3. It is clear in the response but not clear in the paper:

1. I noticed that you do not mention these details in the revised paper (description about Fig. 8). According to the added content (from line 856 to line 859), I still cannot understand it. You'd better add these contents to the paper since it is really confusing if there is no detailed description.
2. Accordingly, BRP in line 269 is actually the count in each patch? No matter right or wrong, it would be better to describe it in the paper. BRP is confusing but the response here is clear.
3. Accordingly, Eq.(2) should be formulated according to your description. Eq.(2) does not provide any information about how to compute BRP or the count in the patch.
4. What is the work of the shuffle operation in Fig. 8? For a transformer-based decoder, I think the precedence of tokens does not affect the computation formulation and each token should also be embedded with positional information.

RQ4. No, you misunderstand my meaning. In Sec. 4.1, you use $D$ to denote GT (line 212), and use $\hat{D}$ to represent the predicted density map (line 221). In line 265, you use $D$ here, so I consider you are using GT as input.

RQ5. Accordingly, "For the decoder, we use random tokens to complete masked tokens in $T_D$."

1. How is the mask implemented? In line 15 of Algorithm 2?
2. Will the performance be affected if different random seeds are used?

What is the $\mathbb{N}$ in Eq.(2)? What is the relationship among $\mathbb{N}$, $\mathcal{N}_I^\text{ret}$ and $\mathcal{P}$?

Thanks for your quick feedback and the invaluable questions.

We denote $\mathbb{N}$ as a random variable that follows a Uniform distribution between $0$ and $1$, which is produced by a random number generator (random.random()), as shown in the second row of the following code. We have also updated the main paper in Line 282-283 to provide detailed descriptions.

if dec_order == None:   # Line 203 in emac/emac.py
    dec_order = random.random() <= 0.8  # 0.8 is a pre-set value of 1-P
noise_for_rgb = torch.nn.functional.unfold( 
    rearrange(ref_mask, 'b c h w->b c h w'), kernel_size=16, stride=16
).sum(dim=1)    # \phi_{sum}(D^i_{patch})

ids_arange_shuffle_rgb = torch.argsort(
    noise_for_rgb,
    dim=1,
    descending=dec_order,
)   # argsort(\phi_{sum}(D^i_{patch}))

In the code implementation (Line 203-213 in emac/emac.py of code), we take $1-\mathcal{P}$ as a threshold to perform the descending order sorting, where $\mathcal{P}$ is the background retention probability. Specifically, when the value of $\mathbb{N}$ is less than $1-\mathcal{P}$, the sorting operation is performed in the descending order; otherwise, it is performed in the ascending order.

Please kindly note that $\mathcal{N}_I^\text{ret}$ is the number of retained image tokens, and we retain the first $\mathcal{N}_I^\text{ret}$ image tokens after the sorting, as presented in Line 897-900 of the supplementary material.

Weakness 2: For the proposed Temporal Collaborative Fusion, it is unclear why we need to calculate $\hat{D}^{res}\_t$ again after obtaining $\hat{D}\_{t}$ and $\hat{D}\_{t-1}$ from the decoder. If $\hat{D}\_{t}$ and $\hat{D}\_{t-1}$ are not accurate enough, then $\hat{D}\_{t}^{res}$ calculated using them will also be inaccurate. Conversely, if $\hat{D}\_{t}$ and $\hat{D}\_{t-1}$ are accurate, there seems to be no need for the Temporal Collaborative Fusion Module.

Thanks for the valuable comment. $\hat{D}\_{t}^{res}$ is predicted by a cross-attention layer, in which $\hat{D}\_{t}$ is taken as query and $\hat{D}\_{t-1}$ is taken as key and value. Please kindly note that $\hat{D}\_{t}^{res}$ is designed to provide temporal information for $\hat{D}\_{t}$.

More specifically, due to the potential inaccurate prediction for some extreme scenarios, we proposed the Temporal Collaborative Fusion model to improve the performance. Actually, the Robustness and accuracy can be effectively improved through multi-frame ensemble, which is a generally adopted approach [4,5,6]. Our TCF aims to provide corrections to the original prediction results to improve accuracy. Different frames may contain more comprehensive information, which could be integrated to produce more accurate results. We provide a visual analysis of the results of the TCF in Sec. A.5 of the Appendix, from which it is clear that the TCF removes some of the background interference. We have also added an experiment to compare the performance of E-MAC with and without TCF, as shown in the table below, the addition of TCF improves performance by 15%. Therefore the design of TCF is reasonable and effective.

References

[1] Dosovitskiy, Alexey. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).

[2] Liu, Chengxin, et al. "Point-query quadtree for crowd counting, localization, and more." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Bachmann, Roman, et al. "Multimae: Multi-modal multi-task masked autoencoders." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[4] Pengyu Zhang, Jie Zhao, Dong Wang, Huchuan Lu, Xiang Ruan; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 8886-8895

[5] Zhang, Lu, et al. "Weakly aligned cross-modal learning for multispectral pedestrian detection." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[6] Kappeler, Armin, et al. "Video super-resolution with convolutional neural networks." IEEE transactions on computational imaging 2.2 (2016): 109-122.

[7] Liu, Weizhe, Mathieu Salzmann, and Pascal Fua. "Estimating people flows to better count them in crowded scenes." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16. Springer International Publishing, 2020.

[8] Avvenuti, Marco, et al. "A spatio-temporal attentive network for video-based crowd counting." 2022 IEEE Symposium on Computers and Communications (ISCC). IEEE, 2022.

[9] Wu, Zhe, et al. "Spatial-temporal graph network for video crowd counting." IEEE Transactions on Circuits and Systems for Video Technology 33.1 (2022): 228-241.

[10] Lin, Hui, et al. "Gramformer: Learning Crowd Counting via Graph-Modulated Transformer." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 4. 2024.

Question 1: As we all know that MAE pre-training can improve many downstream task fine-tuning. For video crowd counting, does the current approaches also benefit from the MAE pre-trained models (e.g., MAE ViT-base)? i.e., using them as the initialization weights for video counting fine-tuning.

Thanks for the valuable question. We have provided a comparison of the performance of our model and the vanilla ViT model when fine-tuning w/ and w/o the MultiMAE pretrained weights.

1. Fine-tuning with pretrained weights indeed improves performance, but the improvement is limited (Exp. 1 vs. Exp. 2 and Exp. 3 vs. Exp. 4 in the Table below).
2. In contrast, our unique design tailored to the data and video counting tasks significantly enhances performance (Exp. 1 vs. Exp. 3 and Exp. 2 vs. Exp. 4 in the Table below).
3. We attempted to replace the backbones of some existing methods [9,10] with pre-trained ViT, but observed varying degrees of decline in performance metrics (Exp. 5 and 6 in the Table below). This might be due to the large number of parameters in ViT, and the existing methods were not specifically designed to accommodate this.

Our approach, however, efficiently learns through density map representation learning with DEMO and foreground-background balance using SAM. Please kindly note that our method is the first work to explore the potential of a pre-trained vision foundation model for video object counting, which provides a new perspective to the community.

Weakness 3: A comparison of model complexity with the other approaches in Table 1 should be provided to ensure a fair evaluation, as the current work may use a relatively larger model.

Thanks for the valuable comment. We have compared the number of parameters and FLOPs with some previous methods and reported the results in the Table below. It is worth noting that the FLOPs of our method and the competing methods are comparable, although we first adopted the vision foundation model. Compared to EPF [7] and PFTF [8], E-MAC achieves superior performance with less required computations. Compared to STGN [9], our method achieved a 58% performance improvement with only 9% more FLOPs.

We thank the reviewer for recognizing our helpful bird counting dataset, interesting framework and SOTA performance. We appreciate your support and constructive suggestions and address your concerns as follows.

Weakness 1: Ambiguous performance gains: The proposed approach integrates a masked autoencoder for representation learning and formulates the previous density regression task as "density reconstruction" within an MAE-like framework for crowd counting. Additionally, the authors use the pre-trained ViT-B from MultiMAE as the encoder for discriminative feature extraction. However, given that pre-trained MAE models typically lead to significant performance improvements during downstream task fine-tuning, it remains unclear whether the observed performance gains stem from the designed new masked autoencoder-based framework or the usage of pre-trained MultiMAE. One suggestion: compared w/ a variant w/o using the pre-trained ViT.

Thanks for the valuable comment. We provide more explanations for the effect of the proposed new masked autoencoder-based framework in two aspects:

1. It is difficult to train a well-performing ViT from scratch using the existing counting dataset. Please kindly note that using the pre-trained ViT alone is hard to achieve considerable counting performance. We have compared the performance of vanilla ViT [1] with the existing method (PET [2], Table 1 in Sec.5.2 of the main paper) in the Table below. As a comparison, vanilla ViT (Exp. 2 in Table below) is significantly worse than PET [2], which demonstrates the contribution of solely using a pre-trained model is limited. The experiments with pre-trained MultiMAE [3] (Exp. 3 in Table below / Exp. I in Table 2 of the main paper) also show similar results. Instead, the proposed E-MAC performed based on the pre-trained model achieves 1.29 and 1.69 of MAE and RMSE, which is an unprecedented improvement of 47% over MultiMAE. This fully demonstrates the effectiveness of our method.

2. Besides, we have added experiments on vanilla ViT (Exp. 2 in the Table below) and E-MAC w/ vanilla ViT pre-trained weights (Exp. 4 in the Table below). Compared with the ablated experiment Exp. I and Exp. V (Table 2 in Sec.5.3 of the main paper), although using the pre-trained MultiMAE weights further enhances the performance, the performance improvement of our method mainly comes from our uniquely designed E-MAC. These experiments validate our effectiveness.

   It is worth noting that we first introduce the pre-trained vision model to the video counting object task, and propose a novel E-MAC framework, which treats the density map as an auxiliary modality to guide the mask modeling, achieving superior performance and offering a new perspective to the community.

   No.	Model	MAE $\downarrow$	RMSE $\downarrow$
   1	PET [2]	1.73	2.27
   2	vanilla ViT [1]	2.63	3.31
   3	MultiMAE (Exp. I in Sec.5.3)	2.45	3.22
   4	E-MAC w/ vanilla ViT	1.49	1.93
   5	E-MAC w/ MultiMAE (Exp. V in Sec.5.3)	1.29	1.69