Retrieval-based Zero-shot Crowd Counting

Q4. Comparison With State-of-the-Art Methods
The comparison methods used in the paper were the latest methods to the best of our knowledge at the time of submission. Since there is no mention of the missing comparisons, we believe the reviewer is referring to "Robust Zero-Shot Crowd Counting and Localization With Adaptive Resolution SAM" [4].

The method uses the foundation model "Segment Everything Everywhere Model" (SEEM) to produce the localization annotations from the segmentation results. Next, these annotations are used to train a crowd-density prediction network to perform both crowd counting and localization. However, [4] is not a fair comparison against the proposed method or annotator-free methods for the following reasons, even though [4] does not rely on ground truth annotations.

1. SEEM is a promotable panoptic segmentation foundation model. Therefore, SEEM already acts as a weak object detector to produce the localization proposal. Unlike other comparison methods in table 1, [4] relies on another downstream task model to produce the annotations.
2. SEEM is trained with human interaction in the loop and supervised labels to create semantic awareness. Therefore, [4] is not purely annotation cost-free, unlike the proposed method or other comparisons used in table 1.

Therefore, [4] is not a fair comparison against other annotator-free crowd-counting works.

Q5. Theoretical Foundations
To explain the contribution of knowledge augmentation to improving zero-shot crowd counting, we use a probabilistic approach.

The goal is to predict the crowd count $c_i$ for the target embedding $\mathbf{e}_i$. Using a probabilistic framework, the prediction can be expressed as:

$P(c_i|\mathbf{e}_i) = P(c_i|\mathbf{e}_i^{'}),$

where the augmented embedding is:

$\mathbf{e}_i^{'} = \mathbf{e}_i + \mathbf{v}_i^L + \mathbf{v}_i^{LV} + \mathbf{v}_i^{VV}.$

Using Bayes' rule, we can rewrite the probability as follows:

$P(c_i|\mathbf{e}_i^{'}) \propto P(\mathbf{e}_i^{'}|c_i)P(c_i),$

where $P(\mathbf{e}_i^{'}|c_i)$ and $P(c_i)$ denote the likelihood of the augmented embedding given the count and the prior probability of the count derived from the source distribution.

The likelihood can be decomposed as:

$P(\mathbf{e}_i^{'}|c_i) \propto P(\mathbf{e}_i|c_i) \prod_n P(\mathbf{v}_i^n|c_i),$

where $\mathbf{v}_i^n$ is each individual augmentation type from the KAM. However, each individual augmentation is computed from the KAM using the retrieved embeddings from the reference database. Therefore, the likelihood can be updated as:

$P(\mathbf{e}_i^{'}|c_i) \propto P(\mathbf{e}_i|c_i) \prod_n \prod_k^K P(r_k^n|c_i),$

where $r_k^n$ denotes the retrieved embedding augmented with multi-head attention (MHA), and $k$ is the index of the retrieved embedding. Each $**v**_i^n$ thus encodes the aggregated likelihood information from its corresponding patches, ensuring that $\mathbf{e}_i^{'}$ effectively aligns with the count $c_i$ as MHA behaves as a projection of the query embedding to the key embedding. Consequently, the retrieved embeddings $**v**_i^n$ encode domain-specific patterns, improving the likelihood estimation.

Without the retrieved embeddings, the likelihood distribution will only depend on $**e**_i$, and as augmentations are introduced, the likelihood distribution is influenced by the source domain information. The influence of the source likelihood increases with the number of retrieved embeddings. In return, the posterior distribution $P(c_i|**e**_i^{'})$ becomes a sharper posterior distribution. As the posterior distribution becomes sharper, the uncertainty involved with the prediction reduces, improving the prediction accuracy.

References
[4] Jia Wan, Qiangqiang Wu, Wei Lin, and Antoni Chan. Robust zero-shot crowd counting and localization with adaptive resolution sam. European Conference on Computer Vision, pp. 478–495. Springer, 2025.

Thank you for your thoughtful and constructive feedback on our manuscript. We appreciate your suggestions to test on more diverse datasets, provide detailed ablation studies, analyze computational complexity, compare with the latest state-of-the-art methods, and delve deeper into the theoretical foundations of our approach.

Q1. Crowd-counting Benchmarks
The public datasets used in the paper are prominently used in the field of crowd-counting. Furthermore, these five datasets are also used by other ground-truth annotation-free methods such as CrowdCLIP [3], CSS-CCNN [1], AFreeCA [2], which also only use the above public benchmarks.

Furthermore, we provide the performance comparison for the NWPU dataset in table 2, which is a public evaluation benchmark that has not released the test ground truth. The NWPU benchmark evaluates the complexity of different scenes and crowd density. We provide the performance comparison for different scene levels for the NWPU dataset in table 2, and the scene level index is proportional to the crowd density of the scene (e.g., S1 is low crowd density, and S5 is high crowd density). Therefore, we believe the amount of datasets presented in this paper is sufficient to convey the robustness and contribution of the proposed work.

Q2. Detailed Ablation Studies
In the proposed work, the pipeline is comprised of three main components: image encoder, Knowledge Augmentation Module (KAM), and count decoder. The key component of the proposed framework is the KAM. Furthermore, the KAM module has three augmentations: $\mathbf{v}_i^L$, $\mathbf{v}_i^{LV}$, and $\mathbf{v}_i^{VV}$. Therefore, we could consider the contribution of each individual augmentation type to the overall performance as well as the KAM as a whole unit. This performance boost of each item should be measured against a baseline where none of the augmentations is used.

We have already given the performance for the baseline method in row 4 of table 3, where the image encoder is pre-trained, as explained in section 3.1, and the count decoder is fine-tuned with simulated data without using the retrieval process. Then, the contribution from the KAM to the overall performance is given in row 5 of table 4, which quantifies the improvement of the proposed KAM. Additionally, we have provided the performance gain from using the augmentation $\mathbf{v}_i^L$ in table 4, along with other augmentation schemes. However, in the following table, we tabulate the performance of each individual augmentation type and the KAM module against the baseline method.

Table A: Ablation of augmentation schemes.

Q3. Computational Complexity Analysis
We provide a comparison for the inference speed in Table 6 in the supplementary material. However, we will itemize the inference time and the computational complexity for the model with and without the KAM, along with the accuracy. For the proposed method, the inference time and computational complexity are influenced by three components: image encoder and count decoder, knowledge retrieval process, and KAM.

Table B: Computational complexity and inference time for different components.

The MAE performance for the JHU public dataset is given in the above table. The baseline corresponds to the network without the KAM and the minimum model latency without the proposed improvements. The MIPS corresponds to the retrieval process with the inner product search to find the 32 nearest neighbors for a given image embedding.

References
[1] Deepak Babu Sam, Abhinav Agarwalla, Jimmy Joseph, Vishwanath A Sindagi, R Venkatesh Babu, and Vishal M Patel. Completely self-supervised crowd counting via distribution matching. European Conference on Computer Vision, pp. 186–204, 2022.
[2] Adriano D’Alessandro, Ali Mahdavi-Amiri, and Ghassan Hamarneh. Afreeca: Annotation-free counting for all. arXiv preprint arXiv:2403.04943, 2024.
[3] Dingkang Liang, Jiahao Xie, Zhikang Zou, Xiaoqing Ye, Wei Xu, and Xiang Bai. Crowdclip: Unsupervised crowd counting via vision-language model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2893–2903, 2023.

Thanks for the author's rebuttal.

I have re-read this paper and combined with the comments of other reviewers, but I still deem that this paper does not meet the criteria for acceptance.

I reread the paper, and now I can understand most of the problems I commented on above. However, the writing should be improved.

The following problems still confused me:

1. The text description contains three keys: [weather condition], [crowd count], and [time of day]. Is there any ablation study demonstrating how these three keys affect the retrieval effectiveness?
2. How is KAM supervised? the output of KAM is a embedding (\mathbf{e}_i) in Eq.(6), there is not connection between it and the count c_i in Eq.(7).
3. In Table 4, the ablation studies on (v_i^{L}), (v_i^{LV}), and (v_i^{VV}) are conducted. If all of them are not involved, but only (\mathbf{e}_i) is used as the embedding, how is the performance?

Thank you for your insightful and detailed feedback on our manuscript. We appreciate your suggestions to include an ablation study on the impact of the three text keys, clarify the supervision mechanism for KAM, and analyze the performance when only the embedding $(\mathbf{e}_i)$ is used.

Q1. Ablation Study on Keywords
The retrieval process only considers the image embeddings to perform the maximum inner-product search and extract the closest neighbors. The text description is not used in the retrieval process and is only used in the knowledge augmentation step. Also, we did not perform an ablation on the effect of the three keywords in our experiments originally. However, we agree that ablation of the effect of the keywords is beneficial to understand whether the additional context is important, as suggested by the reviewer.

We performed three experiments with the Knowledge Augmentation Module (KAM) setting corresponding to row 4 in table 4. The first ablation was performed with only the "crowd count" as the keyword. Then, we considered the "time of day" as the only additional keyword. Lastly, we considered the "weather" as the only additional keyword.

Table A: Ablation of keywords.

The performance for the three ablations was similar to the baseline performance for the ShanghaiTech-A (SHA) and UCF-QNRF (QNRF) datasets. Therefore, the additional context keywords did not significantly affect the performance. However, for the JHU-Crowd++ (JHU) dataset, using the "time of day" keyword improved the performance, unlike the "weather" keyword. Therefore, the "weather" keyword could be removed from the text prompt even though there is no significant improvement to the inference speed or computational cost. However, the "time of day" keyword cannot be simply discarded.

The difference between SHA, QNRF, and JHU is that the JHU is a large dataset compared to the other two and contains a distribution of scene illuminations, whereas the other two datasets mostly have bright scenes. Since the training of the KAM and the count decoder are similar among these three datasets, the performance difference should come from the ability of the model to generalize. Since SHA and QNRF have a skewed illumination distribution, the "time of day" keyword has no effect. But, for the JHU dataset, the "time of day" keyword can provide context on the level of illumination as the "morning" and "evening" labels allude to bright illuminations, as opposed to the "night" label.

We will add the ablation of the effect of the keywords on performance in the revised version.

Q2. Supervision Mechanism for KAM
The KAM is supervised during the fine-tuning stage along with the count decoder. The embedding $\mathbf{e}_i$ is the output of the image encoder, and the final output of the KAM is the augmented image embedding $\mathbf{e}_i^{'}$ given by equation (6). The input to the count decoder is $\mathbf{e}_i^{'}$, which connects the KAM and count $c_i$ to supervise the augmentation module. Hence, the relationship between $c_i$ and $**e**_i$ can be written as $c_i = f(\mathbf{e}_i + \mathbf{v}_i^L + \mathbf{v}_i^{LV} + \mathbf{v}_i^{VV}),$ where $f$ is the count decoder.

Q3. Performance with $\mathbf{e}_i$ Only
When the augmentations are removed, the input to the count decoder will be the output of the image encoder, which is $\mathbf{e}_i$, and the count decoder is then fine-tuned to map $\mathbf{e}_i$ to $c_i$ using the simulated or synthetic data. This is the baseline method to analyze the contribution of the KAM and different augmentation methods. The performance of the baseline method is given in row 4 of table 3. However, we will add the performance of the baseline method to table 4 to improve the informativeness of the table in the revised version.

We sincerely believe we have addressed the reviewer’s questions, including thorough explanations, additional ablations, and justifications.

As the deadline for the discussion period approaches (November 26th), we wanted to gently follow up to see if you've had the opportunity to review our response.

If you have any further questions or need clarification, please let us know. We would be more than happy to provide detailed answers to ensure all concerns are fully addressed.

FQ2 & Q9: Does the ground truth, $c_i$, come from the retrieved synthetic label? There are $K$ synthetic image-text pairs acquired. How is $c_i$ computed from these $K$ pairs? I did not find any description of this in the paper.

FQ4: You should not assume that your readers know how the Maximum Inner Product (MIP) is computed. In Sec. 3.2, you should describe the formulation of it, rather than just replying to me.

FQ7: The pipeline presented in Fig. 2 greatly reduces the difficulty in understanding the paper. Nice figure.

FQ10: Fig. 7 is too abstract. It would be better to use synthetic or real crowd samples to illustrate the process.

Q11: This model requires too many transformer modules. Table 9 presents the efficiency, but it should be compared with other models.

Q12: The pipeline is elaborate, but the performance is not as good as some previous models, such as DANN (in 2019) and BLA (in 2022). Using the same training data (the encoder is also trained with extra data from other real datasets), what is the advantage of the proposed method?

• DANN: "Domain-adaptive Crowd Counting via High-quality Image Translation and Density Reconstruction," Junyu Gao, et al., 2019.
• BLA: "Bi-level Alignment for Cross-Domain Crowd Counting," Shenjian Gong, et al., 2022.

FQ2 & Q9. Ground truth source For simulated data, we have the ground truth data unlike for real data. For each image in the simulated dataset, we know the crowd count, weather conditions, and time of day. As mentioned in lines 191 - 192, 80$\%$ of the simulated data will be used for the reference dataset as well as pre-train the image encoder, and the remaining 20$\%$ is used to train the KAM and count decoder. As aforementioned, we have the ground truths for all the images in the simulated data. Hence, we know the crowd count of each image in the 20$\%$ training set. Therefore, we don't need to estimate $c_i$ from the retrieved pairs for training images as we readily have the count.

• We added lines 289 - 292 to mention the source of $c_i$ required for the loss computation in equation 7.

FQ4. MIPS formulation

• We have added the lines 220 - 225 to formulate the MIPS.

FQ7. Pipeline figure

We appreciate your feedback to improve the presentation of the pipeline and the paper.

FQ10. Two-stage retrieval process

• We have added figure 8 in the appendix with real and simulated crowd samples to illustrate the process. Additionally, we included lines 752 - 755 elaborating figure 8 for the two-stage retrieval process.

Q11. Computational efficiency In table 9 GFLOPS measures the rate at which a computing system can execute floating-point operations. This rate is influenced by the number of retrieved data we feed into the KAM module. Which is why the GLOPS is higher than the baseline. However, for a single retrieved image-text pair, the GLOPS is 4.368. Besides that, we compare the inference speed with other models, such as CSS-CCNN and CrowdCLIP, in table 8 which is critical for practical applications. As a fellow transformer-based model, crowd clip has a higher number of transformer modules compared to our architecture since CrowdCLIP uses two ViT-B/16 for visual encoding in addition to the transformer-based text encoder.

• We have added this description in lines 915 - 928 in section G of the Appendix.

Q12. Comparison with domain adaptation We agree that both methods use the GCC dataset and real data to train. However, both methods are domain-adaptation crowd-counting methods, whereas our method is domain-general crowd-counting. Domain general crowd-counting aims to estimate the crowd count for unseen image distributions. In domain adaptation, a model is trained on one dataset (source) to perform well on a target dataset. Domain adaptation tries to bridge the performance gap caused by the domain shift. In both DANN and BLA, the network is trained for each real dataset and tested for the particular dataset. Hence, in both these methods, the model already has seen the target distribution. However, in our method, we do not use the real dataset that we are testing in any way. Hence, the testing distribution is truly unknown. Therefore, it is unfair to compare the performance of domain adaptive methods against domain generalized methods.

For a fair comparison, these two methods should be evaluated for cross-dataset performance. For example, the model will be trained with GCC and SHA datasets and will be tested on the SHB dataset. Then, we can compare the performance with our proposed method. Neither method has released the training code or models for the datasets used. Regardless, if we compare the performance drop in our proposed method compared to BLA (beats DANN in SHA, SHB, and QNRF), the percentage-wise reduction is $16$%, $50$%, and $7$% for SHA, SHB, and QNRF, respectively. Out of the three datasets, QNRF is the most diverse compared to SHB and SHA. BLA was only able to get a $7$% improvement by having access to the training for QNRF, whereas the proposed method does not require the training set of the test distribution. Besides that, we compare the performance of existing methods under the domain-general setting in table 3 using the GCC dataset.

Thank you for your valuable feedback on our manuscript. We appreciate your suggestions for conducting an efficiency analysis, clarifying the baseline method and its critical design components, and improving the explanation of the count decoder structure.

Q1. Efficiency Analysis
For the retrieval process, we use the naive maximum inner product search. This involves computing the similarity between image embeddings and crop embeddings in the reference database and sorting the similarity scores to find the closest neighbors.

Suppose the reference database is of size $N$, the embedding dimensionality is of size $d$, and we need to find the nearest $k$ neighbors. Then, the computational efficacy of the whole process is $O(N \cdot d \ + \ N \cdot \log k)$. Accordingly, as the retrieval space scales, the time it takes for the retrieval process will increase. However, for larger reference databases, using approximation methods such as the k-d tree, the computational complexity can be reduced to $O(\log N)$ for smaller dimensional sizes, but still, the time consumed will increase with the size of the reference database.

Q2. Baseline Method and Augmentations
The crucial module of the pipeline is the Knowledge Augmentation Module (KAM). In the KAM, we use three different embedding augmentations, as depicted by the first three columns of table 4. Therefore, the baseline method to compare the performance of different augmentation types would be the counting performance without any kind of augmentation.

For the baseline method, the image encoder is pre-trained with both simulated and real data, as described in section 3.1. Then, the count decoder is trained with the simulated data without any image-text retrieval and knowledge enhancement. The performance of the baseline method is given in the fourth row of table 3. We will update table 4 to include the baseline performance.

The proposed architecture's baseline method surpasses the performance of the fully supervised method MCNN. MCNN uses multi-scale convolutional networks with three different receptive fields and maps the image to a downscaled density map. MCNN is a CNN-based network where the receptive field is mostly local, in contrast to the receptive field of a transformer-based network. Furthermore, it should be noted that CrowdCLIP performs closer to MCNN even though CrowdCLIP has an inherent error introduced by the discrete counting levels. This shows the primitive nature of methods like MCNN, where the latent space is directly compared to the density space without any projection.

The performance gain by each augmentation type is provided for only $\mathbf{v}_i^{L}$ in table 4. Therefore, to understand which augmentation types improve the performance, we provide the counting performance for each augmentation type compared against the baseline performance in the following table.

Table A: Performance ablation for KAM.

The most performance gain has come from the components $\mathbf{v}_i^{L}$ and $\mathbf{v}_i^{VV}$ compared to the baseline method. The two augmentations deliver text information and visual information, respectively, but the cross-attention is taken between the image embeddings and retrieved patch embeddings. However, the performance gain from $\mathbf{v}_i^{LV}$ is less compared to the other two augmentations where the cross-attention is taken between the image and retrieved text embeddings.

Q3. Count Decoder Structure
For this two-part question, we will first address the structure of the count decoder. In our proposed work, we use a simple MLP with a single Linear layer following Transcrowd [1]. Secondly, the decoded values of the count decoder are real numbers and not explicitly integers. However, we rounded the decoded values to the closest integer in figure 3, and there is no constraint imposed on the decoder to produce integer values for the estimated count.

References
[1] Dingkang Liang, Xiwu Chen, Wei Xu, Yu Zhou, and Xiang Bai. Transcrowd: Weakly-supervised crowd counting with transformers. Science China Information Sciences, 65(6):1–14, 2022.

We sincerely believe we have addressed the reviewer’s questions, including thorough explanations, additional ablations, and justifications.

As the deadline for the discussion period approaches (November 26th), we wanted to gently follow up to see if you've had the opportunity to review our response.

If you have any further questions or need clarification, please let us know. We would be more than happy to provide detailed answers to ensure all concerns are fully addressed.

Q1. Efficiency Analysis
We have provided the computational efficiency analysis for the retrieval process. However, due to space constraints, we have included the analysis in section F of the Appendix. Furthermore, we have included the computational cost associated with the proposed method compared to the baseline architecture in section G of the Appendix as per the comments of reviewer XkFU.

Q2. Baseline Method and Augmentations We updated table 4 to include the baseline method. Furthermore, we have provided a more detailed version of the ablation on the effect of different augmentation configurations for the KAM in table 7 in section B of the Appendix.

Q3. Count Decoder Structure
We updated lines 303 - 304 to indicate the structure of the count decoder.

We appreciate the constructive feedback and recognize the proposed method's strengths. Your suggestions for improving the title, content, and qualitative analysis will enhance the clarity of our work.

Q1. Title Revision:
We agree with the fact that the approach is similar to cross-dataset crowd-counting, and it uses the paired data from the simulated domain. However, we used "zero-shot" in the paper title as the proposed method is not privy to the training data corresponding to the test dataset. Nevertheless, we agree that changing the title to convey the generalization capabilities of the proposed method will be helpful for the reader. Following the paper titled Domain-General Crowd Counting in Unseen Scenarios [1], where inference is performed without seeing the training data of the test dataset, we believe an appropriate title for the proposed work would be Retrieval-based Generalized Crowd Counting.

Q2. Tabulation Improvements:
We appreciate the reviewer's suggestion to improve the tables' informativeness to the reader. In the revised version, we will modify the tables to indicate the type of data used in each method.

Q3. Qualitative Analysis:
We acknowledge that a qualitative analysis could enhance the understanding of the inner workings of the proposed pipeline. However, instead of the features learned, we think that the behavior of the cross-attention maps would be more suitable for understanding the most influential aspects of the pipeline, especially in the Knowledge Augmentation Module.

For example, if we consider the attention weights when computing vᵢˡᵛ, we take the cross-attention between the image embeddings of the test image and the text embeddings of the retrieved description. By averaging and normalizing the attention weights, we could compute attention scores assigned to each word token.

For instance, we considered an image crop of an indoor scene with low illumination. A retrieved text description for the test crop was:

The image has a clear weather with 52 people in the night.

For this example, the attention scores (rounded to the nearest third decimal) produced for each word token were:

• The: 0.000
• image: 0.002
• has: 0.000
• a: 0.000
• clear: 0.030
• weather: 0.000
• with: 0.000
• 52: 0.905
• people: 0.003
• in: 0.000
• the: 0.000
• night: 0.060

The attention scores are high for the crowd count, time of day, and weather, as these three keywords carry information among different images because the remaining text words are common across all text descriptions. Since both test image and retrieval embeddings are generated from the same encoder, these attention scores highlight which feature maps are more influential due to the way the attention mechanism is developed.

Furthermore, we will include cross-attention maps for the retrieved text description in the revised draft.

References:
[1] Zhipeng Du, Jiankang Deng, and Miaojing Shi. Domain-general crowd counting in unseen scenarios. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 561–570, 2023.

We sincerely believe we have addressed the reviewer’s questions, including thorough explanations, additional ablations, and justifications.

As the discussion period deadline approaches (November 26th), we wanted to follow up and see if you've had the opportunity to review our response.

If you have any further questions or need clarification, please let us know. We would be more than happy to provide detailed answers to ensure all concerns are fully addressed.

Q1. Title Revision:
We have changed the title of the manuscript to Retrieval-based Generalized Crowd Counting.

Q2. Tabulation Improvements:
We have modified Table 1 and Table 2 to accommodate your suggestions.

Q3. Qualitative Analysis:
We included a new paragraph, Qualitative analysis, in lines 447 - 465 to enhance the understanding of the proposed method's functioning with a qualitative approach. Further, we provide figure 6 for the qualitative analysis.