SYRAC: Synthesize, Rank, and Count

We thank the reviewer for their valued insights. We would like to address some of their concerns:

1. Counting vs. Detection: We appreciate this insight. We would be happy to expand our discussion to emphasize how crowd counting is a different task than detection and thus can rely on different features, for two reasons: 1) Counting is a more relaxed task than detection in that the former does not require the localization of objects like the latter, 2) counting may leverage features that respond to multiple quantities of objects (not only to one object) in different spatial constellations, e.g. clusters of pedestrians, whereas in object detection, the features delineate a single object. Additionally, typical object detection datasets will have around 1 to 20 objects in an image. Crowd counting datasets tend to have an average of more than 100 pedestrians per image, and density based methods tend to outperform detection based methods in these scenarios.
2. Ranking Noisy Synthetic Counts: The goal of our synthetic ranking pre-training strategy is to mitigate label noise by narrowing down the feature set to those specifically linked to crowd quantity before initiating fine-tuning with the noisy synthetic counting data. While we could pseudo-label the noisy synthetic data using the ranking network, the ranking network may itself be unreliable due to a domain shift between the synthetic ranking dataset and the noisy synthetic counting dataset.
3. Cropping before Synthesis: We did run experiments where we cropped a portion of a source image and generated a synthetic ranking pair that way, similar to your idea. However, this performed poorly in practice. We suspect that was due to the way that pedestrians are distributed in crowd counting images, resulting in many crops that had no crowds in them. As an example, the perspective of an image might cause the top half of the image to contain only sky or buildings or other unrelated features while the bottom half contains the crowd. This setup would result in comparing a real image with no crowds to a synthetic image with no crowds, this potentially impacts feature learning.
4. Reliability of Ranking Labels: We agree that it would be good to provide a more concrete analysis of the reliability of the synthetic ranking data. We manually inspected 50 ranking pairs from each dataset to estimate the accuracy. For ShanghaiTechB, there were no incorrectly ranked examples. For ShanghaiTechA there was 1 incorrectly ranked example. For QNRF there were 2 incorrectly ranked examples. And, for JHU++, there were 2 incorrectly ranked examples. Additionally, we use a pre-trained crowd counting model (DM-COUNT [1]) to estimate the crowd count for 500 real and synthetic ranking pairs to provide an additional estimate of the reliability of the synthetic ranking data. The estimated accuracy is 99.6% for ShanghaiTechB, 99.2% for ShanghaiTechA, 97.2% for QNRF, and 90.2% for JHU++. These additional experiments corroborate the idea that the synthetic ranking data is highly reliable.
5. Fine-tuning Backbone: We agree that it would be useful to have an ablation study examining different strategies for tuning the pre-trained network with the noisy counting data. We did run experiments where we included the noisy synthetic counting data during pre-training with the synthetic ranking data. However, that did not achieve good results.
6. Comparison to CrowdCLIP: Our method outperforms CrowdCLIP on 2 of the 4 datasets. We had considered using early stopping in a similar fashion to CrowdCLIP. However, during our discussion with the authors of CrowdCLIP (which we have documented for reference), we had found that they used the test set for early stopping. We could not justify this as an experiment, as it would seriously contaminate the results.
7. Motivation for Synthetic Ranking: We do utilize existing datasets to generate the ranking pairs, as those images are the source of the synthetic ranking data. Methods like CrowdCLIP generate ranking pairs by performing “intra-image ranking”. In Table 3 we demonstrate that intra-image ranking underperforms compared to our synthetic ranking strategy.

[1]. Wang, Boyu, et al. "Distribution matching for crowd counting." Advances in neural information processing systems 33 (2020): 1595-1607.

1. Are the results chosen from the latest epoch?

2. See the log from CrowdCLIP, and we can find that CrowdCLIP might not utilize early stops since all models are trained on 100 epochs.

Thanks for the response. I would like to know the following information:

(1) Are the results chosen from the latest epoch in the SYRAC?

(2) I have checked the log of CrowdCLIP. However, if we also choose a fixed epoch (e.g., 50) and then utilize the final epoch as the final result, their results are still higher than SYRAC. For example, the 50th epoch's result is 150 MAE of CrowdCLIP on Shanghai PartA, significantly better than the proposed method. This is the reason why I ask the problem (1).

(3) The training epoch is a hyperparameter. I think that the authors utilize 40 epochs for training, which might also based on experiential tuning.

We are grateful for your comprehensive feedback on our paper. We would like to address the points that you have raised.

1. Quality & Settings for Noisy Synth Images: To assess the quality + variance of our synthetic crowd counting image dataset, we've included a table below. The table reports the quality by using a fully-supervised model to estimate, for each synthetic image, the count and compares it to the desired count dictated by the prompt. The table reveals noise across count predictions for each count category. Despite this noise, the average predicted count closely aligns with the given count in the prompt.
   With regards to the count category selections in N, we used the following rationales for setting N = {1, 5, 10, 20, 50, 100, 200}
   • Exponential: We select values from small to large in an approximately exponential fashion. We chose 200 as a maximum because we use patching to split the image into regions with smaller counts (<200) during inference.
   • Computational Cost: We avoid generating images from a more granular count range (i.e. 1, 2, 3, 4, …, 200) due to the increased computational cost this would incur.
   • Simplicity: Our aim wasn't to optimize for specific prompts. While there might exist combinations of N that yield better performance, determining these without introducing validation data, thereby adding supervision, is challenging. We see this as a strength of our method, as the selection of N is based on simple heuristics, and yet it still works.
2. Regression vs. Classification: The count categories chosen for N = {1, 5, 10, 20, 50, 100, 200} represent a subset of counts among positive unbounded integers. Given the nature of our task, predicting counts intuitively aligns with regression rather than classification. We did consider different methods for fitting the pre-trained features to the noisy crowd counting data, including classification. We found that nothing performed better than linear regression. However, it’s possible that we did not explore this sufficiently and there may be a formulation where classification outperforms regression.
3. Zero Count Images: The synthetic images do include images with 0 crowd count. We didn’t explicitly show this in Figure 1. However, we do explain our strategy for generating images with 0 crowd count in the last two sentences of section 3.2.
4. Computational Cost: The question of computational cost is an important one, as generating synthetic data does incur an additional computational cost beyond the burden of training the network. We generated all synthetic data on a single GPU, and we also trained our models on a single GPU.
5. Repititions: Thank you, noted.
6. Comparison to Full-Supervised Features: This is a great point, thank you! We included Figure 6 to corroborate the idea that the pre-training features capture a quantity based distribution. However, it would be useful to further explore that distribution and compare it to features learned by different methods with different supervision budgets. We expect that more supervision would lead to similar but better organization of the latent space with regards to object count.
7. ImageNet Pre-training: In fact, we already leverage ImageNet pre-trained features for intra-image ranking. In Table 3, the performance for intra-image ranking is achieved by starting from ImageNet pre-trained features.
8. Real vs. Synth Spurious Correlations: This is a critical question. Deep learning performance normally suffers due to spurious correlations, and so we expect that our method may be prone to these spurious correlations, as well. However, we rely on the following evidence to suggest that our method produces a network which learns something coherent beyond the separation of real from synthetic:
   • Our method outperforms other unsupervised methods, and our pre-training strategy outperforms other pre-training strategies.
   • Figure 6 demonstrates a quantity based distribution.
   • There is significant domain overlap between the synthetic and real distributions, as the synthetic images are generated using real images as a source. We provide a measure of this domain overlap in the supp. materials.
   • In sec 5, we mention the use of a non-negativity constraint in the final layer of the network (+ ReLU activation), which prevents the weights and output from becoming negative. Thus, we expect that it will be harder for the network to exploit spurious correlations, as it cannot set the output of the network to be negative when synthetic features are present.

That said, we could explore ways to further ensure that we mitigate learning features that distinguish real from synthetic images by producing ranked pairs of images where both images in each pair are synthetic (e.g. remove objects then remove some more; or by using SD to synthesize images then removing objects from those synthetic image), or by using domain adversarial learning.

The table below is produced by using DM-Count [1], which is trained on the NWPU dataset (with full supervision using GT counts), to estimate the count in each noisy synthetic counting image. We report the mean, StD, max, and min count across all synth images. The predicted count can differ significantly from the prompt count for images with higher prompt counts. This suggests that more label noise in larger crowd counts.

[1]. Wang, Boyu, et al. "Distribution matching for crowd counting." NeurIPS 2020: 1595-1607.

We thank the reviewer for their useful feedback and helpful questions. We would like to address the points that the reviewer has raised.

1. Explanation for Figure 6.: Figure 6 illustrates the UMAP representation of the training images based on the pre-trained synthetic ranking features, with points color-coded according to the logarithm of the ground truth count. While interpreting dimensionality reduction results can be complex, we included this figure to demonstrate that there is a coherent quantity-based structure to the latent space learned during the pre-training step. We can interpret this qualitatively by noticing clustering of similarly colored points, which suggests that images with similar counts tend to have similar features.
2. GCC vs. SYRAC: There are several advantages to using synthetic data generated by Stable Diffusion even when GCC already exists.
   • Labor Burden: producing 3D synthetic counting data is not labor free, despite not requiring manual annotations. It requires acquiring new 3D assets for each new object category whereas our method can extend to any novel category that is recognized by Stable Diffusion. And, it requires designing complex 3D scenes as well as environmental conditions such as rain or limited light. Our approach does not carry this burden.
   • Domain Shift: GCC exhibits a significant domain shift when compared to real crowd counting datasets, given that it is produced from 3D assets. Our method maintains a smaller domain shift by using real images as a seed for the synthetic ranking images and also due to the fact that Stable Diffusion is trained using natural images. We have added details to the supplementary material where we provide a measure of domain overlap between GCC and the synthetic ranking datasets, which demonstrates that our synthetic ranking data is closer to the real distribution than the GCC dataset.
   • Different Requirements: While we currently underperform when compared to GCC based methods, we are also working on a fundamentally different problem. GCC requires human labor to produce, whereas our method does not. GCC provides density map annotations, whereas ours does not. Finally, methods that utilize GCC typically focus on domain adaptation due to the large domain gap between GCC and real datasets, whereas our method produces data that is already close to the true distribution.
     Given these reasons, we believe our paper serves as a valuable contribution to the research community
3. Comparison with Liu et al.: We do include intra-image ranking as a pre-training strategy in Table 3, which is the self-supervised component of their work. It is worth clarifying that Liu et al. do not perform self-supervised crowd counting, as their method relies on fully-supervised crowd counting data as the primary training signal. In row 3 of Table 3, we only include the results from the self-supervised component of Liu et al.’s method, which demonstrates that our method consistently outperforms intra-image ranking as a pre-training signal for each dataset.
4. Patch Size: To understand the impact of patch size, we consider two points:
   • Label Noise: There is significant noise in the true underlying count for synthetic crowd counting data produced by Stable Diffusion. This noise increases as the expected count embedded within the prompt increases (see table below). Thus, we expect that prediction noise will be concentrated around images with larger underlying counts. To counter this, we adopt patching during inference, aiming to reduce the object count within each patch. Our findings (Table 2) indicate that increased patching benefits higher count datasets, improving performance by mitigating noise. However, this strategy may compromise images with smaller crowd counts, as larger objects get fragmented across patches. This was apparent in a performance drop among those images.
   • Limited Representation for Dense Crowds: The synthetic ranking generation process often removes many objects in dense crowds, resulting in limited ranking examples for densely crowded images. Consequently, our model might struggle to learn features specific to highly dense crowds. Splitting these images into smaller patches during inference may align them with a distribution where the model performs more effectively.