Fetch and Forge: Efficient Dataset Condensation for Object Detection

Q1 [Algorithm needs to be optimized]

In this work, we analyze why the traditional bidirectional framework is challenging for detection tasks (lines 35-40). We propose the fetch and forge two-phase framework, which avoids the computational cost of second-order derivatives and achieves critical optimization for detection tasks. Due to the complexity of detection tasks, further detailed optimizations will be explored in future research.

Q2 [The description of the second stage is not clear enough]

For unclear areas, please refer to the specific responses provided in the following questions. We will include more details in the revised version.

Q3 [Add several more datasets.]

We understand the reviewer's concerns and will consider adding larger-scale general detection datasets, such as Object-365. However, due to time constraints, completing these experiments in the short term is challenging. If the paper is accepted, we will include additional detection datasets in the revised version. Moreover, Pascal VOC and MS-COCO are the two most influential benchmark datasets in the object detection field. As far as we know, many well-known detection papers have also only used these two datasets. We have demonstrated the effectiveness of our method on these datasets.

Q4 [Discuss about two-stage framework vs bidirectional framework ]

The bidirectional method optimizes outer image updates by comparing performance differences between real and synthetic sets, using gradients or model parameters to compute matching loss. The model also requires inner loop optimization involving gradient calculations, resulting in second-order derivatives and high computational costs. As image resolution and model complexity increase, the efficiency of second-order derivative calculations decreases, making outer loop optimization for synthetic images challenging. As stated in lines 32-35, these methods are only effective on small datasets and networks. Our two-stage method decouples the original second-order calculations, allowing separate training of a complex detector and optimization of high-resolution images (lines 43-45). This separation improves efficiency.

Q5 [Figure 3 lack target.]

Thank you for pointing that out. The term "target" refers to the objects randomly sampled along with the images during initialization. We will supplement the targets in Figure 3 in the revised version.

Q6 [Add formula 5's introduction.]

We are pleased to provide a more detailed explanation: In synthesizing the image $\widetilde{X}$, we separate the foreground and background regions using a binary mask. For the background ($x_{\text{back}}$), we apply a suppression strategy with a hyperparameter $\alpha = 0.7$, limiting updates to preserve contextual meaning. For the foreground ($x_{\text{fore}}$), we employ a random erasure method, ensuring that the model focuses more on refining the foreground pixels during the inversion phase. By strategically applying suppression and enhancement, we prioritize the processing of important visual information, optimizing the model's ability to generate high-fidelity synthesized images.

Q7 [Patch optimization and goal complexity in IPE module.]

We employ an incremental approach to enhance the model's adaptability to an increasing number of ground truth labels and bounding boxes. Initially, we introduce a modest number of gt labels and bboxes, allowing the model to assimilate the initial dataset without overwhelming its learning capacity. Our methodology progresses through three stages, each comprising 1000 iterations. Within each stage, we incrementally increase the instance count, denoted by k, starting from 11 and progressing to 22, and finally to 3*3. This curated incrementation ensures a robust and sustainable learning trajectory.

Q8 [Missing annotations in Table1&2 and "±" explanation. ]

Thank you for pointing out this issue. The performance on the full dataset is based on the open-source implementation from official ultralytics/yolov3. The "±" symbol represents the standard deviation calculated from 25 results, obtained by generating five synthetic sets and testing each set five times.

Q9 [Lack of comparison with similar methods]

To our knowledge, our work is the first to study dataset condensation for object detection. Traditional methods for dataset condensation in classification are difficult to apply to detection tasks, making direct comparisons challenging. However, we understand the reviewer's concerns and are considering adding a discussion section to analyze the difficulties and challenges of adapting classification dataset condensation pipelines to detection tasks. This discussion could inspire future research in this area. In fact, early papers on classification dataset condensation (such as [Dataset Distillation] and [Dataset Meta-Learning from Kernel Ridge-Regression]) only compared against random and core-set methods. Therefore, we are essentially following their experimental framework.

Q10 [How is the synthesized image updated under the guidance of loss? add multiple instances in the visualization.]

Please refer to Figure 3 in the supplementary PDF, where we show the variation in guidance loss over 3000 iterations and the test performance of synthetic images(tested every 100 iterations). We also add more visualization for multi-instance in Figure 4 of the supplementary PDF.

Q11 [Lack of reference to Figure 5.]

Apologies for the oversight. We will add a reference to Figure 5 in the revised paper. Thank you for pointing it out.

Q12 [ Limitation ]

The two limitations are actually discussed in the "Future Work" section of our paper. 1.We will investigate more complex detector architectures following the inversion approach. 2.We plan to build a model pool of different detectors during the fetch phase and perform inversion using multiple detector weights on the synthestic image in the forge phase to incorporate diverse information.

Dear Reviewer SZRX,

We hope our response addressed your questions. As the discussion period comes to a close, we eagerly await your valuable feedback. We would greatly appreciate it if you could consider improving your rating.

Thank you for your valuable time!

Sincerely,

Authors of Paper #8952

Q1 [Notation issue: see IPE in Eqn. 6 and Step 5 of Alg.1.]

We note the inconsistency in the notation for I_PE between Equation 6 and Step 5 of Algorithm 1. We will correct this inconsistency in the revised version of the paper. Thank you for bringing this to our attention.

Q2 [Why is row 3 of Table 3 blank?]

The blank row in Table 3 was due to a LaTeX formatting error. It should have displayed the results for the "baseline+IPE" method. We will correct this issue in the revised version of the paper. Thank you for pointing it out.

Q3 [Implementation Details are not comprehensive. Is it the training epoch for IPE set to M/batchsize? And I would suggest not calling it a training epoch.]

Yes, we set M/batchsize for IPE. We will correct the misleading terminology by using "iterations" instead of "epochs." Thank you for the suggestion, and we will include more detailed implementation details in the revised version.

Q4 [What would happen if we increase the condensation ratio, e.g., 5%, 10%, 20%, 50%?]

Based on your suggestion, we add experiments on VOC with compression ratios of 5%, 10%, 20%, and 50%. See tabel:

We find that as the ratio increases, the advantage of the condensation method compared to the baseline gradually decreases, which is consistent with some studies in classification([DC-BENCH: Dataset Condensation Benchmark]). In future work, exploring how to train using only synthetic images to surpass the performance upper bound of real images is a very interesting topic.

Q5 [it should be ''ratio'' in Table 3.]

Thank you for pointing out the spelling error. "Ratio" refers to the compression ratio of the entire dataset. We will correct this in the revised version of the paper.

Stage-II: Forge Phase

We are pleased to provide a more detailed explanation:

Initialization: Randomly sample original images and their corresponding targets from the real dataset to form the initial synthetic set $\widetilde{\mathcal{S}}$, using the detector \psi_{\theta_\\mathcal{T}} trained in Stage-I (with weights frozen).

Training (Inversion): The optimization goal is to update the pixels of the synthetic images.

1. Divide $\widetilde{\mathcal{S}}$ into $k \times k$ patches using the IPE module. The FBD module constructs a binary mask to differentiate between foreground and background, applies random erasure to the foreground, and sets the update weight for background pixels to $\alpha=0.7$.
2. Input $F_{BD}$($I_{PE}$($\widetilde{\mathcal{S}}$)) into the detector, and calculate the loss according to Equation 7 in the paper: $L = L_{\text{det}}(\psi_{\theta_{\mathcal{T}}}(\widetilde{X}), \widetilde{A}) + R_{\text{reg}}$.
3. Update the pixels: $\widetilde{\mathcal{S}} \leftarrow \widetilde{\mathcal{S}} - \eta_s \nabla_s L$.

Under the guidance of the task loss, the synthetic images gradually stabilize, as shown in Figure 3 and 4 of the supplementary PDF.

We appreciate your attention to detail.

Yes, the number of training iterations is 3,000.

In Figure 3, we recorded the synthetic images within a batch, as indicated by "mAP for current Batch" on the right column of the images. This mAP was calculated by directly predicting on the current batch of synthetic images using a pre-trained YOLOv3-SPP model. However, in the experiments described in the paper, we trained a detector from scratch on the full synthetic set and then tested it on the original test set. Thank you for your reminder; we will include all the specific details of Figure 3 in the revised version.

we apply learning rate decay using the CosineAnnealingLR scheduler, with eta_min set to 0.0005.

Thanks for the quick reply. So, is the training iteration 3K? As shown in Fig. 3 of the supplementary PDF, mAP goes for 94+ at 3000 iterations. While in the main paper, there are no such experimental settings that can exceed 94 mAP. So I wonder what is the experimental setup corresponding to this figure. Also, is there any learning rate decay?

Q1 [Background suppression and its impact on object detection.]

We acknowledge the importance of background in detection tasks but prioritize updating the foreground in image synthesis while retaining the background for better results. "Suppress background" means limiting updates to background pixel values to preserve contextual integrity. For example, a background originally containing the sky could be mixed with information like the ocean or grass, disrupting the original context. We will clarify that background is not "unimportant" and add a discussion on this topic. Our ablation study on 1% VOC shows that increasing background update weight decreases performance, confirming our suppression strategy.

Q2 [Discussion about the input initialization.]

In this paper, we randomly sampled images and labels from the real dataset for initialization, leading to variations in the synthetic set's foreground categories and object size distribution. This caused result fluctuations. Defining a good initialization for detection dataset condensation is complex, considering factors like category frequency and bounding box characteristics. We supplemented experiments on VOC 0.5% with different methods, including noise, random, k-center, herding, and a 'size-balance' method suggested by Reviewer #z3Af.

Noise performed the worst, consistent with experiences in classification dataset condensation. Random and Size-balance achieved good results. Future studies on initialization in the detection domain could be key to improving condensation performance. We will continue exploring how to select representative samples to enhance efficiency and effectiveness.

Q3 [Section 3.1 should be more on detection data]

Thank you for your suggestion. We will revise Section 3.1 to include a detailed description of the setup for the detection dataset condensation task

Q4 [what happens with different initializations during the Fetch part.]

The Fetch phase involves standard training of detectors, with initialization referring to the detector's initial weights. The weights obtained after Fetch training are also used as initial model weights for the Forge phase, influencing the quality of synthesized images. We tested models trained for different numbers of epochs for inversion, leading to varying performances, as shown in the table below for 0.5% VOC:

Prolonged Fetch phase training can negatively affect synthesized image quality, possibly due to excessive data compression, complicating reconstruction. This issue needs further investigation in future work.

Q5 [Importance of location vs. classification in condensation]

We conducted a new experiment to address this concern by adjusting the weights of the classification and location losses during the inversion process. For YOLOv3-SPP, we used GIoU loss for location and CrossEntropyLoss for classification. The results are shown in the table below with a 0.5% compression ratio on VOC.

We found that a 1:1 weight ratio is very suitable. When the weight for the localization loss is further reduced, the performance metrics show a noticeable decline.

Q6 [Report percentage of condensed data meetting real data.]

Due to time constraints, we increase the compression ratio on the voc as follows:

We find that as the ratio increases, the advantage of the condensation method compared to the baseline gradually decreases, which is consistent with some studies in classification([DC-BENCH: Dataset Condensation Benchmark]). In future work, we will further explore how to achieve lossless methods.

Q7 [why IPE is needed if the network is already trained]

Incremental Patch Expand is used in the Forge phase to increase the diversity of the synthetic images. Taking into account the complexity of multi-instance detection task, we introduced the incremental approach to gradually increase the number of instances being optimized.

Q8 [pixel vs feature regularization]

We have add an ablation study on pixel versus feature regularization on 1%ratio VOC, as shown in the table below:

For inversion techniques, Rfeature is crucial for reconstructing the information content in the synthesized images. Rpixel helps maintain performance to some extent by balancing the image pixel distribution and reducing sharp regions in the pixels.

Thanks for the authors' response it helped clarify my concerns and adding these to the paper will definitely make it stronger. I appreciate the authors' efforts in the rebuttal. I will improve my rating to weak accept.

Q1 [Analyze the class distribution of small/medium/large objects in the K-Center and Herding.]

Based on our analysis, we compare the distribution of small(area<32x32), medium, and large(area>96x96) objects sampled by the k-center and herding methods with the original dataset, presenting the results as stacked bar charts. Please refer to Figure 1&2 in the supplementary PDF. In Figure 1, for the VOC dataset, we observe that k-center tends to select larger images, while herding includes both medium and large images, with both methods selecting fewer small-sized objects. In Figure 2, for the COCO dataset, K-center's selection is more aligned with COCO's inherent proportions, though some categories show significant differences. This may be due to COCO's more balanced object size distribution. These sampling discrepancies might contribute to the poor performance of the methods. Subsequently, we design a new baseline approach, called 'Size-balance':

1. Calculate the frequency of occurrence for each category based on the sampled subset's size.
2. For each category, sample the corresponding number of small, medium, and large samples according to the original dataset's proportion. Finally, we compare this baseline sampling approach with subsets from VOC 1%, and COCO 1%. The results outperform the random baseline; however, our method still surpasses this new baseline. A more comprehensive comparison table will be provided in the revised submission. | Sample ratio | Methods | mAP | AP50 | |-------------| --------------|-------------- | ------------ | | 1% voc | K-center | 6.1±0.2 | 21.9±0.9 | | 1% voc | Herding | 5.5±0.2 | 19.3±0.5 | | 1% voc | Size-balance | 9.1±0.3 | 26.7±0.5 | | 1% coco | K-center | 4.0±0.1 | 12.9±0.1 | | 1% coco | Herding | 4.1±0.2 | 12.5±0.2 | | 1% coco | Size-balance | 8.7±0.2 | 20.3±0.4 |

Q2 [How foreground be derived?]

The foreground coordinates are sampled from the real dataset annotations. During the initialization phase, we random sample the images and their corresponding targets from the dataset, and the bounding box coordinates provided in the dataset annotations are used to define the foreground regions.

Q3 [course learning->curriculum learning?]

Yes, the term "course learning" was intended to refer to "curriculum learning." We appreciate your feedback and will make the necessary correction in the revised manuscript. Thank you for pointing this out.

Q4 [Expand: Are image patches upsampled be the same size as the original image?]

"Expand" in Equation 6 refers to the process during the train/test pipeline where synthestic image are upsampled to a specified size, typically 512x512. We will clarify this in the revised manuscript.

Q5 [The rate of k increase in IPE.]

The growth setting for k is straightforward; we linearly increase k based on iterations. Specifically, k increases from 1 -> 2 -> 3, with an increment every 1000 iterations.

Q6 [How does $\widetilde{A}$ be smapled?]

Yes, $\widetilde{A}$ corresponds to the images sampled during initialization. Since we use random sampling, the performance of the synthesized dataset may experience slight variations due to different initializations. We reported this standard deviation in Experiments. Additionally, we have supplemented the experiments as follows:the performance of the synthesized dataset on 0.5% VOC when using other sampling methods as initialization, including the new baseline 'size balance' established in W1. The study of initialization strategies could be a promising direction for future research.

Q7 [ What is 'the Uniform'?]

The 'Uniform' refers to uniform class sampling, meaning that each class appears approximately the same number of times. We will add appropriate annotations in the tables to clarify this term.

Q8 [Are these the standard deviation over multiple runs of dataset condensation?]

Yes, the error bars represent the standard deviation over multiple runs of dataset condensation. Following the standard dataset condensation training and testing pipeline, we performed dataset condensation five times. For each generated synthetic set, we conducted five testing runs, resulting in a total of 25 results. The standard deviation was then calculated based on these 25 outcomes.

Q9 [The description in line 99-102 isn't quite correct.]

Thank you for your observation. We will revise the description of object detectors in related works.

Q10 [what open-source implementation is used?]

We follow this open-source implementation:official ultralytics/yolov3

Q11 ['The Coreset method',terminology should stay consistent]

In lines 213-214, we mentioned that core-set selection methods include random, K-Center, and herding. We will emphasize this point again in line 230 and ensure consistent terminology throughout the paper. Thank you for your suggestion.

Q12 [Train a dataset condensed on other detector.]

We agree with the suggestion to deploy more detectors for condensing; however, due to time constraints and the significant differences in environments and configurations of different detectors, extensive code modifications are required. We plan to include these detectors in future versions. Additionally, we will consider applying this framework to other tasks, such as segmentation, in future work.

Q13 [It would be interesting to see Figure 5 on multi-class images.]

We have added visualizations of multi-class images. Please refer to Figure 4 in the supplementary PDF for details.