Multisize Dataset Condensation

Thank you for the positive feedback. We will address the listed questions one by one.

1. It's not clear what's the purpose of baseline B. It looks like the results are only compared to baseline A and C.

Before explaining the purpose of Baseline B, here is the recap of the all three Baselines. We will use examples of 10 images per class (IPC=10).

• Baseline A:
  • The ultimate goal of this task, allowing all subset to have the same performance as directly condense to a dataset with specific IPC (Image Per Class).
  • Approach: conduct 10 separate condensations and obtain 10 different datasets.
  • The purpose is the provide an ''upper bound'' of the experimental result.
• Baseline B:
  • Approach: Condense to 10 different datasets with IPC=1, construct large synthetic dataset using small IPC datasets.
  • Purpose: this approach ensures that the accuracy of the smallest IPC is preserved as opposed to preserving the accuracy of the largest IPC (Baseline C). The baseline B illustrates that combining many small datasets do not give comparable results on large IPC.
• Baseline C:
  • Approach: traditional condensation approach.
  • Purpose: show the subset dagradation problem.

The main reason we introduce Baseline-B is that it has the same storage as Baseline-C while addressing the subset degradation problem (accuracy of IPC1 is preserved). However, as illustrated in the experimental results, Baseline-B introduces another problem: using many small synthetic images to construct a big datasets does not give comparable results. That it, Baseline-B has good performance when IPC is small, but fails when IPC is large. Baseline-C is the exact opposite, achieving good performance of large IPC, but fails at small one. Therefore, the proposed method effectively address the problem from both sides, achieving good performance when both IPC is small and large.

2. It's not clear why the freezing is used in MLS selection. If adaptive is good, why not just use adaptive method to choose the subset?

Adaptive selection and Freezing serve for different purposes.

• adaptive selection: it is not practical to incorporate the information of all IPC at the same time due to incurring more computations or slowing learning process. We use adaptive selection to find one IPC that is the most learnable. By adding information of one IPC at a time, the mentioned problem is addressed.
• freezing: freezing aims to prevent the ''already learned" images from being overwritten by information of large IPC. By extracting a smaller subset, ideally speaking, the subset should not contain information of larger IPCs, otherwise the performance will be affected.

This is the Tab. 2 from our paper, it shows the difference of condensation with (row 4) and without (row 3) freezing. Row 3 delivers a poorer performance, especially IPC=1 when compared to Row 4. The issue is that performance of IPC1 is affected by large IPC since it is not frozen (keep receiving pixel updates according to larger IPC).

3. Will the additional loss bring extra computational cost?

Yes, we provide an example in Section. 4.3 More Analysis: Redcued Training Time Needed. The complete training of our method requires 30% more training time (15.1 hrs vs 11.5 hrs) on CIFAR-10 IPC10. However, to reach the average accuracy of the traditional approach, we only requires 0.6 hrs compared to 11.5hrs (traditional approach). Fig. 4 in the paper presents a visualization of the comparison.

Thank you for the positive rating. The main question provided has 3 small question, and we will address each of them accordingly.

1. Compared to baseline A, the accuracy is not higher. Please explain the reason.

Take IPC10 as an example, the Baseline-A is composed of 10 separate condensation processes and 10 different datasets. During the evaluation of the Baseline-A, we evaluate each synthetic dataset separately. Therefore, there is NO INTERFERENCE between datasets. Compare to the propose method, it requires 5x more storage. Our method, on the other hand, does have the inference between different subsets. For instance, IPC1 should not have information of large IPCs (IPC>1) during the evaluation of IPC1. However, during the evaluation of larger IPCs, we actually expect IPC1 to contain information of larger IPCs. Therefore, there is a conflict of interests and trade-offs should be made.

2. Is it possible to reach Baseline A's accuracies?

We believe it is possible to reach a similar average performance (a very close one), but we don't have the proof at this stage. One interesting observation is that: at larger IPCs, the performance of ImageNet outperforms Baseline-A. The below table is a taken from Tab. 1 (c) in our paper.

The performance of IPC15 and IPC20 indeed outperforms the Baseline-A, and it exceeds by a noticeable margin. This gives us a sign that updating the subsets (smaller IPCs) may eventually help larger IPCs.

3. Equation 7 is not that clear. How to calculate the distance between the full dataset and subset?

The feature distance comparison is possible even if number of images are different. Since both real and synthetic images are fed into the network, the output feature only differs in the number of images. For example, if we sample 40 real images from full datasets and forward them, the output dimension is (40, 4, 4, 512), where 40 is the number of images, 4 is the feature size, and 512 is the number of channels. Lets say if we forward 20 synthetic images, the output will have shape like (20, 4, 4, 512), where the difference is on the first dimension. We then average the feature along the first dimension, we will then obtain two averaged features of shape (4, 4, 512). Therefore we can compare. A similar approach is used by Wang etal. [1].

[1] K. Wang et al., “CAFE: Learning to Condense Dataset by Aligning Features,” in CVPR, 2022.

Thank you so much for bringing up the questions and concerns. We try to address them one by one.

1. When the IPC (Inter-Process Communication) is small, there still exists a large accuracy gap between the proposed model and Baseline-A as shown in Figure 2 and Table 1.

The IPC we use stands for Image Per Class, indicating how many images are for each class. The Baseline-A uses much more resources compared to the proposed method. It is the ideal setting if the resources being used is the same as our method. To the best of our knowledge, there is no method could achieve Baseline-A using the given resources (i.e., one condensation process with one dataset).

2. The impact of the calculation interval (∆t) on the performance of the MDC method needs to be further analyzed to determine the optimal interval size.

Optimizing the calculate interval $\Delta t$ will further improve the performance, and we admit finding the optimal one will not be easy, especially this hyper-parameter can vary from dataset to dataset. We will leave this problem for future exploration. Nevertheless, Tab. 7 shows that the choice of $\Delta t$ fall a wide range, and each of them brings a considerable improvement.

3. Can you provide the computational resource consumption and algorithmic complexity compared to Baseline-A, B, C, and other SOTA methods? It can help authors better understand the effects of algorithms in devices with limited computational resources.

Tab. 2 (b) compares the resources for different baselines. (A simplified version is shown below.)

Take CIFAR-10 as an example, condensing using IDC takes roughly 11 hours on average for IPC1-10. Note the time used to condense to different IPCs is different, but will not be extremely different, e.g., IPC1 ~ 10 hrs, IPC10 ~ 12 hrs. Here are the resources needed for each baseline:

• Baseline-A: 110 hours + 550 images (storage)
• Baseline-B: ~100 hours + 100 images
• Baseline-C: 11.5 hours + 100 images
• Ours (+0.0%): 0.6 hours + 100 images
• Ours (+6.4%): 15.1 hours + 100 images

In conclusion, our method incurs 30% more training cost to attain 6.4% increase of average accuracy. To attain the same accuracy as Baseline-C, 0.6 hrs is enough. The visualization is presented in Fig. 4 in the paper.

Comparing to other SOTA. We did not conduct experiments on other SOTA except for IDC-based method, but the expected additional cost is 30% - 50% (capped at 100%). The reason is that the selected IPCs are usually not large, incurring small additional cost (as shown in Fig. 5).

4. Can you provide the values of hyperparameters such as λ and η in Formula 2?

The $\lambda$ and $\eta$ are learning rates using in dataset distillation frameworks, and we did not modify this. We follow the same setting used in IDC [1] (Details can be found in Appendix C.1 in [1]).

[1] J.-H. Kim et al., “Dataset Condensation via Efficient Synthetic-Data Parameterization,” in ICML, 2022.

5. The section on Visualization of MLS is currently difficult to understand. It would be helpful to provide more detailed and accessible explanations to ensure a clear understanding for readers.

Thank you for the feedback, we will try to adjust the figure for a clearer presentation.

Thank you for your comments. We will response point by point.

1. The synthetic samples within the subset seem to be fixed, which may not reflect “Multisize Dataset Condensation” correctly.

The keyword (multisize) in our title means the condensed dataset can be treated as a multi-sized dataset. That is, the dataset of $IPC=N$ can be used as dataset of $IPC={1, 2, ..., N}$ without requiring additional condensation process. This is currently unattainable using traditional approach, since traditional one condenses a synthetic dataset to a specific IPC. For example, dataset with $IPC=N$ will perform poorly on smaller $IPC$ ones, and we take the rights to name the phenomenon "Subset Degradation Problem". Our method addresses the problem, turning the dataset available for multiple size of IPCs.

2. In Fig 2c, for baseline C, how to select subsets to calculate accuracy? Is it random? Let’s assume we have a subset of 2 images. Do we select 2 images from the condensed data randomly?

For baseline C, all subsets are selected using the $N$ images. For example, if there are $10$ images in the class, we take the first image as IPC=1, and first two images as IPC=2.

To address the concern, here is the result of randomly sampled images (see below table). We do not see an obvious advantage with one method over the other.

3. In basic condensation training (Sec 4.1), for each initialization the network is trained for 100 epochs. Is it the inner loop E?

Yes. The 100 epoch is exactly the inner loop E depicted in Fig. 3.

We hope the response addresses your concerns.