FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation

We sincerely thank you for your constructive comments. We are encouraged that you found our method novel, and appreciate your recognition of its performance and efficiency improvements. We would like to address the comments and questions below.

W1: Figure 2 is not clear.

We sincerely thank the reviewer for the thoughtful question. To clarify, each Data Residual Block does indeed include a Recover step. We also appreciate the reviewer for pointing out the labeling issue in Figure 2, upon review, we confirm that the downsample and upsample labels were inadvertently swapped, which may have caused confusion. We will correct the labels and revise the figure to improve clarity in the final version.

W2: Comparison to non-optimization based methods.

We thank the reviewer for the question. RDED is a non-optimization-based framework that selects representative patches using a pretrained model, without involving any further image-level optimization. In contrast, our method is optimization-based and involves iterative refinement of synthetic images, which inevitably incurs higher computational cost. Nonetheless, such optimization typically leads to superior performance over non-optimization approaches.

To ensure a fair comparison of efficiency, we report results against optimization-based baselines that involve similar iterative procedures and are scalable to large-scale datasets such as ImageNet-1K. As a result, RDED is not included in Table 2.

W3: Further explanation of Information Vanishing.

We thank the reviewer for the question. First, we would like to clarify that our goal is not to eliminate information vanishing entirely, but to mitigate it. As optimization progresses, fine-grained local details are gradually suppressed in favor of dominant global structures. Once global information becomes saturated, continued suppression of local details inevitably leads to information vanishing.

The sharp surge observed in Figure 3 occurs as ARC injects rich fine-grained details back into the data. This sudden reintroduction of diverse semantic content naturally leads to a spike in feature-level entropy. Importantly, this surge reflects the restoration of previously suppressed local details, rather than instability or noise.

To verify that ARC preserves semantic consistency rather than introducing disruptive information, we conduct two complementary analyses. First, we measure the cosine similarity of features before and after ARC injection across three ARC stages ($k=3$). Second, we evaluate the accuracy of the distilled images before and after ARC using a pretrained model as a verifier.

As shown in the table below, feature-level cosine similarity remains high (≥ 0.88) across all stages, indicating that the semantic structure is well preserved. While verifier accuracy drops slightly (from 100% to 97%) at the first injection stage, it quickly recovers in later stages, suggesting that ARC introduces semantically aligned residual content rather than harmful noise. These results demonstrate that the observed entropy increase stems from meaningful details, not instability in the distillation process.

W4: More analysis on assumption 8.

We thank the reviewer for the insightful suggestion. We first elaborate on the intuition behind Assumption 8 and then empirically verify whether it generally holds. Starting from Assumption 8 and rearranging the right-hand side, we obtain:

$$\mathfrak{R}\_n(\mathcal{H} \circ \tilde{\mathcal{C}}^{\text{res}}) - \mathfrak{R}\_n(\mathcal{H} \circ \mathcal{O}) > \left( \frac{L\_h L\_\ell}{2B} + \frac{L\_h}{2} \right) \cdot \Delta,$$

where $\mathcal{O}$ denotes the original data patch, and $\tilde{\mathcal{C}}^{\text{res}}$ refers to the class induced by the optimized version of the original patch.

Since both $L\_\ell$ and $L\_h$ are small due to the use of high-temperature KL loss and strong regularization (e.g., CutMix), the coefficient $\left( \frac{L\_h L\_\ell}{2B} + \frac{L\_h}{2} \right)$ is sufficiently small. As a result, the right-hand side becomes negligible in our setting and can be conservatively omitted to simplify the analysis, which is:

$$\mathfrak{R}\_n(\mathcal{H} \circ \tilde{\mathcal{C}}^{\text{res}}) > \mathfrak{R}\_n(\mathcal{H} \circ \mathcal{O}),$$

The intuitions is that if the Rademacher complexity of the optimized patches exceed that of the original ones, theorem 2 typically holds.

To validate if this inequality holds empirically, we estimate the empirical Rademacher complexity, defined as:

$$\widehat{\mathfrak{R}}\_n(\mathcal{H} \circ \mathcal{S}) = \mathbb{E}\_{\sigma} \left[ \sup\_{h \in \mathcal{H}} \frac{1}{n} \sum\_{i=1}^n \sigma\_i h(x\_i) \right],$$

where $\sigma\_i$ are i.i.d. Rademacher variables from $\{-1, +1\}$ and $\mathcal{S}$ is the dataset. Although computing the population complexity is intractable, it is well known that:

$$\lim\_{n \to \infty} \widehat{\mathfrak{R}}\_n(\mathcal{H}) = \mathfrak{R}\_n(\mathcal{H}),$$

so the empirical proxy becomes more accurate as $n$ increases.

In practice, we approximate this quantity via random label memorization on a IPC=50 dataset (n=50,000). A standard empirical estimate only involves binary classification, which we found to be too simple and less discriminative in our setting. To address this, we increase task difficulty by assigning random labels uniformly from a large label space (e.g., 500 and 1000 classes), making the memorization task substantially harder.

Concretely, for a given dataset $\mathcal{D} = \{x\_i\}\_{i=1}^n$, we construct a randomized label version $\mathcal{D}\_{\text{rand}} = \{(x\_i, y\_i^\text{rand})\}\_{i=1}^n$ by sampling:

$$y\_i^\text{rand} \sim \mathcal{U}\{1, 2, \dots, K\}, \quad \text{where } K \in \{500, 1000\}.$$

We apply this procedure to both the original patch set and its optimized counterpart, and train three distinct models on each using cross-entropy loss. The minimum training loss across runs is reported as a proxy for Rademacher complexity: lower loss indicates higher capacity to fit random labels, and thus higher empirical complexity.

Across all settings, whether under moderate (500-way) or challenging (1000-way) classification tasks, we consistently observe that the optimized data achieves lower training loss than the original. This suggests that the optimized data is easier to fit, reflecting a higher capacity of the hypothesis class to memorize random labels, thereby indicating a higher empirical Rademacher complexity.

These findings offer strong empirical support for Assumption 8, suggesting that optimized patches indeed yield higher empirical Rademacher complexity. Therefore, the theorem 2 becomes applicable to real-world scenarios.

We sincerely thank the reviewer again for the follow-up. We respectfully disagree with several points and would like to clarify them further to avoid potential misunderstandings.

(1) RDED comparison and fairness

Our main comparison focuses on optimization-based methods, as FADRM involves iterative refinement while RDED does not. For completeness, we include RDED below:

FADRM is slower due to its optimization steps but achieves a +5.7% accuracy gain, which we believe is a meaningful trade-off. This result will be included in the revision.

(2) Effectiveness of ARC.

We respectfully disagree with the claims that "continued optimization could lead to convergence with the W/O ARC curve" and that the "final trends are the same," as these are not supported by the trends in Figure 3.

Applying ARC at iteration 3500 yields higher information density after 500 iterations than applying it at iteration 3000 with the same budget (see w/ ARC at 3400 vs. 4000 iterations). This indicates that longer optimization with ARC preserves and stabilizes information at a higher plateau. In contrast, the W/O ARC curve exhibits a steady decline throughout optimization, reflecting continuous information loss, making it fundamentally impossible for the two curves to converge.

To further refute the reviewer’s suggestion that ARC’s improvement is merely temporary, we present the table below (also included in our response to reviewer ccas Weakness 4), which shows consistent accuracy improvements with ARC across both shallow and deep optimization stages. This directly contradicts the notion that ARC’s effect diminishes over time.

(3) Technical novelty.

We respectfully note that the key components of our method are not drawn from existing distillation techniques. We introduce data-level residual connections, which reinterpret skip connections for synthetic data refinement, bridging the gap between model optimization and data synthesis.

We also shift the focus to optimization-level efficiency, moving beyond model size or initialization to improve efficiency by identifying and reducing redundant computation within the optimization process. To our knowledge, this perspective has not been explored in prior uni-level framework.

We sincerely thank the reviewer for the response, and we are pleased that our earlier discussion has addressed your concerns regarding the effectiveness of ARC.

We would like to clarify that Reviewer Ne6j has consistently recognized the novelty of our main contribution ARC (data-level residual connection). After we explained that MPT (mixed precision training) and MRO (multi resolution optimization) are designed to shift the focus toward optimization-level refinement, a direction that has been consistently overlooked in prior uni-level frameworks, Ne6j agreed this also addressed their concern regarding the novelty.

We respectfully note that, to the best of our knowledge, ours is the first work to explicitly identify the issue of information vanishing and to apply data-level residual connections as a solution. We also note that the novelty of our work has been clearly recognized by all other reviewers.

We sincerely thank you for your encouraging feedback. We appreciate your recognition of the clear theoretical justification and the novelty of our data-level residual design. We address the remaining concerns in the detailed responses below.

W1: Contribution of individual componenets on efficiency gains.

Thank you for the suggestion. To clarify the individual contributions of Mixed Precision Training (MPT) and Multi-Resolution Optimization (MRO) to computational efficiency, we conducted an ablation study as shown in the table below, the experiment is conducted on ImageNet-1k, where efficiency is measured based on the time needed for generating a single distilled image.

As shown, MRO is primarily intended to reduce the overall wall-clock time required to optimize a single image by iteratively performing training on downsampled versions. Since the process still involves full-resolution updates, the peak GPU usage remains unchanged. In contrast, MPT is designed to reduce both computation time and peak memory consumption by leveraging lower-precision operations throughout.

While MPT accounts for the majority of the efficiency improvement, MRO provides complementary gains. For instance, when comparing the configurations with and without MRO under MPT, we observe a reduction of 0.05 seconds per image. Although this difference may appear marginal at the individual level, it becomes meaningful when scaled. For example, generating a 50 IPC ImageNet-1K dataset (i.e., 50,000 images) results in:

$$50{,}000 \times 0.05\ \text{seconds} = 2{,}500\ \text{seconds} \approx 42\ \text{minutes}$$

In summary, MPT is the primary contributor to efficiency, and MRO further improves performance by iteratively optimze the images on a smaller resolution.

W2: Efficiency enhancements over SRe$^2$L++.

Thank you for the question. We clarify that SRe$^2$L++ does not adopt Mixed Precision Training (MPT) or Multi-Resolution Optimization (MRO), and requires 4000 iterations to optimize each image. In contrast, our proposed method FADRM applies both MPT and MRO, and completes optimization in only 2000 iterations. This results in substantial efficiency gains, as shown in Table 2 (left), including a reduction of approximately 2.05 seconds in wall-clock time per image and a significant decrease in peak GPU memory usage.

To illustrate the impact at scale, generating a 50 IPC ImageNet-1K distilled dataset (50,000 images) would lead to:

$$50{,}000 \times 2.05\ \text{seconds} = 102{,}500\ \text{seconds} \approx 28.5\ \text{hours}$$

This demonstrates the practical advantage of FADRM over SRe$^2$L++ in large-scale settings. Additionally, we clarify that Table 3 serves a different purpose: it evaluates the impact of applying MPT across various methods and confirms that MPT does not degrade the quality of the distilled dataset. The analysis is not limited to FADRM and is intended to show the broader applicability of MPT in the field of dataset distillation.

W3: Clarification for performance gains.

We thank the reviewer for raising this important point.

Firstly, we would like to clarify that the performance gains in the ImageNet-1K ResNet-18 setting are substantial. For single-model distillation, our method FADRM achieves clear improvements over the strongest baseline RDED. For example, at IPC = 10, FADRM outperforms RDED by +5.7%, which we believe is both meaningful and robust. For multi-model distillation, FADRM+ also consistently surpasses other approaches across all IPC levels, as reported in Table 1. The only case of relatively moderate improvement is at IPC = 50, where FADRM slightly outperforms CV-DD by +0.6%.

To confirm the consistency and statistical reliability of our results, we have conducted additional runs and report the mean and standard deviation across 3 independent trials on ImageNet-1K with ResNet-18, as shown in the table below (CV-DD didn't report statistical variations). The results further demonstrate the stability and robustness of FADRM across runs.

Q1: Clarification on evaluation criteria of efficiency.

Thank you for the question. There are two distinct evaluation criteria used to assess time efficiency in the paper: (1) seconds per image generation, as shown in Table 2 (left), and (2) milliseconds per image update, as shown in Table 3. The choice of metric depends on the evaluation objective.

In Table 2 (left), we report seconds per image generation, which captures the total wall-clock time required to generate a single distilled image. This metric accounts for various factors, including the total number of optimization iterations, the varying update time across different image resolutions introduced by MRO, and additional overhead such as loading precomputed statistics in EDC. We choose this metric because, under the MRO setting, image resolution changes throughout the optimization process, making it difficult to fairly measure efficiency using a single per-update timing. In contrast, seconds per image generation provides a more comprehensive and resolution-agnostic view of overall efficiency.

In contrast, Table 3 focuses on optimization-level efficiency under fixed-resolution settings, where only MPT is applied and MRO is excluded. In this case, we use milliseconds per image update to isolate and assess the efficiency gains specifically attributable to MPT.

In summary, we adopt different metrics to match the evaluation focus: Table 2 reflects overall generation efficiency, accounting for all contributing factors, while Table 3 isolates the per-iteration benefits of MPT without the confounding effects introduced by varying image resolutions.

Q2: Clarification on Peak Memory Usage.

Thank you for the question. We would like to clarify that in line 289, the phrase "optimization-level refinement and MRO" refers collectively to two efficiency enhancing components used in our method: optimization-level refinement corresponds to Mixed Precision Training (MPT), while MRO refers to Multi-Resolution Optimization. The sentence aims to highlight that the integration of both techniques leads to improved efficiency.

To answer the question directly: the observed reduction in peak GPU memory usage is primarily attributed to MPT, which reduces memory consumption by employing lower-precision arithmetic during optimization. In contrast, MRO is designed to improve overall training efficiency by reducing computation time, as discussed in our response to Weakness 1.

We hope this clarifies that the efficiency gains mentioned in line 289 result from the combined use of MPT and MRO, with MPT responsible for the peak memory reduction.

We sincerely thank you for your constructive comments. We are encouraged that you found our method both novel and practically valuable, and appreciate your recognition of its performance and efficiency improvements. We would like to address the comments and questions below.

W1: Labelling Error.

We thank the reviewer for pointing this out. We will eliminate this error in the revised version to accurately reflect the intended operations within the resampler module.

W2: Evaluation task clarification.

We thank the reviewer for pointing this out. We confirm that the evaluated task throughout the paper is indeed image classification, and we will explicitly clarify this in the revised version to avoid any ambiguity.

W3: Ambiguity in model usage and generalization claims.

We thank the reviewer for raising this important concern. To clarify:

1. Training vs. Evaluation Models:
   If we understand the question correctly, training refers to the models used to optimize the distilled dataset, while evaluation refers to the models trained on the distilled data to assess generalization. We have already provided this information in the original submission:
   • For single-model distillation (FADRM), the training model is stated in the caption of Table 1.
   • For multi-model distillation (FADRM+), the training models are detailed in Table 7 of the Appendix.
   • The evaluation models used are listed in the top row of Table 1.

We fully agree that these details are crucial to the clarity and reproducibility of our method. To avoid any ambiguity, we will consolidate this information and explicitly state it in the main paper.

2. Generalization Beyond ResNet:
   Our distilled datasets are designed to generalize beyond the ResNet family. As shown in the right block of Table 2 in the main paper, we have already conducted cross-architecture evaluations to demonstrate that our method generalizes beyond the ResNet family. To further strengthen this point, we also include additional results in the table (ImageNet-1k IPC=50) below to more comprehensively support the cross-architecture generalizability of our approach.

We appreciate the reviewer’s thoughtful feedback and will revise the manuscript accordingly to enhance clarity and completeness.

W4: Clarification for Figure 2.

We thank the reviewer for raising this point. We would like to clarify that in our framework, the original data patches shown in Figure 2 are not meant to represent the full original dataset, but rather a subset selected as initialization for the distilled dataset. Specifically, for a target of N images per class (e.g., 50 IPC), we first select N patches per class from the original dataset, which are then used as the initialization for optimization. Therefore, the number of original and distilled patches is intentionally the same at this stage. To avoid confusion, we will revise the caption of Figure 2 to better explain this initialization process.

W5: Inconsistent memory results.

We sincerely thank the reviewer for pointing this out. Upon careful re-examination, we confirm that this was indeed a reporting error, the correct peak GPU memory usage should be 11.0 GB. We apologize for the oversight. Importantly, this correction does not affect any of the experimental conclusions or comparisons in the paper. We will update the result accordingly in the revision.

Q1: Performance evaluation task.

We thank the reviewer for the question. Yes, the downstream task used for performance evaluation throughout the paper is image classification.

This task was chosen because, given only class labels and distilled images, image classification provides a standardized way to assess whether the synthetic data captures sufficient discriminative information for training a performant classifier. It is also the standard evaluation protocol in the dataset distillation literature, particularly for the image modality. Therefore, it provides a fair and widely accepted basis for comparison across methods.

Q2: Trained Models and hyper-parameter configurations.

We appreciate the reviewer’s question.

In our main experiments (Table 1), we trained ResNet-18, ResNet-50, and ResNet-101 on the distilled datasets. To further evaluate the generalizability of our method across architectures, we conducted cross-architecture experiments (Table 2 right) involving:

• Transformer-based models: Swin-Tiny, ConvNext-Tiny, DeiT-Tiny
• Lightweight CNNs: MobileNetV2, ShuffleNetV2
• High-capacity models: EfficientNet, DenseNet201, Wide ResNet50-2

All models were trained following the configuration used in EDC: 1000 epochs for CIFAR-100 and Tiny-ImageNet IPC=1, and 300 epochs for all other datasets, to ensure consistency and fair comparison across methods. Other hyperparameters (e.g., learning rate, optimizer) were selected based on model capacity and the complexity of the distilled dataset. Detailed training configurations are provided in Appendix Section F.

We sincerely thank you for your constructive comments. We are encouraged that you find our work novel with strong empirical performance on both efficiency and effectiveness. We would like to address the comments and questions below.

W1: Concern if weak or mismatched, distillation may degrade.

We thank the reviewer for the insightful comment.

As shown in Theorem 1, FADRM, like other uni-level distillation methods, is most effective when the pretrained model not too confident (relative weak), i.e., when its output has high entropy. In this regime, the mutual information upper bound is looser, allowing distilled data to better capture information from the original dataset. This is supported by both theory (Theorem 1) and experiments (see table below).

Regarding architectural mismatch, our original submission includes cross-architecture results (Table 2, right), where FADRM consistently outperforms baselines. We further reinforce this with additional experiments in the table below on ImageNet-1k IPC=50.

W2: Ground the merge ratio in theoretical anlysis.

We thank the reviewer for the question. To ground the choice of merge ratio $\alpha$ in theory, we define the generalization bound of FADRM as:

$$D(\alpha) = \widehat{\mathcal{L}}\_{\text{FADRM}} + 2B \cdot \mathfrak{R}\_n(\mathcal{H} \circ \tilde{\mathcal{C}}\_{\text{FADRM}})$$

Assuming the loss is $\lambda$-strongly convex and letting $\Delta := \frac{1}{n} \sum\_i \|\tilde{x}\_i^{\text{res}} - x\_i\|$, we have:

$$\widehat{\mathcal{L}}\_{\text{FADRM}} \le \alpha \widehat{\mathcal{L}}\_{\text{res}} + (1 - \alpha) \widehat{\mathcal{L}}\_{\mathcal{O}} - \frac{\lambda}{2} \alpha(1 - \alpha)\Delta$$

and, as shown in Appendix A.3:

$$\mathfrak{R}\_n(\mathcal{H} \circ \tilde{\mathcal{C}}\_{\text{FADRM}}) \le \alpha \mathfrak{R}\_n(\mathcal{H} \circ \tilde{\mathcal{C}}\_{\text{res}}) + (1 - \alpha) \mathfrak{R}\_n(\mathcal{H} \circ \mathcal{O}) + L\_h \cdot \alpha(1 - \alpha)\Delta.$$

Substituting both yields:

$$D(\alpha) \le X\alpha + Y(1 - \alpha) + Z(\alpha - 1)\alpha,$$

where:

• $X := \widehat{\mathcal{L}}\_{\text{res}} + 2B \cdot \mathfrak{R}\_n(\mathcal{H} \circ \tilde{\mathcal{C}}\_{\text{res}})$
• $Y := \widehat{\mathcal{L}}\_{\mathcal{O}} + 2B \cdot \mathfrak{R}\_n(\mathcal{H} \circ \mathcal{O})$
• $Z := \frac{\lambda}{2} \Delta - 2B L_h \Delta$

Minimizing this quadratic yields the optimal merge ratio:

$$\alpha^\star = \frac{1}{2} + \frac{Y - X}{2Z}.$$

Since in practice we use temperature-scaled KL divergence, the strong convexity constant satisfies $\lambda \approx 1 + (C - 1)\exp(-\frac{d}{T})$, which approaches the number of classes $C$ as the temperature $T$ increases. Consequently, the bias term $\frac{Y - X}{2Z}$ becomes negligible relative to the dominant $\lambda$ term, and the optimal merge ratio $\alpha^\star$ remains close to 0.5.

This aligns with our empirical results in Table 4, where $\alpha = 0.5$ performs consistently well across datasets. Thus, our selection is both theoretically grounded and empirically validated.

W3: Discussion where FADRM fails.

We appreciate the reviewer’s comment and would like to clarify that the potential failure modes of FADRM to certain hyperparameters are indeed discussed:

• MRO (lines 240–243): We show that applying MRO without ARC leads to performance degradation, as the reduced resolution impairs the representational fidelity without the residual correction.
• Merge ratio $\alpha$ (lines 243–248): We demonstrate that a large $\alpha$ leads to excessive reliance on the optimized data, reducing robustness to overfitting.
• Residual frequency $k$ (lines 262–267): We observe that overly frequent residual injection introduces redundancy and leaves insufficient iterations per stage, leading to degraded synthetic images.

We agree these points deserve further emphasis and we will include an explicit subsection in the revision to summarize failure case. Thank you again for pointing this out.

W4: Lacks a clear metric to quantify information vanishing.

We appreciate the reviewer’s suggestion. To better quantify the issue of information vanishing, we report the generalization accuracy of models trained on distilled images across varying optimization steps (500–4000 iterations). As shown in the table below, prolonged optimization without ARC leads to performance degradation, indicating the issue of information vanishing. In contrast, applying ARC enables continued performance gains, demonstrating its ability to extract additional information during extended optimization. We will include and visualize this metric in the revision.

Q1 (1): Sensitiveness of the Hyper-parameter.

We thank the reviewer for the question. We clarify that FADRM is robust to both the merge ratio $\alpha$ and the number of residual injections $k$.

For $\alpha$, our theoretical analysis (see response to W2) suggests that the optimal value lies near 0.5. Empirically (Table 4), performance remains stable around this value, with less than 0.7% variation. Notably, $\alpha = 0.5$ achieves the best result on ImageNet-1K, and consistent trends are observed on CIFAR-100 (see table below), where $\alpha = 0.5$ also yields the strongest performance.

For $k$, moderate values (e.g., 2–4) balance optimization depth and information retention. Performance varies within 0.2% (ImageNet-1K) and 1% (CIFAR-100) as shown in left panel of table 5, with $k{=}3$ performs the best across diferent datasets.

Overall, FADRM requires minimal tuning and $k=3$ and $\alpha=0.5$ performs stably across different datasets.

Q1 (2): Adaptive Hyper-parameters during training.

We thank the reviewer for this insightful question. While our method supports adaptive adjustment of hyperparameters, we emphasize that the default configuration, merge ratio $\alpha = 0.5$ and number of residual injections $k = 3$, already yields strong and stable performance across diverse settings.

As established in our theoretical analysis, the optimal merge ratio $\alpha$ lies near 0.5. The remaining two hyperparameters are the downsampled input resolution $D\_{\text{ds}}$ and the number of residual injections $k$, which determines when residual information from the original dataset is injected.

To avoid information vanishing caused by prolonged optimization in each stage, we inject residuals only when the current synthetic image is deemed converged. We adopt the following convergence criterion:

• Cross-entropy loss $\mathcal{L}\_{\text{CE}} < 0.01$
• BatchNorm regularization $\mathcal{L}\_{\text{BN}} < 60$

To adaptively determine $D\_{\text{ds}}$, we use total variation (TV) to quantify image smoothness. Smoother images can be optimized at lower resolutions without loss of fidelity. Specifically, for a batch of images $x \in \mathbb{R}^{B \times C \times H \times W}$:

$$\text{TV}(x) = \frac{1}{B} \sum\_{i=1}^{B} \left( \| x\_i[:, 1:, :] - x\_i[:, :-1, :] \|_1 + \| x\_i[:, :, 1:] - x\_i[:, :, :-1] \|\_1 \right)$$

The downsampling factor is then computed as:

$$\text{ShrinkRatio} = \min\left(0.8, \frac{0.5}{\max(\text{TV}, 10^{-4})} \right), \quad D\_{\text{ds}} = \max\left( \lfloor D \cdot \text{ShrinkRatio} \rfloor,\ 128 \right)$$

As shown in the experiments blow on IPC=10, this procedure achieves performance comparable to manually tuned configurations. We will include these results in the revision. We agree this is a promising direction and sincerely thank the reviewer for highlighting it.

We sincerely thank you for your constructive comments. We are encouraged that you find our work novel with impressive theoretical insights and strong empirical validation. We would like to address the comments and questions below.

W1. While the limitations of the uni-level decoupled framework are discussed, the theoretical effectiveness of FADRM within this framework remains unproven.

Thanks for raising this point. We will formally analyze the impact of FADRM, particularly ARC to mitigate the limitations of the uni-level framework.

In FADRM's setting, $\hat{x}$ is generated by using both $f\_\theta$ and real image patch from $\mathcal{O}$ by iteratively fusing the real image patch to the intermediate optimized image. This introduces a new direct dependency:

$$\mathcal{O} \rightarrow \hat{x} \leftarrow f\_\theta$$

This breaks the conditional independence:

$$\hat{x} \text{ is not conditionally independent of } \mathcal{O} \text{ given } f_\theta$$

Thus, the Markov structure assumed in Data Processing Inequality no longer applies. By the chain rule:

$$I(\hat{x}; \mathcal{O}) = I(f\_\theta; \hat{x}) + I(\hat{x}; \mathcal{O} \mid f\_\theta)$$

• In the uni-level framework:

$$I(\hat{x}; \mathcal{O} \mid f\_\theta) = 0$$

• In our FADRM setting, the second term is strictly positive:

$$I(\hat{x}; \mathcal{O} \mid f\_\theta) > 0$$

Hence the mutual information is strictly increased in our FADRM settings.

W2. The use of Mixed Precision Training (MPT) primarily for efficiency lacks sufficient novelty.

We thank the reviewer for asking this question. We would like to clarify that our contribution lies not in the use of MPT itself, but in introducing optimization-level efficiency into dataset distillation, an area where most prior work focuses on architectural or initialization-level shortcuts (e.g., lighter backbones or real-image initialization). Our use of MPT is thus novel within this setting and highlights a promising direction for improving efficiency in future research.

W3. Misleading notation of "budget".

Thank you for the suggestion. We will revise the terms $b$ and $\mathcal{B}$ to $n\_{\text{iter}}$ and $N\_{\text{iter}}$, respectively, to improve clarity and avoid confusion.

Q1. Loss condition not used in the proof.

We thank the reviewer for raising this concern. The specific loss function (cross-entropy + BN matching) is not required for the proof; it is included to reflect typical objectives used in uni-level frameworks. Theorem 1 remains valid regardless of the loss formulation, as long as the original dataset $\mathcal{O}$ is not involved and the pretrained model is kept frozen during optimization, following directly from the data processing inequality.

Q2. Further clarification on Theorem 2.

We appreciate the reviewer’s request for further clarification on Theorem 2. Below we provide an intuitive and formal explanation, focusing in particular on Equation 8.

Theorem 2 aims to theoretically justify the generalization advantage of FADRM over training with uni-level framework's distilled data alone. Specifically, Equation 8 establishes a complexity gap between the original dataset $\mathcal{O}$ and the uni-level framework's distilled data $\tilde{\mathcal{C}}^{\text{res}}$:

$$\mathfrak{R}\_n(\mathcal{H} \circ \mathcal{O}) - \mathfrak{R}\_n(\mathcal{H} \circ \tilde{\mathcal{C}}^{\text{res}}) < -\frac{L\_h \Delta (L\_l + 2B\alpha)}{2B}$$

This inequality reflects the assumption that uni-level framework's distilled samples $\tilde{\mathcal{C}}^{\text{res}}$ being directly constructed to match the behavior of the target model, are easier to fit and therefore induce higher Rademacher complexity than orginal images. In contrast, $\mathcal{O}$ contains richer semantic variability, which naturally regularizes overfitting, resulting in a lower complexity.

Theorem 2 then shows that combining uni-level framework with the structured real data leads to a better generalization bound. Under the assumption, the generalization bound of $h\_{\mathrm{FADRM}}$ is rigorously shown to be tighter than that of $h\_{\mathrm{res}}$, i.e.,

$$\widehat{\mathcal{L}}\_{\mathrm{FADRM}} + 2B \cdot \mathfrak{R}\_n(\mathcal{H} \circ \tilde{\mathcal{C}}\_{\mathrm{FADRM}}) < \widehat{\mathcal{L}}\_{\mathrm{res}} + 2B \cdot \mathfrak{R}\_n(\mathcal{H} \circ \tilde{\mathcal{C}}^{\text{res}}).$$

The insight of this theorem is that, when synthetic data is highly optimized and thus induces greater hypothesis complexity, combining it with the more structured and regular original data can lead to a tighter generalization bound, accounting for both empirical risk and theoretical analysis.

Q3 (1): Comparison betwwen image parameterization and MRO.

We thank the reviewer for this insightful question. Multiple Resolution Optimization (MRO) is a training efficiency technique that reduces the computational cost of pixel-level optimization by periodically operating at lower resolutions. In contrast, image parameterization replaces pixel optimization with a latent code and a shared decoder, aiming to reduce parameter count and improve representational compactness.

Key advantages of MRO over image parameterization include:

• Lower Hyperparameter Sensitivity: MRO introduces only a single downsampling factor $D_{\text{ds}}$, while parameterized methods require careful tuning of latent size, decoder architecture, and regularization.

• Preservation of Fine Details: MRO retains full-resolution pixel expressiveness, which is crucial for high-resolution datasets like ImageNet.

• Simplicity and Broad Compatibility: MRO is easy to implement, requires no auxiliary modules, and can be seamlessly integrated into any optimization-based distillation framework.

Q3 (2) Apply image parameterization within FADRM Framework.

We thank the reviewer for this insightful question. FADRM is compatible with image parameterization techniques such as HaBa, etc. In this setting, instead of initializing pixel-space learnable synthetic images, we initialize trainable latent bases and trainable hallucinator networks. During training, images are decoded via the hallucinators, and the FADRM loss is computed on the decoded outputs. Gradients are backpropagated to update both components. Residual feature injections can still be applied periodically (e.g., every 500 iterations) to enhance information retention. Additionally, MRO can be integrated by performing forward passes at lower resolutions to reduce training cost.

We appreciate the reviewer’s suggestion and believe that combining image parameterization with uni-level frameworks opens up promising directions for future research.

Q3 (3) Adaptive $D_{\text{ds}}$.

We thank the reviewer for the thoughtful question. While DDiF is resolution-agnostic during generation, it still requires a fixed output resolution and cannot dynamically select the optimal scale for each image. Therefore, it does not directly help determine the downsampling factor $D_{\text{ds}}$ automatically in our setting.

To avoid manually select $D_{\text{ds}}$, we use an adaptive strategy based on total variation (TV) to estimate image smoothness. Smoother images can tolerate greater downsampling without loss of fidelity. Given a batch $x \in \mathbb{R}^{B \times C \times H \times W}$:

$$\text{TV}(x) = \frac{1}{B} \sum_{i=1}^{B} \left( \| x_i[:, 1:, :] - x_i[:, :-1, :] \|_1 + \| x_i[:, :, 1:] - x_i[:, :, :-1] \|_1 \right)$$

We compute:

$$\text{ShrinkRatio} = \min\left(0.8,\ \frac{0.5}{\max(\text{TV}, 10^{-4})} \right),\quad D_{\text{ds}} = \max\left( \lfloor D \cdot \text{ShrinkRatio} \rfloor,\ 128 \right)$$

As shown below, this adaptive strategy achieves performance on par with manual tuning:

We will include these results in the revision and thank the reviewer for inspiring this direction.