Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning

Thank you for your constructive and insightful review, these suggestions have helped us identify and strengthen important aspects of our experimental analysis.

Weaknesses

W1.1: Including more runtime comparison.

A1.1: We have included a comparison of the time cost of our method with generator-based model inversion baselines (e.g., ABD, R-DFCIL, DCMI) using the ResNet-32 backbone on the CIFAR-100 5-task setting, as shown in Table below. While our method requires more time, it achieves superior performance. Additionally, we compare our method with a generator-based approach using the CLIP backbone on CIFAR-100 with the MoE-Adapter baseline. As shown in Table below, our method achieves higher accuracy with lower runtime.

W1.2: Control iteration steps for performance comparison.

A1.2 For the full-model inversion and SMI baselines, we originally used their default iteration settings to ensure optimal performance. As shown in Table below, we re-evaluate all methods using a fixed budget of 800 iteration steps. Under this constraint, the performance of both baselines drops noticeably, further demonstrating the effectiveness of our method within the same iteration budget.

W2.1: GPU efficiency comparison.

A2.1 We have included GPU FLOPs and memory usage for data generation in each task in Tables above. Our method requires more FLOPs and GPU memory when using the ResNet-32 backbone. However, with the CLIP backbone, our method is more efficient using fewer FLOPs and less memory, since generating only a small amount of synthetic data (5 samples per class in our experiments) is sufficient to mitigate forgetting. While generator-based methods are efficient with small backbones, they struggle to effectively capture the knowledge encoded in large pre-trained models. Even with a larger generator (containing more parameters than those used in the ResNet-32 experiments) and 6,000 training steps, generator-based inversion fails to significantly improve performance over the baseline. These results demonstrate the effectiveness of our proposed method.

W2.2: Discussing differences between image-noise-based and generator-based methods.

A2.2 While generator-based methods are more efficient at generating samples after training, image-noise-based methods offer two key advantages:

1. Since input samples are independent, they can be generated in parallel across multiple GPUs without synchronization. This allows image-noise-based methods to scale more efficiently with additional GPUs, further reducing the overall time cost.
2. When using knowledge distillation, teacher logits can be computed in a single forward pass after sample generation for image-noise-based methods. In contrast, generator-based methods generate a synthetic batch and compute teacher logits at each training step, requiring an extra forward pass in every iteration.

Our PMI method reduces the number of iterations through better initialization, and can further leverage these advantages.

Thank you again for your thoughtful comments. We will incorporate all the analyses and discussions into the revised version of our paper.

Thank you for your constructive and insightful review, these suggestions have helped us identify and strengthen important aspects of our experimental analysis.

Weaknesses

W1: Including more runtime comparison.

A1: We have compared the training time of our method with generator-based model inversion baselines (e.g., ABD, R-DFCIL, DCMI) using the ResNet-32 backbone on the CIFAR-100 5-task setting, as shown in Table below. While our method requires more running time, it achieves superior performance. Additionally, we compare our approach to a generator-based method using the CLIP backbone on CIFAR-100 with the MoE-Adapter baseline. As shown in Table below, our method achieves higher accuracy with lower runtime. Moreover, unlike generator-based methods, our approach treats each sample independently, allowing for efficient parallel generation across multiple GPUs, which can further reduce the overall time cost.

W2: Missing non-exemplar CIL baselines.

A2: Thank you for your comments. We have included these baselines in our experimental setup on the CIFAR-100 dataset using the ResNet-32 backbone for additional comparison, as shown in Table below. We will also include these baselines in the revised version of our paper.

W3: Minor clarity issues.

A3: Thank you for your suggestions! We will make the following revisions in the next version of our paper:

1. simplify the notation of class indices,
2. change the layout of Figure 1 to a left-to-right orientation,
3. correct the wrong bold formatting in Table 4,
4. ensure a consistent style across all performance tables.

Questions

Q1: Ablation study on temperature.

A1: We conducted experiments with different temperature values on the CIFAR-100 dataset using the CLIP backbone and MoE-Adapter baseline, as shown in Table below. Since the contrastive model is lightweight and converges quickly, its output is stable across temperature settings. Moreover, because the contrastive loss is used for feature similarity ranking, the temperature scaling does not affect the relative ordering. As a result, the temperature has negligible impact on performance, and we omit this term in our implementation.

We sincerely thank you for your detailed and thoughtful review, these comments have been particularly helpful for enhancing and clarifying our work.

Weaknesses

W1.1: Motivation to work with CLIP model.

A1: As we claimed in line 8-11, generating samples with model inversion solely from integer labels can lead to a feature distribution gap between real and synthetic data. This issue is particularly pronounced in the CLIP feature space, which encodes rich semantic information. When synthetic features drift from the real data distribution, it implies a shift in semantic meaning. Training on such misaligned samples where the label corresponds to incorrect semantics can degrade both the pre-trained knowledge of CLIP and the task-specific knowledge learned from previous tasks, resulting in more severe forgetting. We conduct an experiment on CIFAR-100 using classification loss for model inversion with the MoE-adapter as the baseline, and report the results in Table below. The performance is worse than the non-replay baseline (79.40 in Table 2), highlighting a more severe synthetic/real drift issue in the CLIP setting. Moreover, model inversion on CLIP requires significantly more computation and more iterations to converge compared to smaller backbones (e.g., ResNet). Thus, both the feature drift and computational cost are more substantial challenges when applying model inversion to CLIP models.

W1.2: “Unseen” class definition is unclear.

A1.2: In section 3.4 of our work, “unseen” classes refer to those that the model has not been adapted to during training. While CLIP models may have been pre-trained on semantically similar classes, our method demonstrates the feasibility of generating data for specific target classes and improving performance on them through adaptation using the generated samples. We will revise our work to provide a clearer definition.

W2: Improvement over baseline.

A2: R-DFCIL is a strong baseline in model inversion for continual learning settings. As shown in Table 5 of Appendix D.1, the more recent method DCMI (2024) does not outperform R-DFCIL under our experimental setup. Our method provides less performance improvement on 20-task settings which is a challenging setting for all data-free continual learning methods, and our method consistently outperforms R-DFCIL one all the settings, demonstrating the effectiveness of our method.

W3: Model for generating data.

A3: The CLIP-based experiments in Table 2 follow the standard data-free setting, where replay data is generated from the continually trained model. In the experiments shown in Figure 2, we assume that images from the last task are unavailable (unseen), and generate them using the original CLIP model, since the continually trained model may have lost some pre-trained knowledge. Replay data for earlier tasks is still generated by the continually trained model. Therefore, our experimental setup adheres to the standard formulation of the data-free continual learning problem.

W4: Storing input statistics.

A4: Thank you for your comments! First, unlike previous inversion methods for data-free continual learning that rely solely on layer-specific statistics from BatchNorm layers, our method maintains a small set of additional statistics—specifically, the mean and standard deviation for each data set. These statistics are highly compressed and do not reveal any original data, making the approach both practical and compliant with the data-free setting. Storing such minimal statistics does not violate the data-free assumption, and using BatchNorm statistics is a common and accepted practice in prior model inversion works. Importantly, storing input statistics is not the main contribution of our method. Since CLIP is pre-trained on a highly diverse dataset, it can support model inversion even without relying on stored statistics. To verify this, we conducted experiments on CIFAR-100 using the MoE-Adapter baseline without input statistics. As shown in Table below, performance dropped noticeably, especially on the fine-grained CUB-200 dataset. These results indicate the importance of input-statistics alignment for maintaining high-quality inversion.

We appreciate the reviewer’s feedback and will include all related discussions and clarifications in the revised version.

Thank you for your constructive feedback and valuable suggestions, these points have helped us better clarify and improve the presentation and evaluation of our method.

Weaknesses

W1: Computation overhead of PMI.

A1: We have compared the time cost of our method with full-model inversion and Sparse Model Inversion (SMI) in Appendix F. Our method uses 200 steps for PMI and 600 for full-model inversion. In contrast, the full-model inversion baseline requires 3,400 steps and SMI requires 4,000 steps. Our method achieves better performance while requiring less total time than these baselines. These results indicate that PMI introduces lower overhead and substantially reduces the total number of iterations. Furthermore, PMI allows for larger batch sizes due to lower GPU memory usage per layer, enabling more efficient utilization of GPU parallelism. As a result, the extra initialization cost can be amortized, reducing the per-sample computation time.

W2: Performance improvement of CFS.

A2: Our motivation for using CFS is to better preserve feature distribution information compared to Gaussian modeling. In our experiments, CFS consistently improves performance, including when applied to the CLIP backbone, as shown in Tables 3 and 4. These results support our insight that preserving richer distributional information through contrastive modeling can lead to further performance gains. Additionally, CFS employs a lightweight MLP to better capture the true feature distribution, incurring only minimal computational overhead.

W3: Limitations not comprehensively discussed.

A3: Due to space limitations, we primarily discussed the limitation that more sophisticated methods could be employed for generating data of unseen classes. We will expand on this and provide a more detailed discussion of limitations in the revised version.