Provably Improving Generalization of Few-shot models with Synthetic Data

We would like to thank the reviewer for your positive evaluation and valuable feedback to our work. We would like to address your remaining concerns as follows:

1. Results for more extreme few-shot conditions

Thanks for your insightful suggestion. We conduct experiments on 3 datasets that was also used in the DataDream (DD) paper. The results are shown in the following table:

Overall, our method underperforms the baseline in the extreme 1-shot scenario. With only one real sample, the regularization terms in our loss function become small, reducing model robustness and possibly causing performance drops. However, our method significantly outperforms the baseline in 4-shot and 8-shot settings. Thus, extremely limited real data case remains a limitation of our approach.

2. Results if only synthetic data were used

We want to gently point out that our method are designed to take into account both real and synthetic data. Our loss function was designed to match between prediction of real and synthetic samples and regularization are computed on regions containing real data only. In order to adapt our method into the case of only synthetic data, one can remove the discrepancy term and loss on real data from the loss function and compute the robustness loss on all regions that contains at least 2 synthetic samples. Overall, the loss function can be rewritten as: $F(\mathbf{G},\boldsymbol{h})+\lambda_2\frac{1}{g}\sum_{i}\sum_{\boldsymbol{g_1},\boldsymbol{g_2}\in \mathbf{G_i}}\frac{1}{g_i}\\|\boldsymbol{h}((\boldsymbol{g_1})-\boldsymbol{h}(\boldsymbol{g_2})\\|$.

We conduct experiments to test the effectiveness of this loss function in some small and medium-sized datasets. The results are shown below:

Our method outperforms the baseline on 3 out of 5 datasets, comparable in 1 and worse in 1 dataset. On average, our methods still perform better than the baseline, showcasing the necessity of the robustness regularization. However, we admit that these increases are marginal, and much less significant compared to our full method, which takes into account both discrepancy and robustness terms.

3. The impact of the number of synthetic samples per class. Results of only the 16-shot real data

We chose 500 to be consistent with the main experiment in the DataDream paper, ensuring a fair comparison in Tables 1 and 3.

To further investigate the effect of the number of synthetic samples, we conduct more experiments summarized below:

The results confirmed that our method consistently ourperform baselines when varying synthetic data sizes. The results of only 16-shot real data was reported in the "Real-finetune" row in Table 1 in our paper.

4. Better solution for generation of synthetic data for the FGVC Aircraft datasets

Stable Diffusion is infamously bad on aircraft data. In practice, we can choose a better generative model on aircraft data. A future direction for our method involves using the theoretical framework for optimizing synthetic data by fine-tuning generator or filtering synthetic data by using our loss function as a criterion.

5. Detailed hyperparameter analysis

We want to gently redirect the reviewer's attention to our response to Reviewer hvE3 (part 4) to see our detailed discussion about the hyperparameter selection. [https://openreview.net/forum?id=L6U7nYc4ah&noteId=sANtK8HtxS]

6. Essential References Not Discussed

We thank the reviewer for highlighting missing references. We will include the following discussion of the mentioned papers and add further references on prompt and adapter tuning in the revised version:

Recently, methods such as prompt and adapter tuning have become prominent for enhancing few-shot classification performance, including [r1], [r3], etc. Moreover, [r2] uses synthetic images to improve performance but relies on combining four large pretrained models, significantly increasing implementation complexity.

7. Not visually appealing figures and awkward sentence

Thanks for suggesting these issues, we will improve the presentation in the new version.

8. Stable Diffusion selection

We chose it to be consistent with the baseline for fair comparison. DataDream also uses Stable Diffusion of the same version as the generator.

We would like to thank the reviewer for praising our work and your positive evaluation. We would like to reiterate the key novelties of our work:

1. We introduced a novel generalization bound for a model trained with synthetic data augmentation. It suggests an effective way to generate synthetic samples, and to train a model by minimizing the distribution gap between real and synthetic data and ensuring local robustness around real samples.
2. Guided by this theory, we designed an algorithm to jointly optimize models and data partitioning to enhace models' performance.
3. Experimental results show that our method outperforms state-of-the-art baselines across multiple datasets. Our more efficient lightweight version also shows comparable performance with the baselines.

Should you have any more questions or concerns, we would be very happy to address them.

We thank the reviewer for your positive evaluation and valuable suggestions. We address your concerns below.

1. Key assumptions made in deriving the generalization bounds, and their realisticity

The only assumptions are (1) data samples are i.i.d. and (2) loss function is bounded and Lipschitz w.r.t. the model. Note that for classification problems, many common losses satisfy our assumptions, such as absolute loss, square loss, Hinge loss. Therefore, those assumptions are realistic and reasonable, widely used in the literature.

2. Tightness and practical implications of the bounds

Tightness: We consider several factors regarding the tightness of our bounds:

• Our bound scales with $O(\sqrt{K})$, which can be sub-optimal for large $K$ (partition size). Therefore, we should not choose a large $K$. Luckily, for few-shot problems, our bound does not allow a large $K$ which goes beyond the number of real samples.
• For fixed $K$ and number $n$ of real samples, our bound scales with $O(g^{-0.5})+const$. This indicates our bound is optimal w.r.t. the number of synthetic samples. The reason is that the error of the best model is at least $O(g^{-0.5})+const$, for a hard learning problem, according to Theorem 8 in [Than et al. (2024). Gentle local robustness implies Generalization].
• Finally, since $n$ can be very small, our bound can be far from the true error of a model. This is a common limitation of any theoretical error bound for limited data. To the best of our knowledge, limited data poses a big (open) challenge for estimating the model's true error.

Practical implications: We have discussed various implications after Theorem 3.3 in the paper. We point out some useful roles of the main terms. We also have Sections 5.3 and 5.4 to investigate empirically the influence of robustness and discrepancy terms.

3. Ablation study limited to 3 datasets

We conducted ablation study on 4 datasets of varying sizes, which we believe sufficiently illustrate our method's effectiveness. We will add more results on the remaining datasets in the Appendix.

4. Hyperparameter selection process

First, we balance the discrepancy ($\lambda_1$) and the robustness ($\lambda_2$), using coordinate descent to determine this optimal ratio. We then chose a suitable value of $\lambda$ associated with the loss on real data. The results are reported below:

We then selected the actual values of ($\lambda_1$/$\lambda_2$) from a set of values that follows the 1/10 ratio. We chose their values to maintain a good ratio between the regularization and loss.

For lightweight version, we directly adopted the learning rate and weight decay from the DISEF paper, as our synthetic sample sizes matched theirs. For the full version, we employed grid search with early stopping to choose them.

These hyperparameter value sets are presented in Appendix B. The others follow exactly as in the baselines.

Note that hyperparameter selection was performed only during classifiers tuning, separately from generator tuning and synthetic data generation, minimizing additional computational overhead.

5. Relation to existing theoretical work on domain adaptation or generalization with synthetic data

We thank the reviewer for useful suggestion. We have discussed the existing theoretical works of generalization with synthetic data in the last paragraph of section 2.1. We will add more related references.

6. Why select parameters from DataDream (DD), DISEF, and IsSynth as baselines? Were hyperparameters tuned for fair comparison?

We chose hyperparameters to be consistent with DD. Since DD's authors reran DISEF and IsSynth on the same number of synthetic data and generative models, it serves as a good baseline for us. Additionally, experiments confirmed that directly using baselines' learning rate and weight decay yielded optimal results across datasets.

7. In ablation studies, why not include results on the 4 datasets used in Table 3?

We followed the Resnet50 setup from DD in table 3 for fair comparison. However, in the ablation study, we aim to evaluate our method's performance across datasets of varying scales, whereas the datasets in Table 3 are similar in size.

8. Lightweight version give better performance for some datasets

The lightweight version performs better on 2 datasets: Caltech and DTD. For these two datasets, we observed that the methods with generator fine-tuning (DD and our full version) perform equal or worse than the others without it (DISEF, IsSynth, and our lightweight).