Beyond Finite Data: Towards Data-free Out-of-distribution Generalization via Extrapolation

1. They claim that their approach is "data-free". But, I disagree with this point since they employ a stable diffusion model to augment data, and the stable diffusion model is trained on a large amount of data. I think their approach shows high performance, especially in the domains in which the stable diffusion is good at generating. Therefore, I think the data to train the stable diffusion includes their target domain. In this sense, I think their approach is still "data-dependent".
2. Their way of generating images from the text2image model is reasonable, yet a bit straightforward and is not very novel, considering the existing work like [1].
3. Compared to the approach like [1], their prompt design relies only on the heuristic knowledge of the dataset. This is because their approach does not feed any training images into their framework. Therefore, if the training images look totally different from images generated from stable diffusion, their approach can fail.

[1] Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation

1. Lots of missing baselines to compare against. Please compare and cite the following relevant works:

• P. Teterwak et al, ERM++: An Improved Baseline for Domain Generalization, https://arxiv.org/abs/2304.01973, ICML 2023 Workshop on Spurious Correlations, Invariance and Stability
• C. Liao et al, Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning, CVPR 2024
• Vishaal Udandarao et al., SuS-X: Training-Free Name-Only Transfer of Vision-Language Models, ICCV 2023
• L. Dunlap et al, Using Language to Extend to Unseen Domains, ICLR 2023

Namely, ERM++ reports an average accuracy of 81.4% in your experimental setup Table 1 (see Table 1 in ERM++ paper). The Descriptor/Word soup paper has been shown to outperform WaffleCLIP and other GPT descriptor baselines, and is currently SoTA. The SuS-X paper is another missing baseline that uses task information to retrieve data and train models. The LADS paper performs domain extension using language to improve accuracy on target domain using language description as input.

2. Lack of key datasets. The authors have not provided results on DG methods on ImageNet and its variants (Adversarial, Sketch, Rendition, V2), and WILDS distribution shifts. The datasets presented are not challenging for DG and saturated in the literature.

3. It's not clear that such a complex method is warranted to obtain these marginal gains. The method uses large LLMs (GPT4) in addition to text to image generation models (Stable Diffusion 2).

4. The authors need to consider the training data used to train the LLM and text-to-image generation model. Namely, addressing important limitation such as what happens when the data is so different, such as a task on satellite image classification for example, or biology problems, when the LLM and image generation is on Internet-scale datasets. It is well known that on such fine grained specific problems, CLIP-trained models are quite poor.

• A big weakness of this paper is the key assumption, that $\mu$ and $\mu'$ are close for a language model and the true domain. The authors are able to completely attribute generalization loss to $\epsilon$ and ignore it for the rest of the paper. However, assuming that an LLM does a good job of approximating $\mu$ is a large claim to make. I think it would be good to see some sort of attempt at measuring $\epsilon$, such as even seeing how much this distance changes between two estimated $\mu', \mu''$ generated by two independent LLMs.
• The authors then say this error can be recovered by increasing $m$, and $n$, but by how much? Would this be a realistic or computationally-feasible amount of data? From an experiment perspective, the baseline methods must be fairly compared with similar values of $m$ and $n$. This is another question I had with the results, but how much more data did your method see than baselines? How would results change if you fixed the same amount of data for baselines, ERM and your method? Otherwise, I do not feel these are fair comparisons.
• In practice, the LLM and DM are trained on large internet data. It is highly likely that they have seen the image datasets used in this paper (for the DM) or a textual description of these domains. This may provide evidence that the $\epsilon$ is actually quite small, but then raises another question: is it fair to leverage the knowledge of these models for domain generalization given they have seen the generalizing domains?
• I disagree that data-free generalization can democratize machine learning. Eliminating real-world data collection just results in our representation and learning of certain groups more biased and creates feedback loops where our generative models create biased data that we use to train biased models. Consider, for instance [1], where the authors find that diffusion models generate stereotyped images of various objects, traits, and ethnicities (See Fig. 1 for an illustrative example). In this case, relying on a diffusion model to generalize to a new domain (ie. images from Africa) will not improve performance for users in Africa but instead create harmful stereotypes and potentially not improve model performance.
• When you increase your domains, do you keep the total number of samples seen constant? Otherwise, it seems difficult to say that it was the domains that led to more performance, rather than just training a model on more data.
• In your analysis you mention "LLM-generated prompt further extracts knowledge" and later that "the importance of knowledge injected by LLMs that benefits generalization". Can you elaborate on this? Its a pretty vague sentence to me and I would like to understand more clearly what your main thesis is with regards to the importance or contribution of LLMs to domain generalization.

Minor suggestions:

• You should bold the best scores in your tables, especially Table 1 as it is so large.
• Line 317, Evaluation is capitalized
• When you use quotes, make sure you use ``text'' so that latex formats the quotes properly.

[1]: Bianchi, Federico, et al. "Easily accessible text-to-image generation amplifies demographic stereotypes at large scale." Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 2023.

• The theoretical argument is fairly weak, the assertion that an LLM can approximate the meta distribution over samples is unfounded

• Unclear how many samples are generated and how many samples the model has access to. Does the model using your method have access to more samples than the other baselines?

• Generating more samples may help the model if the samples are of sufficient quality. The method does not guarantee this and seems entirely dependent on the generation performing well.

• Unclear how many times the experiments are run, it is also difficult to compare results as the models have seen different samples

• Experiment on the scaling benefits somewhat unclear, which dataset is being considered?

• Variance analysis on PACS only does not really say much of the variance we can expect in other tasks

Please see the questions section for more information.