Experimental suggestion: Error estimate for FID scores

We thank the reviewer for the valuable suggestion regarding error estimation.

We would like to clarify that the real-world experiments in Section 5.2 were run only once due to the long training time. Based on these trained models, we additionally performed multiple samplings using five different random seeds to estimate the randomness in calculating FID scores following the reviewer's suggestion. The mean values and standard deviations of FID scores over multiple samplings are reported in the table below (corresponding to Table 1 of our submission).

We will include these results in the revised version.

Typo: Missing parentheses

Thank you for pointing this out. We will correct this in the revised version.

Q1: Experiments on diffusion models and the theory for ARMs & EBMs

We thank the reviewer for the insightful question.

EBMs, as mentioned in lines 51-55 in our submission, are a general and flexible class of generative models closely connected to diffusion models. To be specific, first, the training and sampling methods in [1,2] are directly inspired by EBMs. The distinction is that EBMs parameterize the energy function, while diffusion models parameterize the score function, which is the energy function's gradient. Second, [3] shows that under a specific energy function formulation (Equation (5) in their paper), EBMs are equivalent to constrained diffusion models. Their experimental results (Table 1, Rows A and B in their paper) indicate that the constraint has a minor impact on generative performance. Thus, our diffusion model experiments provide insight into EBMs' behavior in real-world settings to some extent.

Additionally, we have added supplementary simulations for ARMs according to the formulation in Section 4.2. The empirical TV errors exhibit similar trends as theoretical bounds in Theorem 4.3 regarding several key factors—the number of sources $K$, sample size $n$, and data length $D$. Due to space constraints in the rebuttal, please refer to our response to Reviewer 7LUU (Q1) for detailed experimental settings and results.

We will add the above discussions for EBMs, and simulation results along with implementation details for ARMs in the revised version of our paper.

Q2: Obstacles for practical application

As discussed in Section 7 in the submission (lines 422-432, right column), our theoretical formulation of multi-source training through conditional generative modeling abstracts real-world scenarios to some extent. In practice, conditions may not be explicitly given (e.g., in language models) or may involve multiple source labels (e.g., large-scale image generation).

Our analysis provides a first step toward understanding multi-source training under a simplified yet reasonable setting. Extending it to more complex, fine-grained multi-source interaction scenarios is a valuable direction for future work. Possible approaches might include: characterizing distribution similarity without explicit conditions [1,2] or investigating the multiple-label case by compositional generative modeling [3, 4].

[1] Ben-David, S., & Borbely, R. S. (2008). A notion of task relatedness yielding provable multiple-task learning guarantees.

[2] Jose, S. T., & Simeone, O. (2021). An information-theoretic analysis of the impact of task similarity on meta-learning.

[3] Okawa, M., Lubana, E. S., Dick, R., & Tanaka, H. (2023). Compositional abilities emerge multiplicatively: Exploring diffusion models on a synthetic task.

[4] Lake, B. M., & Baroni, M. (2023). Human-like systematic generalization through a meta-learning neural network.

Q3: Connection between FID and theoretical guarantees

Our theory provides guarantees for the average TV distance (lines 142-155, left column), which quantifies distribution estimation quality but is incomputable without access to the true conditional distributions.

Therefore, in real-world experiments (Section 5.2), we use FID as a practical alternative. FID measures the similarity between generated and real data distributions by comparing their feature representations in a pretrained neural network. It is widely used to evaluate image generation quality and serves as the best available metric for our setting.

We will add the above discussion to clarify the choice of FID in the revised version.

Q1: Intuitive explanation for upper bracketing number

The $\epsilon$-upper bracketing number is a notion to quantify the complexity of an infinite set of functions. The key idea is to construct a finite collection of "brackets" that enclose every function in the set within a small margin.

To illustrate this, consider a simple example. Suppose we have the infinite function set $\mathcal{F} = \lbrace f(x) = c:x\in [0,1];c \in [0,1] \rbrace,$ which consists of all constant functions taking values in the interval $[0, 1]$. We can construct an $\epsilon$-upper bracket for $\mathcal{F}$ by defining a finite set $\mathcal{B} = \lbrace b(x)=k\epsilon : k=0,1,\dots,\lceil 1/\epsilon \rceil \rbrace,$ which contains $\lceil 1/\epsilon \rceil+1$ functions. Then, for any function $f \in \mathcal{F}$, there exists a bracket function $b \in \mathcal{B}$ such that: (1) For all $x \in [0,1]$, the bracket function is always an upper bound: $b(x) \ge f(x)$. (2) The total "gap" between $b$ and $f$, measured by the integral $\int_0^1 b(x) - f(x) dx$, is at most $\epsilon$. Therefore, the $\epsilon$-upper bracketing number of $\mathcal{F}$ is at most $\lceil 1/\epsilon \rceil+1$.

In our paper, we extend this idea to conditional probability spaces. There, each condition defines its own function set, and we construct corresponding upper brackets that ensure every conditional distribution is approximated with a small error uniformly across conditions.

We will include additional intuitive explanations and diagrams to make this idea more accessible in the revised version.

Q2: Definition of the estimation error

Your interpretation is essentially correct. In our paper, we define the error in terms of the average TV distance (see Equation 4 on line 147, left column). This metric evaluates the accuracy of conditional distribution estimates across all $K$ sources by averaging the error over each source. Therefore, even for single-source training, the error bound is related to $K$ because it aggregates the errors from the $K$ separate models.

Q3: Guarantee on one specific source

You have correctly captured the main idea. Our theory demonstrates that, in terms of the average distribution error, multi-source training has a better guarantee than single-source training. However, this does not necessarily imply that for every individual source, the corresponding multi-source model will yield lower error than a dedicated single-source model. Thus, your example is consistent with our theoretical findings.

Q4: Real-world experiments with larger $N$ and $K$

We would like to clarify that the selection of sample sizes and the number of classes in the experiments in Section 5.2 was influenced by several inherent characteristics of ILSVRC2012 dataset:

• Sample Sizes: The maximum number of images per class in ILSVRC2012 is 1300, so we selected sample sizes of 1000 and 500 images per class, which are common choices.

• Number of Sources: Given that distribution similarity levels were manually defined, it was difficult to establish a large number of structured subdivisions. To be specific, to ensure reasonable similarity levels for the controlled experiment, we designed two-level tree structure for the dataset, as shown in Figure 3 on Page 35 of our submission. Overall, we divided the whole ILSVRC2012 into 10 high-level categories (mammal, amphibian, bird, fish, reptile, vehicle, furniture, musical instrument, geological formation, and utensil). Each category was further subdivided into 10 subsets (e.g., for mammals, we have Italian greyhound, Border terrier, standard schnauzer, etc.). Defining such semantically meaningful and mutually exclusive divisions is not trivial. As a result, the number of classes within each similarity level in our experiments is limited to 10.

Additionally, for the 10-dimensional Gaussian example in Section 5.1, we used a maximum sample size of $N = 5000$ and $K = 15$ (see Figure 1(a) and (b)), which we believe are sufficiently large to verify the theoretical predictions in that case.

We will add the above explanations for the experimental settings in our revised version.

Q5: Experiments for ARMs or EBMs

Following the reviewer's suggestion, we have added supplementary simulations for ARMs and further illustrations for EBMs. Generally speaking, for ARMs, the empirical TV errors exhibit similar trends as theoretical bounds in Section 4.2 regarding several key factors—the number of sources $K$, sample size $n$, and data length $D$. For EBMs, we clarify their connection with the diffusion model experiments. Due to space constraints in the rebuttal, please refer to our response to Reviewer 7LUU (Q1) for details.

Q6: Consistency of models used for multi/single

Yes, for both theoretical analysis and empirical experiments, the models used for multi-source and single-source training are exactly the same across all settings, such as model architecture, number of parameters, initialization, and optimizers.

Q1: Characterization of multi-source advantage

We thank the reviewer for the insightful comment.

We would like to clarify that the advantage of multi-source training is indeed measured by the model parameter sharing, while the degree of the model parameter sharing reflects the source distribution similarity under our theoretical formulation (lines 94-108, right column).

We understand the reviewer’s concern. In the Gaussian model (Section 4.1), $\beta_{sim} = \frac{d - d_1}{d}$ measures the proportion of shared mean vector dimensions, which seems to correspond to the property of the ground truth distribution. While for EBMs (Section 4.3), $\beta_{sim} = \frac{S}{S+d_e}$ is based on model parameters, which does not explicitly represent the data distribution itself.

Despite this difference, in both cases, $\beta_{sim}$ is fundamentally defined by the extent of parameter sharing across sources. The distinction arises from the modeling paradigm: the Gaussian case assumes a parametric form for distributions, where model parameters (e.g., mean vectors) explicitly encode data properties, whereas EBMs use neural networks as a function approximator to fit probability densities without a predefined distributional form, making no explicit connection between parameters and data.

We will add detailed clarification on the relationship between parameter sharing, distribution similarity, and the advantages of multi-source training in the revised version.

Q1: Close alignment in Figure 1

We appreciate the reviewer’s careful examination of Figure 1.

As detailed in lines 339-340 of our submission (the caption of Figure 1), the empirical and theoretical values are plotted on separate vertical axes: empirical values correspond to the left axis, while theoretical values correspond to the right axis. This visualization normalizes differences in constants between empirical results and theoretical bounds, emphasizing the comparison of trends rather than absolute values.

We will highlight the dual-axis annotation in Figure 1 in the revised version to avoid any potential confusion.

Q2: Experiments for ARMs

We thank the reviewer for the valuable comment.

Following the reviewer's suggestion, we have conducted supplementary simulations for ARMs according to the formulation in Section 4.2. Experimental settings and results are presented below. Generally speaking, the empirical TV errors exhibit similar trends as theoretical bounds in Theorem 4.3 regarding several key factors—the number of sources $K$, sample size $n$, and data length $D$.

In all experiments, we define a ground-truth sequential discrete distribution, enabling exact computation of the practical TV error. We fix the vocabulary size $M=2$, neural network configurations with $d_e=W=64$, $L=5$, $B=1$. Then we vary $K$ in $[1, 3, 5, 7, 10]$, $n$ in $[1000, 3000, 5000, 10000, 30000]$, and $D$ in $[10, 12, 14, 16, 18]$ to examine alignment of practical TV error with theoretical bounds. For each setting, the batch size and learning rate are selected from $\lbrace 100, 300, 500 \rbrace$ and $\lbrace 10^{-5}, 10^{-4}, 10^{-3} \rbrace$ for the minimum likelihood. Empirical TV errors are presented in the following tables:

The results show consistent trends between practical TV error and theoretical bounds with respect to $n$, $K$, and $D$, i.e., the TV error decreases as $n$ and increases with $K$ and $D$, and multi-source training generally outperforms single-source training.

We will add the simulation results and implementation details for ARMs in the revised version.

Q3: Real-world experiments are on small datasets

We would like to clarify that the selection of sample sizes and the number of classes in the experiments in Section 5.2 was influenced by several inherent characteristics of ILSVRC2012 dataset:

• Sample Sizes: The maximum number of images per class in ILSVRC2012 is 1300, so we selected sample sizes of 1000 and 500 images per class, which are common choices.

• Number of Sources: Given that distribution similarity levels were manually defined, it was difficult to establish a large number of structured subdivisions. To be specific, to ensure reasonable similarity levels for the controlled experiment, we designed two-level tree structure for the dataset, as shown in Figure 3 on Page 35 of our submission. Overall, we divided the whole ILSVRC2012 into 10 high-level categories (mammal, amphibian, bird, fish, reptile, vehicle, furniture, musical instrument, geological formation, and utensil). Each category was further subdivided into 10 subsets (e.g., for mammals, we have Italian greyhound, Border terrier, standard schnauzer, etc.). Defining such semantically meaningful and mutually exclusive divisions is not trivial. As a result, the number of classes within each similarity level in our experiments is limited to 10.

While our experiments are not on large-scale datasets, there are existing studies that provide valuable empirical observations for large-scale multi-source training as mentioned in our Introduction section (lines 8-13, right column), including: cross-lingual model transfer for similar languages [Pires et al., 2019], pretraining with additionla high-quality images to improve overall aesthetics in image generation [Chen et al., 2024], and knowledge augmentation on subsets of data to enhance model performance on other subsets [Allen-Zhu & Li, 2024a]. They have offered relevant findings that inform our work.

We will provide a more detailed explanation of our experimental settings in the revised version.

To summarize, we sincerely thank the reviewer for the constructive comments regarding our experiments, which we believe can improve the quality of this paper.

Q1: Experiments for ARMs or EBMs

We thank the reviewer for the valuable comment.

Following the reviewer's suggestion, we have conducted supplementary simulations for ARMs according to the formulation in Section 4.2. Experimental settings and results are presented below. Generally speaking, the empirical TV errors exhibit similar trends as theoretical bounds in Theorem 4.3 regarding several key factors—the number of sources $K$, sample size $n$, and data length $D$.

In all experiments, we define a ground-truth sequential discrete distribution, enabling exact computation of the practical TV error. We fix the vocabulary size $M=2$, neural network configurations with $d_e=W=64$, $L=5$, $B=1$. Then we vary $K$ in $[1, 3, 5, 7, 10]$, $n$ in $[1000, 3000, 5000, 10000, 30000]$, and $D$ in $[10, 12, 14, 16, 18]$ to examine alignment of practical TV error with theoretical bounds. For each setting, the batch size and learning rate are selected from $\lbrace 100, 300, 500 \rbrace$ and $\lbrace 10^{-5}, 10^{-4}, 10^{-3} \rbrace$ for the minimum likelihood. Empirical TV errors are presented in the following tables:

The results show consistent trends between empirical and theoretical values with respect to $n$, $K$, and $D$, i.e., the TV error decreases as $n$ and increases with $K$ and $D$, and multi-source training generally outperforms single-source training.

Additionally, we would like to clarify the connection between our diffusion model experiments and the theoretical analysis of EBMs. As mentioned in lines 51-55 in our submission, EBMs are a general and flexible class of generative models closely connected to diffusion models. To be specific, first, the training and sampling methods in [1,2] are directly inspired by EBMs. The distinction is that EBMs parameterize the energy function, while diffusion models parameterize its gradient (the score function). Second, [3] shows that under a specific energy function formulation (Equation (5) in their paper), EBMs are equivalent to constrained diffusion models. Their experimental results (Table 1, Rows A and B) indicate that the constraint has minor impact on generative performance. Thus, our diffusion model experiments provide insight into EBMs' behavior in real-world settings to some extent.

We will provide the implementation details and simulation results for ARMs in the revised version of our paper, along with the above discussions for EBMs.

[1] Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution.

[2] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations.

[3] Salimans, T., & Ho, J. (2021). Should EBMs model the energy or the score?

Q2: Quantifying similarity for general datasets

We thank the reviewer for raising this insightful question.

In our paper, $\beta_{sim}$ is defined by induction based on our three specific model instantiations in Section 4. It is not an inherent, directly measurable property of the source distributions themselves, meaning it cannot be directly computed given general datasets.

A fundamental question underlying the reviewer's inquiry might be: How can we quantify dataset similarity in practice with theoretical guarantees? We acknowledge that there is no single method currently that provides a solution to this problem, and we are still exploring ways towards this goal.

Possible approaches might include: (1) From a practical perspective, a small proxy model can be used to estimate source distributions' interaction [4]. (2) From a theoretical perspective, several existing notions in multi-task learning and meta-learning could be adapted for this purpose, such as transformation equivalence [5], parameter distance [6], and distribution divergence [7].

[4] Xie, S. M., Pham, H., Dong, X., et al (2023). Doremi: Optimizing data mixtures speeds up language model pretraining.

[5] Ben-David, S., & Borbely, R. S. (2008). A notion of task relatedness yielding provable multiple-task learning guarantees.

[6] Balcan, M. F., Khodak, M., & Talwalkar, A. (2019). Provable guarantees for gradient-based meta-learning.

[7] Jose, S. T., & Simeone, O. (2021). An information-theoretic analysis of the impact of task similarity on meta-learning.