Does Data Scaling Lead to Visual Compositional Generalization?

We appreciate your thorough review. We will incorporate the feedback into the updated manuscript.

They do not test a large variety of models when training from scratch (only ResNet-50 is reported), but they claim to have used other baselines in the text. It would be good to include some of these results, even if they are not used in all the conditions. One way to justify not including all test conditions could be to start with simple ones and, should they fare worse than ResNet-50, exclude them from subsequent evaluations.

Thank you for this suggestion. We agree that including results from additional models would strengthen the completeness of our empirical analysis. We include some of the results of vision transformers and compare them with ResNet50 here by varying the % of combinations for CMNIST (since the model achieves comparable results with 80% of combinations); additionally, we highlight that even under increased number of training points, vision transformer still underperforms (all experiments assumed 60,000 datapoints, except for the last row):

We attempted to be as charitable to ViT architectures as possible: we varied the patch size (8 and 16), ViT depth (4, 6, 8), width of each layer (384, 512), number of heads (8, 12), MLP width (384, 512), and learning rate ($1e{-}5$, $1e{-}4$, $1e{-}3$). In the table above, we report the best-performing configuration for each setting. All the models were able to achieve 99.7% accuracy on the ID data, but ViT typically underperformed on the testing data.

My main suggestion is regarding the use of the term compositionality.

Thank you for raising this point. We agree that the term “compositionality” may not have been the most precise description for the notion we are exploring. Our motivation for using it stemmed from the natural emergence of linear factorization in the from-scratch setting with increased data diversity. However, we acknowledge that compositional structure does not have to be linear. Instead, we propose to use the term “linear factorization” as a specific instantiation of compositionality. We will clarify this in the text.

In fact, the authors’ own findings contradict the claim that these are compositional representations, since they fail to generalize systematically to unseen combinations. [...] Thus the representations must either be unstable or such a representation is not enough to ensure compositionality.

Our goal is not to claim that the representations are fully compositional, but rather to study to which extent they are compositional, instantiated in our case as exhibiting a linear factorization. The failure to generalize to unseen combinations highlights the limitations of this structure in the models we study. If our writing gave the impression that we claim full compositionality, we’d be happy to clarify--please do let us know if a particular phrasing seemed misleading.

What happens if in the analysis in Section 4.2, instead of training the decoder and determining PCA directions with both training and testing combinations, the authors only use the training ones and project the test combinations with the resulting model/PCA? Do the test combinations clearly separate from the training ones? Is the nice lattice structure observed in the projections disrupted?

The displayed points in Section 4.2 are from the OOD data, i.e., unseen combinations. Specifically, we sample a 2×2 grid of combinations that share concept values (i.e., (i, j), (i, k), (l, j), (l, k)), all from the test set. We will clarify this procedure in text.

Regarding the suggestion to perform PCA only on the training data and project the test points using the resulting model: in this case, the lattice structure is not preserved. Intuitively, this is because combinations of concept values can span up to a $2n - 1$-dimensional space. Even if the training data admits a linear factorization, projecting unseen combinations using a PCA fit on the training data alone does not guarantee that the structure will be preserved since the model is trained to classify concepts. This is the reason why we used a smaller set of combinations for visualization purposes.

Thank you for your thorough review. We will incorporate these suggestions in the final version of the manuscript.

"large pretrained models" can be very misleading

We agree about emphasizing the vision aspect. However, our models are large within the vision domain—for example, DINO ViT-L contains 300M parameters. Would it be clearer if we added a footnote explaining our criterion for "large-scale models" in the vision context?

L90 is not very clear what it means

What we meant by this is that prior work usually considers fixed compositionality settings where the scale of compositions is fixed.

It is challenging to see the relation between this work and others mentioned in related work, especially in the last four paragraphs.

We aimed to cover prior work relevant to (compositional) generalization and vision, and highlight challenges. In brief:

• Compositionality and vision models: We discuss existing limitations in vision models, e.g. under spurious correlations and how it affects generalization.
• Scaling and emergent abilities: Prior work shows scaling can improve performance in (compositional) generalization tasks. We build on this by systematically varying data and model scale across multiple axes.
• Improving compositionality: While others introduce architectural or loss-based modifications, we evaluate whether modern scaling practices alone suffice for compositional generalization.
• Representation learning: Our approach focuses on the structure of learned representations--specifically, how they support generalization to unseen compositions.

We will revise the text to more clearly explain how our work relates to prior work.

"ID data" is not explained.

We agree that this terminology is confusing without an explicit mention of it. By ID data we are referring to the data sampled from the training region of combinations defined by the $(n, k)$ framework we introduced.

L186: it is unclear why authors introduce more variants to the compositions while claiming that the paper limits its compositionality as 2 for simplicity. & Is there any reference to follow and justify whether the simplification is reasonable and effective for its extension?

We wanted to emphasize that the total dataset size is not restricted by the two concepts we study. While the specific pair of concepts we focus on is limited in size, other concepts in the data can vary freely. This allows us to study a more realistic setting for compositionality. For example, in the real world, when we restrict the pair (color, animal), we know that blue and green pandas aren't common in datasets. However, the pairs that are present (e.g., white pandas) can still vary along many other dimensions, such as size, pose, location, and other attributes.

Additionally, we focus on the two-concept case because we believe it is the simplest compositional setup that still presents challenges for vision models. Given that the total number of concepts is greater than 2, applying further restrictions would only make the task harder. Therefore, we view the two-concept focus as a reasonable simplification.

Zerroug et al. (2022), Zhou et al. (2023), and Stone et al. (2017) are essential related works not discussed. & It is unclear what the difference is in the empirical findings from existing work

Thank you for these references. We briefly summarize how these works relate to ours:

• Zerroug et al. focus on visual reasoning and data-efficiency, combining representation learning and reasoning over them with repeatedly sampled (i.e. the train and test set has overlap), or procedurally-generated tasks (visual primitives aren't overlapping between training and testing). In contrast, our work centers on classification tasks that rely on strong vision representations and examines generalization when seen concept values at test time are combined in unseen ways.
• Zhou et al. study data compositionality in NLP using primitive-level data augmentation to enhance generalization. While we find a similar conclusion that diverse training data is beneficial, our focus on the vision domain introduces challenges not encountered in language tasks (e.g. making primitive-level augmentations is simple in language, but not in vision; and scaling ID data quantity isn't sufficient to improve generalization).
• Stone et al. modify architectures and losses with object masks to promote compositionality, which relies on injecting priors and using dataset-specific annotations. Our approach, however, investigates compositional generalization from a data-scaling perspective.

We believe our work isolates a specific and underexplored aspect of compositional generalization in vision. Our work, to our knowledge, is the first to consider the scale of compositions across multiple dimensions: types of concepts, number of concept values, number of datapoints used in training, types of datasets, and densities of compositions in a controlled manner.

Thank you for your thorough review. We will clarify the raised points in the revised manuscript.

L223: What is the significance of reporting average accuracy per epoch? Does it effectively reflect the model's final compositional generalization capability?

Thank you for pointing this out. We should have stated this more clearly. The metric reported in Line 223 corresponds to the best-performing epoch (i.e., oracle model selection), not an average over epochs. Specifically, we track the average accuracy across both concepts at each epoch and report the highest value observed during training. This provides an estimate of the upper bound on achievable performance under ideal model selection. We chose this evaluation to emphasize that even with perfect model selection, the performance gap between ID and OOD is present.

The use of two independent classification heads to predict each concept in a composition is debatable (Line 230)

If we understood your question correctly, by similarity-based, do you mean a CLIP-like objective? Our current design mirrors the supervision used in CLIP and standard classification settings, up to norm-rescaling of the features and weight vectors. The main difference is that our setup assumes a closed-vocabulary setting, and we believe that using two independent classifiers is appropriate since the concepts are independent.

We would be happy to hear more about alternative formulations or suggestions on how to frame this classification problem!

The authors claim to have evaluated three probe architectures (Line 255), but these are not mentioned in the experimental results.

Our main goal was to evaluate the pre-trained models' representations as charitably as possible. We did evaluate multiple probe architectures, but our main text only reported the best-performing configuration.

In response to this comment, we now include the average testing accuracies on both concepts across all datasets across 4 settings of data diversity (% of combinations in training), comparing linear probes and MLPs. MLPs most often performs better, but not universally across the datasets. The first number is the linear probe and the second is the MLP ([512, 512]):

Additionally, the relevance of evaluating architectures with only minor differences (e.g., one hidden layer or activation function) remains debatable.

While the architectural variations may appear small, we found that going beyond two hidden layers didn't result in substantial increase in performance. Below, we provide comparisons using the FSprites dataset (again for 4 diversity settings); we used these results as justification for focusing on simpler configurations in this work.

Does the number of training samples remain constant while increasing training data diversity? (Sec. 4.1)

Yes, we do control for this factor, although we acknowledge that it was not made sufficiently clear in the text. All experiments in Section 4.1 (except for Figure 4) use a fixed dataset size of 60,000 samples where available; for Shapes3D and CMNIST we used 30,000 samples.

Does the authors consider conducting experiments on real-world datasets?

We considered real-world datasets with known generative factors, but they are currently limited in compositional complexity. Moreover, real-world datasets offer sparse coverage of concept values, making controlled compositional generalization experiments intractable. We are open to suggestions for real-world datasets that offer controllability.

Finally, although our datasets are synthetic, Section 5 evaluates pre-trained models used in real-world practice and tests how pre-trained vision models cope in compositional settings.

Line 318, what does performance specifically refer to?

By performance, we refer to the mean classification accuracy over the considered concept values.

Thank you for your thorough review. We will add clarifications to the updated manuscript.

Clarify of definitions

In our terminology, “concepts” refer to interpretable properties of the data (e.g., colour, shape, size), while “features” refer to the latent representations extracted by the model.

Investigating non-uniform distributions

The goal in our setup was to ensure uniform coverage over observed combinations to avoid introducing imbalances across concept values. While certainly interesting, we are careful in complexifying the data setting in a way that makes the problem harder.

Evaluating language models and VLM.

Extending to language or vision-language models is challenging. Autoregressive models lack comparable representations. For non-autoregressive ones like CLIP, we probed the vision encoder; probing the text encoder is harder, as it requires generating many prompts to describe concept values. Nonetheless, we consider this an important direction for future work.

Transformer-based vision models.

Due to space constraints in this response, we couldn’t include the results for the vision transformer here. Another reviewer also expressed interest in the ViT’s performance--please refer to the table in the response to reviewer TQRD.

Related study by Dorrell et al. (2024).

Thank you for pointing us to this excellent reference--we will be sure to discuss it in more detail. In short, this work, as well as the work it builds on (Whittington et al., 2023), characterises when the minimal-energy non-negative solution is modular, i.e when each neuron responds to a single task factor. We believe both this framework and ours can be viewed in a similar light. The key distinction lies in the goal: while their focus is on the structure of solutions over the observed data, our work studies whether representations can be predicted from their constituent concept values under novel combinations in the test set, and how data factors influence the structure of representations.

Simplicity bias and spurious correlations, underrepresented concepts

Our reasoning is based on prior observations in the debiasing literature. When training data exhibits underspecification--i.e., when multiple hypotheses are consistent with the data--a model may favour simpler hypotheses. In such cases, it may learn to generalize to majority groups and memorize the underrepresented groups [a]. In compositional generalization settings, this happens when some combinations are missing entirely (as in OOD splits), and models default to entangled solutions instead of learning disentangling concepts in their representation. Concepts, especially in the real world, are highly co-varying in some combinations and not in others, even if the observed concept pairs are not entangled. Because of this sampling bias, models may exploit such spurious correlations.

[a] Sagawa, Shiori, et al. "An investigation of why overparameterization exacerbates spurious correlations." ICLR 2020.

In Section 3.1, formula for unique images

In Section 3.1, we were referring specifically to the number of training combinations observed in the $(n,k)$ framework, but we will clarify this distinction in the revised text.

Figure 3: boundary value of $k$

If we understood your question correctly: The boundary value of $k$ corresponds to $n-1$, that is, training on all but one combination per concept value.

Figure 3: For maximum target classes, does the improvement hold for lower $n$?

Yes, and we believe Figure 10 in the appendix demonstrates this: for fixed $k$, increasing $n$ lowers generalization performance; conversely, for fixed $n$, increasing $k$ improves it.

Figure 4: What is the $y$-axis measuring? Also, why does FSprites exhibit lower values in general?

The $y$-axis in Figure 4 shows OOD accuracy. FSprites performs worse than other datasets, likely due to its more complex and visually similar shapes, which are harder to disentangle. Also, orientation may not align well with shape features, encouraging memorisation of joint shape-orientation co-variations instead of separate concept learning.

Figure 7: Does the gap indicate that the retrained model demonstrates better compositional generalization performance than a from-scratch model, or does it reflect the limitations of the pretrained model?

It indicates both. The retrained model benefits from better compositional generalization due to exposure to more diverse or better-structured training data. However, the presence of a gap also reflects the limitations of the pretrained models to generalize compositionally.

Linear Probing in Section 5: What is the $k$ value used in the linear probing analysis? How does it compare to the threshold required for compositional representations to emerge?

We varied $k$ from 2 to $n-1$, showing only aggregated results across datasets. While the exact threshold varies, $k=2$ is theoretically the minimum needed per dataset.