Bridging the gap to real-world language-grounded visual concept learning

We appreciate the valuable comments. Below we respond to individual concerns,

Q1. Ablations over the prompt

We find that the design of prompt mainly affects the granularity of the axes. For example, in CelebA-HQ dataset, simple task description without providing examples would lead to finding relatively coarse concepts such as hair and color instead of more fine-grained attributes such as hair-color, hair-length, eye color. As discovered axes are significantly different depending on the prompt, we cannot directly compare them in quantitative measures. We agree with the reviewer that investigation on prompt design might improve overall performance of our work and we leave it for future work.

Q2. Ablations on VLM choices

Following the reviewer’s suggestion, we conduct two studies on the choice of VLM:

(1) We conduct ablation study on different choices of VLMs. We consider two popular opensourced VLMs (Qwen2.5-VL and Ovfi2) that have shown high ranks in the leaderboard on reasoning benchmarks. (2) We also investigate the effect of partial observation of concept axes. While (2) is not a direct ablation study on VLMs, it can provide a controlled evaluation between the performance of VLM’s performance and representation learning quality of our framework. Since it is difficult to directly control and quantify the performance of VLMs, we instead control the quality of the output from VLMs by explicitly dropping partial axes (e.g., 10%) among the axes given by the VLMs. It is a practical scenario as it is impossible for VLM to always capture the complete axes for given scenes. To this end, we first extract concept axes per image for entire images and only 80% or 90% of axes (i.e., 10%/20% random drop) are provided to our model in the training stage. We evaluate our method on the visual concept editing task in CelebA-HQ dataset.

Below Table reveals that our method is hugely robust to both the choice of VLMs and missing axes. We conjecture that this is because even though some image-related axes are missed in each example, those axes will be eventually, repeatedly exposed across the dataset. Moreover, since our compositional anchoring loss encourages the compositionality of the concept representations, our framework might be internally trained for better compositional generalization, which improves adaptation with relatively few samples. In fact, we observed that our method is capable of generating OOD samples (Figure 5 in our paper.). These results highlight the robustness of our framework regarding the performance of VLMs, and implies that our method can scale to more complex scenarios, since VLMs do not have to always capture complete axes for given scenes.

=== [Table 1] ===

Q3. Interplay between components

We conduct ablation studies on each of the components as follows: (1) Frozen T2I decoder versus LoRA-finetuned decoder, (1) Choice of vision encoder (Dinov2 versus CLIP), (3) universal concept decoder versus shared MLP architectures.

First, finetuning the decoder with LoRA does not affect the overall performance. Large scale pretrained T2I models are known to be already learned expressive data priors on natural images, which helps faster training of the generation model. Thus the frozen T2I model does not bottleneck our framework. Replacing Dinov2 encoder with CLIP encoder leads to significant performance drop. This is because CLIP is only trained for text-alignment and thereby it’s discriminative property of representation is likely to be inferior to recent self-supervised learning methods (Dinov2).

Lastly, we replace our universal concept encoder with shared MLP architecture, which is naive version of axis-agnostic encoder. Specifically, visual feature is first represented as vector through mean pooling, and it is concatenated with axis embeddings followed by MLPs to encode concept representations. To make this encoder generally work for diverse concept axes, we shared this MLP layer for all of the axes. However, this modification causes severe performance drop. We conjecture that this is because shared MLP encodes each concept independently, possibly blocking complex interaction between visual concepts. It clearly highlights the effectiveness of our universal concept encoders.

=== [Table 2] ===

Q4. Experiments on diverse dataset

To validate our framework on larger, diverse, and unstructured data, we randomly sampled 20 ImageNet classes, spanning animals (e.g., tree frog, American black bear, sulphur butterfly, giant panda), everyday objects (e.g., padlock, grand piano, motor scooter), scenes (e.g., boathouse, water tower), and etc, yielding around 28k training images (~1.4k images per class). This is a significantly more challenging scenario as it contains diverse image-specific visual concepts and unstructured images. Across this dataset, our method successfully discovers both coarse and fine-grained across different categories. Spanning from primitive concepts like color, material, shape, it also discovers fine-grained axes such as wing color/shape, body color of butterflies and sail color, boat color, sky condition for boat images. It also captures categorical attributes, such as species, vehicle type, fruit type. Although our dataset does not include human-centered images as in CelebA-HQ dataset, our framework still robustly captures human-specific concepts such as age, gender, hair color/length. For quantitative assessment, we conduct visual concept editing on the 50 most frequent axes (please refer to response to Q2. of Reviewer 4ztc for top-50 frequent axes) and report CLIP/BLIP Scores against text-conditioned editing baselines. Manually predefined-axis based approach (LICVL [1]) becomes infeasible here, since a fixed set of axes fails to cover a wide range of concept space. Even within the same category, diverse image-specific concepts would appear, for example, humans, animals and plants may appear in motor_scooter class images, making it difficult to predefine a fixed set of concept axes representing the entire images. It highlights the effectiveness of our method, as our framework can adaptively decompose the given scene into image-related axes specified by VLMs. Below Table demonstrates superior performance of our method compared to text-conditioned editing baselines. As the discovered visual concepts become more detailed such as body color of butterflies or stem color of flowers, most of the text-conditioned editing baselines fail to perform faithful editing. Also, we find that our framework is capable of generating OOD samples such as green bears/pandas or a scooter floating above the sea. We will add qualitative results in our main paper.

Q5. Challenges (outside compute) for scaling your method to larger and more diverse natural images dataset

Yes, mapping visual concepts to verbal words is an important motivation of our work. As there could be infinitely many concept axes in real-worlds, aligning with our perception and those semantic representations are crucial and effective way to describe the complex world with a feasible number of concepts. We found that rare and too fine-grained concepts such as the exact brand name of the calculator are hard to be trained, as our encoder rarely meets those concepts and hardly get signals from compositional anchoring loss.

Reference

[1] Lee et al., language-informed visual concept learning, in ICLR 24.

We sincerely appreciate the reviewer's valuable comments. Thanks to the reviewer's thoughtful comments and feedback, additional experiments on ablation studies and diverse dataset make our work more comprehensive. We will add qualitative OOD samples and additional experimental results in our revised paper.

We appreciate the valuable comments. Below we respond to individual concerns,

Q1. Reliance on Visual Language Models (VLM)

We agree with the reviewer that VLM cannot always perceive a complete set of concept axes. Nevertheless, we believe that the use of VLM provides a significant advantage in complex scenarios, where it is almost infeasible to manually annotate diverse image-specific concepts. Although VLM would not be perfect, it can still capture meaningful and diverse visual concepts ( Please refer to our response to Q2 below.)

In addition, we conduct additional experiments to investigate the robustness of our framework to VLMs. We perform two additional experiments: We measure performance on (1) different choices of VLMs, and (2) randomly drop partial axes from total axes provided by the VLMs. For the first experiment, we consider two additional opensourced VLMs (Qwen2.5-VL and OVis2) that have shown high ranks in the leaderboard on reasoning benchmarks. For the second experiment, as it is difficult to control and quantify how much of the axes are missed by VLM, we instead investigate the performance differences when we explicitly drop partial axes (10% and 20%) from the axes provided by VLMs. To this end, we first extract concept axes per image for entire images and only 80% or 90% of axes (i.e., 10% and 20% of random drop) are provided to our model in the training stage. We evaluate our method on the visual concept editing task in CelebA-HQ dataset.

The below Table reveals that our method is hugely robust to both the choice of VLMs and missing axes. We conjecture that this is because even though some image-related axes are missed in each example, those axes will be eventually, repeatedly exposed across the dataset. Moreover, since our compositional anchoring loss encourages the compositionality of the concept representations, our framework might be internally trained for better compositional generalization. In fact, we observed that our method is capable of generating OOD samples (Figure 5 in our paper.). These results highlight the robustness of our framework regarding the performance of VLMs, and implies that our method can scale to more complex scenarios, since VLMs do not have to always capture complete axes for given scenes.

=== [Table 1] ===

Q2. Experiments on more complex dataset.

Following the suggestion of the reviewer, we validate our framework on larger, diverse, and unstructured data. We randomly sampled 20 ImageNet classes, spanning animals (e.g., tree frog, American black bear, sulphur butterfly, giant panda), everyday objects (e.g., padlock, grand piano, motor scooter), and scenes (e.g., boathouse, water tower) yielding around 28k training images (~1.4k images per class). This is a significantly more challenging scenario as it contains diverse image-specific visual concepts and unstructured images.

Across this dataset, our method successfully discovers both coarse and fine-grained across different categories. Spanning from primitive concepts like color, material, shape, it also discovers fine-grained axes such as wing color/shape, body color of butterflies and sail color, boat color, sky condition for boat images. It also captures categorical attributes, such as species, vehicle type, fruit type. Although our dataset does not include human-centered images as in CelebA-HQ dataset, our framework still robustly captures human-specific concepts such as age, gender, hair color/length.

For quantitative assessment, we conduct visual concept editing on the 50 most frequent axes (We provide top-50 frequent axes at the end of the response) and report CLIP/BLIP Scores against text-conditioned editing baselines. Manually predefined-axis based approach (LICVL [1]) becomes infeasible here, since a fixed set of axes fails to cover a wide range of concept space. Even within the same category, diverse image-specific concepts would appear, for example, humans, animals and plants may appear in motor_scooter class images, making it difficult to predefine a fixed set of concept axes representing the entire images. It highlights the effectiveness of our method, as our framework can adaptively decompose the given scene into image-related axes specified by VLMs.

Below Table demonstrates superior performance of our method compared to text-conditioned editing baselines. As the discovered visual concepts become more detailed such as body color of butterflies or stem color of flowers, most of the text-conditioned editing baselines fail to perform faithful editing. Also, we find that our framework is capable of generating OOD samples such as green bears/pandas or a scooter floating above the sea. We will add qualitative results in our main paper.

Q3. Try to modify the current architecture or clarify that it is not merely a building-block work.

We would like to first clarify that our main scope is not designing a specific model architecture to improve the performance. Instead, our main contribution is to build a framework that can adaptively capture image-specific visual concepts in real-world scenes. To this end, we focus on designing our framework to address such challenges, while leaving other components general for further applicability.

Additionally, to justify the choice of each component, we conduct ablation studies as follows: (1) Frozen T2I decoder versus LoRA-finetuned decoder, (1) Choice of vision encoder (Dinov2 versus CLIP), (3) universal concept decoder versus shared MLP architectures.

First, finetuning the decoder with LoRA does not affect the overall performance. Large scale pretrained T2I models are known to be already learned expressive data priors on natural images, which helps faster training of the generation model. Thus the frozen T2I model does not bottleneck our framework. Replacing Dinov2 encoder with CLIP encoder leads to significant performance drop. This is because CLIP is only trained for text-alignment and thereby it’s discriminative property of representation is likely to be inferior to recent self-supervised learning methods (Dinov2).

Lastly, we replace our universal concept encoder with shared MLP architecture, which is naive version of axis-agnostic encoder. Specifically, the visual feature is first represented as vector through mean pooling, and it is concatenated with axis embeddings followed by MLPs to encode concept representations. To make this encoder generally work for diverse concept axes, we shared this MLP layer for all of the axes. However, this modification causes severe performance drop. We conjecture that this is because shared MLP encodes each concept independently, possibly blocking complex interaction between visual concepts. It clearly highlights the effectiveness of our universal concept encoders.

=== [Table 2] ===

Reference

[1] language-informed visual concept learning, in ICLR 24.

Top-50 axes discovered by our framework :

[subject_type, background, color, material, shape, species, size, type, vehicle_type, environment, position, cap_color, gender, age, object_type, hair_color, hair_length, lighting, accessory, location, water_body, expression, clothing, activity, sky_condition, eye_color, body_color, windows, wing_shape, fur_texture, stem_color, surroundings, sail_color, seat_color, wing_color, wing_pattern, fruit_type, color_pattern, surface_texture, limbs, flower_color, style, design, wall_color, texture, brand, container_count, flooring, cap_texture, boat_color]

We sincerely appreciate the reviewer's detailed feedback and thoughtful comments. Thanks to the reviewer's valuable suggestions, our work has became more comprehensive. As promised, we will include additional visual results and provide a more in-depth discussion of the VLM's perceptual capabilities in our revised paper.

We appreciate the valuable comments. Below we respond to individual concerns,

Q1. Dependence on a single VLM for axis generation.

We appreciate the valuable feedback and suggestions. We agree with the reviewer that the performance of VLM would largely affect our framework. While incorporating multiple VLMs or employing fixed reference taxonomies would be an interesting and effective idea, we found that it is not trivial for extend and formulate the method during the rebuttal period. Thereby we leave it for our important future work.

Instead, we provide a clarification that our method is quite robust to the performance of VLMs. We perform two additional experiments: We measure performance on (1) different choices of VLMs, and (2) randomly drop partial axes during training. For the first experiment, we consider two additional opensourced VLMs (Qwen2.5-VL and OVis2) that have shown high ranks in the leaderboard on reasoning benchmarks. For the second experiment, as it is difficult to control and quantify how much of the axes are missed by VLM, we instead investigate the quality of axes provided by VLMs. We explicitly drop partial axes (10% and 20%) from the axes provided by VLMs to simulate imperfect prediction of VLMs. To this end, we first extract concept axes per image for entire images and only 80% or 90% of axes (i.e., 10%/20% random drop) are provided to our model in the training stage. We evaluate our method on the visual concept editing task in CelebA-HQ dataset.

The below Table reveals that our method is hugely robust to both the choice of VLMs and missing axes. We conjecture that this is because even though some image-related axes are missed in each example, those axes will be eventually, repeatedly exposed across the dataset. Moreover, since our compositional anchoring loss encourages the compositionality of the concept representations, our framework might be internally trained for better compositional generalization. In fact, we observed that our method is capable of generating OOD samples (Figure 5 in our paper.). These results highlight the robustness of our framework regarding the performance of VLMs, and implies that our method can scale to more complex scenarios, since VLMs do not have to always capture complete axes for given scenes.

Q2. Assumption of axis independence

We would like to clarify that we do not impose statistical independence between the axes. The only constraint imposed by our framework is compositionality of visual concept axes. Therefore, in our experimental result, our method often disentangle highly correlated factors such as breed of the dog and folding of the ear shape and generates OOD samples (Figure 5 of our main paper). However, we agree with the reviewer that handling the correlation between semantic axes is indeed an important problem and leave it to important direction for future work.

Q3. Evaluation scope is narrow

We believe our evaluation protocol on image editing is already implicitly measuring the compositionality of the learned representations. As we compose two or more sets of concept representations and measure alignment between resulting composite and those composite attributes, it can be interpreted as measuring the compositionality of the representation. We appreciate the valuable comment from the reviewer and will find the proper evaluation protocol for measuring compositionality and update it in our main paper.

Q4. Frozen T2I model as a bottleneck.

We agree that fine-tuning the T2I backbone, e.g. via LoRA, would boost overall image fidelity, but we would like to emphasize that even with a fully frozen diffusion model, our learned embeddings already achieve high-quality, semantically accurate edits. This implies that our framework does not require to train huge generative models every time or store task-specific parameters (e.g., LoRA) for T2I model, which eases the training process of our framework. To support our claim, we finetune our T2I model with LoRA, and report CLIP/BLIP Score for visual concept editing tasks. We find that the effect of finetuning LoRA seems to be negligible in terms of concept learning.

Nevertheless, we agree with the reviewer that joint training of the T2I model would have clear benefits in image generation or faster adaptation to unseen samples and tasks. We appreciate the valuable comment and leave it for the future work.

=== [Table 1] ===

Dear Reviewer xMmD,

We would like to sincerely thank the reviewer again for their valuable comments. We kindly follow up to inquire whether our response has sufficiently addressed the reviewer’s questions. Should the reviewer have any further queries or concerns, we would be happy to discuss.

We appreciate the valuable comments. Below we respond to individual concerns,

Q1. The paper doesn't offer a fundamentally new approach or a fundamental, principles-driven or theory driven approach.

We would like to first clarify the contribution of our work. It is generally difficult for machine learning algorithms to capture underlying visual concepts in real-world scenarios, since each image consists of very diverse, image-specific axes. While it is infeasible to disentangle all of those factors without supervision, a recent breakthrough has been made by incorporating a vision language model to ground each concept axis with textual description [1]. However, this method still requires manual definition of concept axes and thereby limits its application to complex real-world scenes having infinitely many axes.

In this work, we address such challenges by designing our framework to adaptively perceive image-related axes aided by VLMs. We believe the problem setting and overall formulation itself is an important contribution to the field as it provides a first step towards visual concept learning in a general, real-world scenes. Moreover, we also propose novel objective (compositional anchoring loss) to disentangle each representation along image-specific axes, and propose a simple yet effective architecture (Universal Concept Encoder) to encode a varying number of concept representations. Our ablation study (Table 3 in our main paper) supports that novel objectives is essential component of our framework. Moreover, we provide additional experiments on more diverse and complex natural image datasets, which highlights the effectiveness and scalability of our framework (Please see our response to Q2.)

Q2. Experiments on more diverse and unstructured datasets

To validate our framework on larger, diverse, and unstructured data, we randomly sampled 20 ImageNet classes, spanning animals (e.g., tree frog, American black bear, sulphur butterfly, giant panda), everyday objects (e.g., padlock, grand piano, motor scooter), and scenes (e.g., boathouse, water tower) yielding around 28k training images (~1.4k images per class). This is a significantly more challenging scenario as it contains diverse image-specific visual concepts and unstructured images.

Across this dataset, our method successfully discovers both coarse and fine-grained across different categories. Spanning from primitive concepts like color, material, shape, it also discovers fine-grained axes such as wing color/shape, body color of butterflies and sail color, boat color, sky condition for boat images. It also captures categorical attributes, such as species, vehicle type, fruit type. Although our dataset does not include human-centered images as in CelebA-HQ dataset, our framework still robustly captures human-specific concepts such as age, gender, hair color/length.

For quantitative assessment, we conduct visual concept editing on the 50 most frequent axes (We provide top-50 frequent axes at the end of the response) and report CLIP/BLIP Scores against text-conditioned editing baselines. Manually predefined-axis based approach (LICVL [1]) becomes infeasible here, since a fixed set of axes fails to cover a wide range of concept space. Even within the same category, diverse image-specific concepts would appear, for example, humans, animals and plants may appear in motor_scooter class images, making it difficult to predefine a fixed set of concept axes representing the entire images. It highlights the effectiveness of our method, as our framework can adaptively decompose the given scene into image-related axes specified by VLMs.

Below Table demonstrates superior performance of our method over text-conditioned editing baselines. As the discovered visual concepts become more detailed such as body color of butterflies or stem color of flowers, most of the text-conditioned editing baselines fail to perform faithful editing. Also, we find that our framework is capable of generating OOD samples such as green bears/pandas or a scooter floating above the sea. We will add extensive qualitative results in our main paper.

=== [Table 1] ===

Q3. Robustness analysis on on VLMs

To analyze the robustness of our framework to VLM, we conduct two additional experiments: We measure performance on (1) different choices of VLMs, and (2) randomly drop partial axes from total axes provided by the VLMs.

For the first experiment, we consider two additional opensourced VLMs (Qwen2.5-VL and Ovis2) that have shown high ranks in the leaderboard on reasoning benchmarks. For the second experiment, as it is difficult to control and quantify how much of the axes are missed by VLM, we instead investigate the performance differences when we explicitly drop partial axes (10% and 20%) from the axes provided by VLMs. To this end, we first extract concept axes per image for entire images and only 80% or 90% of axes (i.e., 10%/20% random drop) are provided to our model in the training stage. We evaluate our method on the visual concept editing task in CelebA-HQ dataset.

The below Table reveals that our method is hugely robust to both the choice of VLMs and missing axes. We conjecture that this is because even though some image-related axes are missed in each example, those axes will be eventually, repeatedly exposed across the dataset. Moreover, since our compositional anchoring loss encourages the compositionality of the concept representations, our framework might be internally trained for better compositional generalization. In fact, we observed that our method is capable of generating OOD samples (Figure 5 in our paper.)

Q4. More details on limitations, linked up to analyses of specific kinds of errors that the system makes, would be helpful.

As the reviewer pointed, dependency on VLM can cause some errors in practice. For we found that VLM often wrongly annotates attributes of the small objects, or often confuses the attribute of close objects, which possibly affects the overall performance. We will add these specific failure cases in the Appendix.

Reference

[1], Lee et al., Language-informed visual concept learning, in ICLR 24.

Top-50 axes discovered by our framework:

[subject_type, background, color, material, shape, species, size, type, vehicle_type, environment, position, cap_color, gender, age, object_type, hair_color, hair_length, lighting, accessory, location, water_body, expression, clothing, activity, sky_condition, eye_color, body_color, windows, wing_shape, fur_texture, stem_color, surroundings, sail_color, seat_color, wing_color, wing_pattern, fruit_type, color_pattern, surface_texture, limbs, flower_color, style, design, wall_color, texture, brand, container_count, flooring, cap_texture, boat_color]

Dear Reviewer vNjW,

We would like to sincerely thank the reviewer again for their valuable comments. We hope that the additional experiments on a more diverse and unstructured dataset (a subset of ImageNet), along with our robustness analysis on VLMs, have addressed the reviewer’s concerns.

We kindly follow up to inquire whether our response has sufficiently addressed the reviewer’s questions. Should the reviewer have any further queries or concerns, we would be happy to discuss.