An Image is Worth Multiple Words: Learning Object Level Concepts using Multi-Concepts Prompts Learning

We are pleased to note that the reviewer recognises the novel task of learning multiple concepts within multi-object scenes under image-level supervision, which has not been previously explored and the proposed Multi-Concept Prompt Learning (MCPL) framework in conjunction with proposed regularisation techniques effectively tackles such challenge. Specifically, we are gratified that the reviewer:

• Confirmed our observations from the motivational study that existing method has limitations in object-level learning and editing without manual intervention;
• Agreed that our extensive quantitative analysis and evaluation show that the proposed approaches effectively enhanced precision in object-level concept learning, synthesis, editing and quantification.
• Acknowledged our proposed regularisation terms effective techniques contribute to learning object-level information under image-level supervision2;

We are also thankful for the reviewer’s identification of certain weaknesses and questions in our work. We have taken these points into careful consideration, performed extensive additional experiments and provided clarifications as follows.

---

… the demonstrations to prove the composing ability are insufficient..upon the interaction between the two concepts is changed.

… However, I see all provided examples contain only two distinct concepts so I doubt the generalization of this method.

• We thank the reviewer for raising this important experiment. Both ‘composing capability’ and ‘learning more than two concepts' are covered in Figure 11-12 of Section A.2, which involves learning three or more concepts from a single image and composing a subset in a new scene.
• Both tasks involve composing with strings differ from training, our method generally learns targeted concepts and showed decent composing capability, outperforming Textural Inversion and DreamBooth, and was closer to Break-A-Scene, which represents the state-of-the-art and a performance upper-bound in masked-based multi-concept learning. Unlike BAS, our approach neither uses segmentation masks as input nor updates model parameters.

---

… The detail and comparison for a setting called “MCPL-diverse” are also unavailable.

• We are thankful for the reviewer’s question and interest in “MCPL-diverse” — a powerful but under-explored variant of our method in the original submission. We perform new experiments and provide clarifications as follows.
• MCPL-diverse v.s. MCPL-one with segmentation masks: we add new qualitative results in Figures 13 and 14 in Section A.3 for the task of learning per-image different concepts. In both examples, as MCPL-diverse is specially designed for such tasks, it consistently outperforms MCPL-one.
• MCPL-diverse has also been found powerful in more competitive tasks of learning more than three concepts from a single image, following R2(RxXV)’s suggestion on comparing competitive baseline Break-A-Scene, which represents the state-of-the-art and a performance upper-bound in masked-based multi-concept learning. Unlike BAS, MCPL neither uses segmentation masks as input nor updates model parameters.
• While R2 noted it's somewhat unfair to compare directly with BAS due to its segmentation inputs, we add Figure 12 of Section A.2 for this ambitious comparison. In Figure 12, we evaluated both MCPL-one and MCPL-diverse, the latter employing random crops akin to BAS’s ’union sampling’ strategy. The random crops strategy leads to each image containing different multiple concepts. In such case, we found MCPL-diverse is a perfect fit, outperforming MCPL-one and comparable to the ambitious BAS baseline.

---

... not to explain the meaning of the first setting called textural Inversion applied to unmasked multi-concept images and the subsequent experiments don’t contain the effects of the first two settings.

• We thank the reviewer for identifying this clarification issue! The first setting called ‘textural Inversion applied to unmasked multi-concept images was a very initial baseline we explored. It was designed to assess the learning of combined concepts (i.e. encoding two concepts in one prompt embedding). This baseline was evaluated by the t-SNE visualisation and because it was less comparable to other experiments hence has been presented in Figures 26 and 27 in Appendix A.5.
• We apologise for the above confusion and revised the entire experiments section to further clarify all experiment setups.

---

Some writing mistakes exist in the paper. In the top right of Figure 3, the labelling of “on” and “under” should be reversed. On page 6, “The” in “Therefore The contrastive loss” should be lowercase in line 8 of the paragraph before “Implementation details”.

• We thank the reviewer for the thoughtful reviews and for pointing out those writing mistakes, we updated Figure 3 and corrected the writing mistake in the revised submission.

We are pleased to note that the reviewer recognises the challenge of learning multiple individual concepts simultaneously from a single image and finds our work to be well-motivated with detailed results. Specifically, we are gratified that the reviewer:

• Confirmed our proposed three regularisation techniques, Attention Masking, Prompts Contrastive Loss, and Bind Adjectives, are effective and would benefit relevant works such as textural inversion;
• Agreed that our extensive experiments show that the proposed MCPL effectively creates attention masks for relevant concepts.
• Acknowledged the description of the method and experiment section is polished and easy to understand;

We are also thankful for the reviewer’s question and interest in “MCPL-diverse” — a powerful but under-explored variation of our method in the original submission. We perform new experiments and provide clarifications as follows

… analysis between the MCPL-diverse mentioned in the preliminary study and the full version of MCPL-one. The authors could provide visualization of generated natural concepts from MCPL-diverse as well as corresponding generated segmentation masks to support the observation.

• MCPL-diverse v.s. MCPL-one with segmentation masks: we add new qualitative results in Figures 13 and 14 in Section A.3 for the task of learning per-image different concepts. In both examples, as MCPL-diverse is specially designed for such tasks, it consistently outperforms MCPL-one.
• MCPL-diverse has also been found powerful in more competitive tasks of learning more than three concepts from a single image, following R2(RxXV)’s suggestion on comparing competitive baseline Break-A-Scene, which represents the state-of-the-art and a performance upper-bound in masked-based multi-concept learning. Unlike BAS, our MCPL neither uses segmentation masks as input nor updates model parameters.
• While R2 noted it's somewhat unfair to compare directly with BAS due to its relying on segmentation inputs, we add Figure 12 of Section A.2 for this ambitious comparison. In the right example of Figure 12, we evaluated both MCPL-one and MCPL-diverse, the latter employing random crops akin to BAS’s ’union sampling’ strategy. The random crops strategy leads to each image containing different multiple concepts. In such case, we found MCPL-diverse is a perfect fit, outperforming MCPL-one and comparable to the ambitious BAS baseline.

We are pleased to note that the reviewer recognises the challenge of multi-concept extractions from a single image and finds our work to be well-motivated with detailed results. Specifically, we are gratified that the reviewer:

• Confirmed our method leverages novel regularisation losses without relying on any groundtruth object segmentation;
• Agreed that our paper is clearly structured and easy to follow. The motivational study introduces the problem and the current limitation in a systematic way.
• Acknowledged that experiments are well-designed and analysis are comprehensive;

We are also thankful for the reviewer’s identification of certain weaknesses and questions in our work. We have taken these points into careful consideration, performed extensive additional experiments and provided clarifications as follows.

---

W.1 - “Some recent works (such as Break-A-Scene) on similar tasks could also be mentioned in the related work section. On the other hand, though it would be a bit unfair to directly compare with Break-A-Scene (due to its given segmentation inputs), it could still be interesting to treat its performance as an upper bound and comment on how a good segmentation mask would affect the learned concepts.”

• We wholeheartedly concur and are thankful for the reviewer highlighting this significant recent reference, which had previously been overlooked in our work. In response, we have conducted extensive experiments to incorporate Break-A-Scene into all our quantitative analyses and visual comparisons. This inclusion enriches our study and provides a more comprehensive understanding of our method in the context of current advancements
• Specifically, we have updated Figures 5, 7, 8, and 16-23 to compare against Break-A-Scene as a performance upper bound in our 16 concepts experiments. This expansion includes an additional 160 single GPU runs, approximately 2 million pairwise similarities, and extensive human-in-the-loop pre- and post-processing for integration with Break-A-Scene, as detailed in Section A.1. Moreover, Figures 11-12 in Section A.2 provide a visual comparison with Break-A-Scene as well as other baselines.
• It is worth noting BAS requires segmentation masks as input and employs separate segmentation models to produce masked objects, hence integration BAS involves extensive human-in-the-loop pre- and post-processing efforts, as detailed in Section A.1. In contrast, our method is mask-free at learning and employs its own AttnMask to generate masked objects.
• The following tables highlight new results in Figure 8, our fully regularised method (MCPL-one+CL+Mask) shows competitive performance against BAS on natural images. For the out-of-domain (OOD) medical dataset, BAS leads in the DINOv1 space, but we're comparable in others. This is due to our less precise object masks compared to BAS's human-assisted MedSAM segmentation, shown in Figures 5 and 6.

Object-level fidelity learned concepts referencing to "ground truth" (natural images)

Object-level fidelity learned concepts referencing to "ground truth" (medical images)

• The new t-SNE results in Figure 7 demonstrate that the learned embeddings from both the mask-based ’ground truth’ and BAS show less disentanglement compared to ours, attributable to their lack of a specific disentanglement objective, such as the PromptCL loss in MCPL.
• We've reflected on all the mentioned points in our revision, including ensuring proper citation of Break-A-Scene throughout the paper.

W.3 - “It would be better to discuss the limitations of current work in the last section and point out some future improvement directions.”

• We appreciate the reviewer's valuable suggestion! We've included a discussion on limitations and future work in the 'Limitation and Conclusion' section, a copy of which is provided below for your convenience in reviewing.
• We found our method to suffer from the following limitations:
  • (1) Imperfect Masking: Our reliance on image-level text descriptions, instead of segmentation masks, grants flexibility in exploring unknown concepts but results in less precise object boundary optimization. Future research could use our AttnMask as an input prompt to segmentation models for mask refinement.
  • (2) Composition Capability: MCPL’s composition strength is weaker than BAS, as MCPL doesn’t update model parameters, unlike BAS. Integrating MCPL with weight optimization methods like BAS or DreamBooth may enhance performance, albeit at higher computational costs, which is a potential direction for future work.
  • (3) Evaluation Metrics: Current quantification methods in this field (e.g. TI, DB, CD, BAS, and P2P), predominantly rely on prompt/embedding similarity due to the absence of more effective quantification mechanisms without known ground truth. This indicates a need for developing better evaluation metrics in future research.
  • (4) Our method relies on adjectives serving as textual descriptors (e.g., colour) to differentiate between multiple concepts. While human-machine interaction using purely linguistic descriptions is generally preferred, challenges arise when two concepts are very similar and lack distinct visual cues in the image. In such cases, our method may struggle, and Break-A-Scene currently offers the best solution.

---

Q.2 - I wonder if there are any examples of cases when the number of objects in the image is mismatched with the number of learnable concepts

• We showcase examples in Figure 11-12 of Section A.2, which involves: 1) learning three concepts from a single image and composing a subset in a new scene. 2) more challenging, entails learning six concepts from one image and composing 1 to 3 concepts in a novel scene.
• In all “mismatched tasks”, our method generally learns targeted concepts and showed decent composing capability, outperforming Textural Inversion and DreamBooth, and was closer to BAS.
• We conclude the 'Bind Adjectives' regularisation directs our model to the correct location. This is evidenced by the performance diminishes when adjectives are removed, as seen in the right example of Figure 12.

---

Q.1 - It seems that all learned concepts (nouns) are associated with a pre-defined adjective description in the prompt (such as “a green *”). Are these adjectives playing roles in disentangling the concepts? What if the initial prompt is provided in the form “A * and a @”?

• We add Figure 15 in section A.4 for an ablation study on the impact of adjective words. Adjective words are crucial in linking each prompt to the correct region; without them, the model may struggle for regional guidance, akin to the role of masks in Break-A-Scene.
• Figure 38 highlights how each regularisation technique enhances object-level prompt-concept correlation in MCPL. The left example shows reduced accuracy in attention and masks without Bind Adjectives.
• The aforementioned Figure 12-right results reinforce the significance of adjectives in maintaining model performance.

---

_W.2 - More implementation details could be added, particularly on prompt initialization. It seems a bit unclear how to initialize all learnable embeddings by the same word, “photo” in a random manner. _

• It’s a writing mistake, thank you for pointing it out! To clarify, all learnable embeddings are initialised using the encoding of pseudowords like ‘*’ or '@', rather than random initialization. We intended to convey that 'during training, sentences/phrases are constructed from randomly selected text templates y derived from CLIP ImageNet'. This has been corrected in the revised manuscript."

---

W.2 - Besides, one may be curious about how the number of prompts is determined, especially for MCPL-all.

• For MCPL-all, we learn all prompts/words in the sentence.

W.2 - However, there are very little quantitative results, and it is very hard to compare with other methods and understand the contribution of this effort.

• We are grateful to the reviewer for highlighting a potential limitation in our study. As our work is pioneering in the field of multi-concept learning without ground-truth object segmentation, we acknowledge the scarcity of directly comparable baselines.
• Our original experiments were extensive, taking about 80 days on a single GPU, a fact recognised by other reviewers. However, following R2's advice, we added Break-A-Scene (BAS) as an additional benchmark. We perform extensive new experiments resulting in an extra seven pages in the revised version. We have updated Figures 5, 7, 8, 16-23, adding Figures 10-12, and detailing in Sections A.1 and A.2 for a comprehensive comparison. This involved 160 more single GPU runs and around 2 million pairwise comparisons.
• This thorough approach ensures our approach is quantitatively compared against three baselines (Textural Inversion, MCPL, and BAS) and a qualitative comparison against an additional five baselines (Figure 11-12), comprehensively covering the scope of the field.
• It worth noting BAS requires segmentation masks as input and employs separate segmentation models to produce masked objects, hence integration BAS involves extensive human-in-the-loop pre- and post-processing efforts, as detailed in Section A.1. In contrast, our method is mask-free at learning and employed its own AttnMask to generate masked objects.
• In the following results (Figure 8 in table format), our fully regularised method (MCPL-one+CL+Mask) shows competitive performance against BAS on natural images. For the out-of-domain (OOD) medical dataset, BAS leads in the DINOv1 space, but we're comparable in others. This is due to our less precise object masks compared to BAS's human-assisted MedSAM segmentation, shown in Figures 5 and 6.

Object-level fidelity learned concepts referencing to "ground truth" (natural images)

Object-level fidelity learned concepts referencing to "ground truth" (medical images)

• The new t-SNE results in Figure 7 demonstrate that the learned embeddings from both the mask-based ’ground truth’ and BAS show less disentanglement compared to ours, attributable to their lack of a specific disentanglement objective, such as the PromptCL loss in MCPL.

---

W.1 - The paper is not very clear to read. The idea seems to be straightforward, but the description of the method is a bit ambiguous…

• We thank the reviewer for pointing out the clarity issue in our manuscript. In response, we have simplified certain descriptions in the methods section to enhance clarity.
• We also acknowledge our methods section combined method descriptions with preliminary motivational experiments, and this design might have disrupted the flow of the method description. To address this, we have reorganised the methods section, including adjustments to Figure 4, to ensure a more coherent and seamless presentation of our methods.

We are pleased to note the reviewer's agreement that our work targets critical research areas in text-guided learning of multiple prompts from a single scene without ground-truth object segmentation. We are also thankful for the confirmation of our contribution in introducing a novel dataset.

Furthermore, we acknowledge and appreciate the reviewer's questions regarding novelty, quantitative evaluation and clarity in our manuscript. We recognise that these issues may stem from misunderstandings. To address these concerns, we have provided detailed responses to each question below, aiming to clarify any ambiguities and enhance the overall comprehensibility of our work.

---

Q.1 - Attention masks and contrastive loss on different concepts seem to be a widely used method. It would be great if the authors can explain a bit more about the novelty of their work.

We appreciate the reviewer's inquiry and are pleased to clarify our contributions as follows:

• Firstly, we highlight the consensus among all reviewers on our work targets on important research areas of mask-free text-guided learning of multiple prompts from a single scene.
• We introducing Multi-Concept Prompt Learning (MCPL), the first to achieve multi-concept learning without relying on object masks as input. This novelty has been recognised by all the other reviewers.
• Our approach differs to the leading mask-based method, Break-A-Scene (BAS) as follows:
  • BAS learns multiple concepts from images paired with object-level masks but we don’t use a mask;
  • BAS updates both textural embeddings and model weights but we don’t update model weights;
• We also summarise other key contributions our work have made:
  • The motivational study introduces the problem and the current limitation with vanilla MCPL. Addressing this, we propose three innovative regularisation techniques: Attention Masking, Prompts Contrastive Loss, and Bind Adjectives, to disentangle multiple object concepts and enhance the accuracy of object-level concept learning.
  • Extensive experiments and analysis demonstrate superior performance with our method against competitive baselines, delivering results comparable to the leading mask-based method, Break-A-Scene. This highlights the potential of our mask-free approach. We also provide detailed visualisations to showcase the practical applications of our method.
  • We have compiled a new dataset for comprehensive evaluation. This dataset involves both in-domain natural images and out-of-domain biomedical images, setting up a new benchmark for the field.
  • Overall, our approach not only enhances current methodologies but also paves the way for novel applications, such as facilitating knowledge discovery through natural language-driven interactions between humans and machines.
• All of the above clarifications and contributions have been reflected in our revised manuscript.