Novel Object Synthesis via Adaptive Text-Image Harmony

Q1: Results by the different assumption.

A1: Thank you for your suggestion. Inspired by $ref-1$ , we only inject the late self-attention layers in our method while keeping other settings the same. This simple adjustment enables our model to effectively modify the visual style of generated images while maintaining the desired structural features. In Figure R3, our method can create an image where the shape is guided by the text ("shark"), and the appearance (color/pattern) is influenced by the referenced image of a dog.

There are some key differences between our method and Visual Style Prompting (VSP) $ref-1$ . First, our method introduces a new task that combines an object text with an object image to create a new object image, while VSP is a novel approach that guides the desired style using a reference image. Second, we propose ATIH to achieve a creative and harmonious fusion of two distinct objects, resulting in surprising and imaginative combinations. VSP aims to produce a diverse range of images while maintaining specific style elements and nuances. Third, while VSP primarily adjusts the visual style, our results demonstrate the novel combination of object text and object image in Figure R3.

$ref-1$ Visual Style Prompting with Swapping Self-Attention. arXiv preprint arXiv:2402.12974.

Q2: Results by using ControlNet-like model to inject structural information.

A2: Thank you for the opportunity to clarify the performance of our method compared to ControlNet, particularly in terms of achieving a balanced and semantically coherent image synthesis.

Comparison with ControlNet:
We rigorously compared our method with ControlNet to evaluate their capabilities in handling complex text-image fusion tasks in Figure R2. Our findings indicate significant differences in how both approaches manage the fusion process. ControlNet tends to maintain the structure from depth or edge maps well but struggles with semantic integration, especially when faced with complex prompts, failing to achieve a seamless blend. By contrast, our method utilizes the full spectrum of RGB image features, including color and texture, alongside structural data.

Q3: Results by controlling image and text strengths.

A3: Thank you for highlighting the relevance of adjusting the guidance strength in image and text synthesis, as seen in methods like InstructPix2Pix.

Comparison with InstructPix2Pix:
We manually adjusted the image and text strengths for InstructPix2Pix to match the controls of our method. We varied the image strength (e.g., 1.0, 1.5, 2.0) and text strength from 1.5 to 7.5 and observed the outcomes. In Figure R3, at optimal settings of image strength 1.5 and text strength 5.0, InstructPix2Pix produced the best fusion, but it is unnatural, such as replacing the rabbit's ears with the rooster's head. In contrast, our results are new and natural combinations of the rabbit and the rooster, demonstrating superior visual synthesis compared to InstructPix2Pix.

More importantly, our method can automatically iterate to the optimal fusion of image and text, eliminating the need for manual parameter adjustments. This automation significantly enhances the usability and efficiency of our synthesis process, allowing for seamless generation of harmonious images that meet both aesthetic and semantic criteria.

Q4: Baseline using only image guidance before t, text guidance after t.

A4: Thank you for suggesting the exploration of different guidance strategies.

Results and Analysis:
Figure R6 illustrates the transition and effects of this guidance strategy: At $t=0$, the output is purely influenced by the text, which in this case describes an ostrich. Starting at $t=1$, image guidance is introduced, leading to a more complex synthesis process where the features of the deer image begin to merge with the previously established text-based ostrich features.

Figure R6 shows the results that the integration is not fully balanced, as the distinctive features of the ostrich—such as its neck and feather texture—are not as prominent as desired. This indicates a need for more refined control over the balance and timing of text and image guidance.

Q5: Discussing subject-driven generation methods.

A5: Thank you for mentioning the subject-driven methods. Recently, subject-driven text-to-image generation focuses on creating highly customized images tailored to a target subject $3,4,5,6$ . These methods often address the task, such as multiple concept composition, style transfer, and action editing $7,8,9$ . In contrast, our approach aims to generate novel and surprising object images by combining object text with object images.

Comparison with Kosmos-G:
Kosmos-G $9$ utilizes a single image input and a creative prompt to merge with specified text objects. The prompt is structured as “<i> creatively fuse with {object text},” guiding the synthesis to innovatively blend image and text elements. Our findings indicate that Kosmos-G can sometimes struggle to maintain a balanced integration of original image features and text-driven attributes. In Figure R4, the images generated by Kosmos-G often exhibit a disparity in feature integration.

Q6: Can the method use text for shape and image for texture?

A6: Thanks. Altering the shape of a given object image is quite challenging. To the best of our knowledge, there is no method that performs this task well. Fortunately, our method can generate a new object with slight deformations, such as the axolotl's mouth in Figure 2. However, our method still cannot significantly deform the shape of the object image, as we incorporate all self-attention to preserve image information during the diffusion process. We will study this large deformation in the future work.

Q1: Dataset and fine-tune baselines.

A1: Thank you for your interest in our dataset and its potential applications.

Dataset Release:
We plan to release a dataset of 1,800 text-image pairs to the research community following acceptance. These pairs are created using an outer product on 60 different object texts and 30 different object images. Specifically, we selected 60 texts from the ImageNet dataset $43$ , detailed in Table 4, including categories such as kit fox, peacock, African chameleon, white shark, acorn, zucchini, and fire engine. The 30 images were chosen from the PIE-bench dataset $25$ , shown in Fig. 10, with each image corresponding to similarly categorized texts outlined in Table 5, such as Corgi, Duck, ladybug, flower vase, apple, jar, man, and Twitter logo. More details are provided in Appendix C. It should be noted that there are no ground truths for novel and surprising objects in this dataset created from text-image pairs.

No fine-tuning for all baselines and our ATIH method:
Due to the absence of ground truths, this dataset cannot be used to train any baselines or our ATIH method. Instead, it is solely for evaluating performance using four key popular metrics: CLIP text-image similarity (CLIP-T) $39$ , Dinov2 image similarity (Dino-I) $36$ , aesthetic score (AES) $46$ and human preference score (HPS) $56$ , and our two proposed metrics: Fscore and balance similarities (Bsim). Our ATIH method can generate novel and surprising objects by designing an adaptive balance between the provided object text and object image.

Q2: Using released or SDXL-Turbo-trained models.

A2: Thanks. We directly used the released models for the baseline methods, including InstructPix2Pix, due to the following reasons. First, our dataset lacks ground truths for novel and surprising objects created from text-image pairs, preventing us from training or fine-tuning these baseline models. Second, using the officially released models allows us to provide a direct and unbiased comparison, as these models have been trained and optimized by their developers. Third, we employed the officially released SDXL-Turbo $45$ as a base model to create novel and surprising objects using our ATIH method, since no ground truths were used to train the SDXL-Turbo model.

Additionally, we manually adjusted the image and text scales of InstructPix2Pix, as shown in Figure R3. Our ATIH method demonstrates significantly better visualization compared to InstructPix2Pix, which displays objects with unnatural combinations.

Q1: Remove the misleading images.

A1: Thank you for your feedback regarding the potential misunderstanding in Figure 2. We acknowledge that the image for text information was intended for visualization purposes only, and we agree that it could mislead readers. Therefore, we will remove the images and present only the text in the same location.

Q2: Adjusting baseline strengths of InstructPix2Pix.

A2: Thank you for your suggestion. We agree that adjusting the strengths of existing baselines, such as InstructPix2Pix, is essential. Using 'cock' (text) and 'rabbit' (image), we manually adjusted the image strength (e.g., 1.0, 1.5, 2.0) and text strength (e.g., 1.5, 2.0, 2.5, 4.0, 4.5, 5.0, 6.5, 7.0, 7.5) for InstructPix2Pix, and the results are shown in Figure R3. Using the optimal image and text strengths of 1.5 and 5.0, respectively, InstructPix2Pix generates unnatural combinations, such as the head of the cock replacing the ears of the rabbit, as seen on the left of Figure R3. In contrast, our method produces natural and novel combinations, with the head and feet of the rabbit fused correspondingly with the head and feet of the cock, in the right of Figure R3.

Q3: Ablation study of Golden Section Search.

A3: Thank you for your insightful feedback regarding the role of the Golden Section Search algorithm.

Role of Golden Section Search:
In our approach, we designed a score function to evaluate the quality of the text-image fusion. We utilize the Golden Section Search algorithm to efficiently find the optimal parameters that maximize our score function, allowing us to achieve a well-balanced and harmonious fusion of text and image features.

Alternative Search Methods:
While the Golden Section Search is our chosen method for parameter optimization due to its efficiency and simplicity, we acknowledge that other methods, such as Ternary search $ref-1$ or even Random search $ref-2$ , could potentially achieve similar results. The primary advantage of the Golden Section Search is its ability to converge to an optimal solution with relatively few iterations. In our task, it reaches convergence two iterations faster than the Ternary Search.

Importance and Flexibility:
The choice of search algorithm is flexible, and the primary contribution is the score function itself, which guides the search for optimal fusion parameters. The Golden Section Search provides a systematic approach, but its role can be substituted by other search methods if desired.

Ablation Study:
We conducted an ablation study comparing the Golden Section Search and Ternary Search, as shown in Figure R7. The images produced using the Golden Section Search exhibit a more novel fusion effect.

$ref-1$ Kiefer, J. (1953). Sequential Minimax Search for a Maximum. Proceedings of the American Mathematical Society, 4(3), 502-506.

$ref-2$ Solis, F. J., & Wets, R. J. B. (1981). Minimization by Random Search Techniques. Mathematics of Operations Research, 6(1), 19-30.

Q4: Discussing evaluation criteria for ambiguous tasks.

A4: Thank you for your understanding. We acknowledge that evaluating (1) complete replacement and (2) harmonious combination is challenging. For complete replacement, we can consider two metrics: the difference between the given object text and its generated object patch, and the harmony and naturalness of the entire image replaced by the given object text. To measure the difference, we can use CLIP text-image similarity (CLIP-T). For evaluating harmony and naturalness, we can employ the aesthetic score (AES) and the human preference score (HPS).

For harmonious combination, our goal is to generate an object that incorporates characteristics of both A and B. If the generated object is unbiased towards either A or B, it indicates a balance between them. To achieve this, we first use CLIP text-image similarity (CLIP-T) and Dinov2 image similarity (Dino-I) to ensure the generated object has high similarities with both A and B. We then introduce a balance similarity metric ($B$sim) to measure the equilibrium between A and B. If the generated object shows a bias towards either, we adjust it using a scale factor $k$.

Q1: More quantitative validation

A1: Thank you for your thoughtful feedback. We recognize the importance of quantitative metrics in validating our algorithm's effectiveness.

We use four widely-used metrics: CLIP-T $39$ , Dino-I $36$ , AES $46$ , and HPS $56$ . Additionally, we propose two new metrics: Fscore and balance similarity (Bsim), to evaluate our ATIH method.

CLIP-T and Dino-I measure similarity between the generated image and the object text or image, indicating how well the generated object incorporates characteristics of both sources. The balance similarity metric (Bsim) assesses the equilibrium between text and image fusion. The Fscore considers both higher CLIP-T and Dino-I, and the best balance. AES and HPS evaluate the aesthetic quality and human preference for the generated object.

Table 1 shows that we achieve relatively high CLIP-T and Dino-I scores, though not the highest single scores. However, we obtain the best Bsim, Fscore, AES, and HPS. We also conducted a comprehensive user study to gather qualitative feedback on the perceived quality and creativity of the synthesized images.

Q2: Expanding dataset.

A2: Thank you for your valuable feedback regarding the dataset. We agree that testing the algorithm across a broader range of scenarios can provide more insights. Our dataset is created using an outer product on 60 different object texts and 30 different object images, representing widely-used categories. For example, the porcupine-bottle in Figure 1 illustrates the diverse text-image pairs.

We selected 30 images from the PIE-bench dataset, representing a diverse set of distinct categories, each with a clear and identifiable subject in Figure 10 and Table 5, including categories such as Corgi, jar, and man.

For the text categories, we utilized ImageNet and employed ChatGPT to filter and select 40 distinct animal categories and 20 non-animal categories, in Table 4. These include categories such as kit fox, peacock, acorn, and zucchini. This selection was intended to cover a broad range of typical scenarios while ensuring that the algorithm could effectively handle both common and varied text-image combinations.

Q3: Handling edge cases and unusual text-image pairs

A3: Thank you for your insightful comments on our method's performance with edge cases and unusual text-image pairs.

To address edge cases, our method effectively captures prominent features like colors and basic object forms from complex textual descriptions. Figure R1 shows a well-defined edge structure for the fawn image and the text 'Green triceratops with rough, scaly skin and massive frilled head.'

Handling unusual text-image pairs is challenging. Our method relies on semantic correlations within the diffusion feature space. When the semantic match is weak, it tends to produce mere texture changes rather than deeper transformations. This suggests our approach may struggle with transformations between categories with weak semantic associations. Future work could enhance semantic matching between categories to improve generalizability. See Appendix B.

Q4: More detailed comparative analysis with existing methods.

A4: Thank you for your insightful feedback. The core of our method is an adaptive text-image harmony approach that balances the similarities between the generated object and the given text-image pair. Unlike existing methods like MasaCtrl, InfEdit, ConceptLab, and MagicMix, our approach ensures better fusion, especially in challenging cases such as the African chameleon-bird, minimizing distortions and maintaining high image quality.

Even with manual adjustments to match our controls, InstructPix2Pix produced unnatural results. In Figure R3, even at optimal settings (image strength 1.5, text strength 5.0), InstructPix2Pix's fusion was still unnatural, such as replacing the rabbit's ears with the rooster's head.

Q5: Technical details of our method.

A5: Thank you for your suggestion.The technical details of our method are provided in Section 3 and Subsection 4.1, and summarized in Algorithm 1 in Appendix F.

We implemented our ATIH method using the SDXLturbo, achieving a processing time of ten seconds per sample. The input consists of a subject-specific object image and a fused object text.

Key parameters in our experiments include:

• The balance parameter λ is set to 125.
• The scale factor k is set to 2.3.
• The similarity thresholds $I _{sim}^{min}$ and $I _{sim}^{max}$ are set to 0.45 and 0.85, respectively.

Q6: User feedback integrated into algorithm refinement.

A6: Thank you for emphasizing the importance of user studies in our research. We clarify that user studies are used solely to evaluate the performance of our ATIH method, not to refine it. User studies aim to evaluate and validate the performance of our ATIH method. These studies were designed to gather qualitative feedback on the novelty, coherence, and visual appeal of the generated images, rather than for algorithm refinement. We plan to incorporate it into subsequent versions to enhance usability and customization.

Q7: Statistical significance tests.

A7: Thank you for your suggestion to provide a more detailed comparative analysis. We conducted H tests to determine the statistical significance of performance differences between our method and other methods across various metrics. The results are presented in Table R1. For example, AES and HPS: Against Instructpix2pix, our method demonstrates statistically significant differences with H statistics of 268.57$(p < 10^{-59})$ for AES and 39.63 $(p < 10^{-9})$ for HPS, indicating potential improvements in aesthetic quality and human preference scoring. These results validate the effectiveness of our approach, demonstrating its superiority across multiple critical metrics.