GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation

We sincerely appreciate your review, acknowledging our work to be “well-written” and possess “superior efficacy to other methods”. Here, we address the concerns and questions that have been raised.

(1) Clarification on the Main Method

We would like to draw your attention to our general response above, where we provide detailed clarifications regarding our problem definition and method. Please note that the notations in the below sections are also based on the clarified versions.

(2) Clarification on Lines 158-161 of Sec. 4.3

It is indeed obvious that if two noisy images $\mathbf{x}\_t$ and $\mathbf{y}\_t$ are initialized with different noises, they will produce different outputs if denoised separately (lines 158-161). However, we reiterate this fact to emphasize that as we increase the number of shared sampling between these two noisy images, $\mathbf{x}\_t$ and $\mathbf{y}\_t$ increasingly produce semantically similar outputs, even when starting from two different initial noises (lines 170-171). This phenomenon, also visualized in Fig. 2-(A), (B) of the main paper, signifies the intriguing effect of semantic sharing when two noisy images pass through the DiT together.

(3) Comparison between Semantic Sharing and Image Interpolation

Thank you for your comment on comparing our proposed method with the task of image interpolation. After reviewing the suggested paper [1], we would like to clarify the main purpose of our proposed shared sampling.

Unlike image interpolation, which takes two (or more) input images and produces an output image with semantic features that lie between those of the input images, the goal of our shared sampling is to transfer the semantic features of one noisy image to the other. This process ensures that the desired concepts appear in the target noisy image.

(4) Clarification on Grounding Loss $\mathcal{L}$

Please note that we provide clarifications on the definition of the grounding loss in our general response. We will also include this clarification in the revised version.

(5) Clarification on Handling Multiple Objects of the Same Class

In Stage 2 of our GrounDiT, please note that a separate object branch and a corresponding noisy object image $\mathbf{z}\_{i, t}$ is defined and processed in parallel for each grounding condition $g_i$. This ensures that each object is individually handled, therefore providing precise local guidance for each bounding box region $b_i$.

(6) Further Clarification on Shared Sampling Mechanism

In Fig. 3 of the main paper, we intentionally depict the noisy object image $\mathbf{z}\_{i, t}$ as larger than the corresponding bounding box. This highlights that even when $\mathbf{z}\_{i, t}$ and the bounding box $b_i$ are not the same size, shared sampling between the two noisy images is still feasible. This is achieved by assigning positional embeddings so that each image is treated as a whole. This case is also visualized in Fig. 2-(B) of the main paper.

Moreover, as describe in our general response, the noisy object image $\mathbf{z}\_{i,t}$ is set to satisfy the following criteria:

• Size of $\mathbf{z}\_{i,t}\in\mathbb{R}^{H’_i \times W’_i \times D}$ is set within DiT’s preferred token sequence range.
• $\mathbf{z}\_{i,t}$ has a similar aspect ratio as its corresponding bounding box $b_i$.

Therefore, the noisy object image $\mathbf{z}\_{i, t}$ does not require a scaling process after it has been initialized. Simply passing $\mathbf{z}\_{i, t}$ and the nosy patch $\mathbf{b}\_{i,t}$ (originally $B_{i,t}$) together through the DiT for denoising is sufficient to perform shared sampling.

(7) Consistency of Images after Noisy Patch Transplantation

To prevent potential inconsistencies between the bounding box regions and the background, our GrounDiT guidance is applied during the initial timesteps of the reverse process, as detailed in Appendix A.1 of our main paper.

Specifically, our reverse process consists of 50 steps, with GrounDiT denoising (Stage 1 and Stage 2) applied for the first 25 steps. For the final 25 steps, the noisy image $x_t$ is denoised using the standard DiT denoising step. This strategy, also employed in previous works such as R&B, leverages the fact that the image structure is primarily determined in the early steps. Therefore, applying our guidance in the early steps is sufficient for accurate grounding, while preventing inconsistencies between the bounding box regions and the background regions.

(8) Clarification on Hadamard Produce in Eq. (9)

Thank you for the comment. We correct Eq. (9) below. We will incorporate the clarification in our revised version.

Let $**UNCROP**(\cdot)$ be a function that zero-pads a patch $\mathbf{b}_{i,t}$ at region $b_i$ to match the size of the original image. Then, the transplantation mechanism originally proposed in Eq. (9) can be clarified as follows:

$

& \text{for}\; i = 0, ..., N-1:\\ & \quad\; \tilde{\mathbf{x}}_{t-1}\leftarrow \tilde{\mathbf{x}}_{t-1}\odot (1-M_i)+ UNCROP(\mathbf{b}_{i,t-1},b_i)\odot M_i\\ & \mathbf{x}_{t-1}\leftarrow \tilde{\mathbf{x}}_{t-1}

$

Please note that the binary mask $M_i$ is passed through the Hadamard product with $**UNCROP**(\mathbf{b}\_{i,t-1},b_i)$, which has the same resolution as $M_i$.

[1] Interpolating between Images with Diffusion Models, Wang et al., ICML 2024 Workshop

We greatly appreciate your review, recognizing our method to be “very interesting” and “show a lot of promise”. Here, we address the concerns and questions that have been raised.

(1) Clarifications on the Main Method

Please find the detailed clarification of the method in our general response above. We will thorougly improve our presentation in the revised version. Moreover, please note that the notations in the below sections are also based on the above clarification.

(2) Clarification on Grounding Loss $\mathcal{L}$

Please note that we provide clarifications on the definition of the grounding loss in our general response. We will also include this clarification in the revised version.

(3) Parallel Denoising of the Main Branch and Object Branch

While in Algorithm 1 we meant to convey the parallel process of main branch and object branches, we notice that the presentation can be improved for clarity. We would like to provide clarifications below and further improve in the revised version.

Main Branch: First, Line 13 of Algorithm 1 indicates the denoising step of the noisy image in the main branch. Importantly, the output from this step, $\tilde{\mathbf{x}}\_{t-1}$ (originally $\mathbf{x}\_{t-1}$), is not directly used as an input for the denoising steps in each object branch.

Object Branch: Lines 6-8 of Algorithm 1 represent the denoising steps of the object branches. Here, the output from the main branch, $\tilde{\mathbf{x}}\_{t-1}$, is not directly used as input for the denoising. Instead, the output noisy patch $\mathbf{b}\_{i, t}$ (originally $B_{i,t}$) in Line 8 is later pasted back into the output $\tilde{\mathbf{x}}\_{t-1}$ from the main branch, as shown in Line 9.

Therefore, since the output from either the main branch or the object branches is not used as an input for the other branch, these processes are indeed parallel.

(4) Details on the Reverse Process of GrounDiT

We clarify the reverse process of GrounDiT, supplementing Appendix A.1 of the main paper.

Our reverse process consists of 50 steps using the DPM-Solver scheduler, where the GrounDiT denoising (Stage 1 and Stage 2) is applied for the first 25 steps. For the final 25 steps, the noisy image $\mathbf{x}_t$ is denoised following the standard DiT denoising step. This strategy is also employed in previous works, including R&B, as the structure of the image is known to be mostly decided in the early steps. Therefore, applying our guidance in the early steps is sufficient for accurate grounding.

(5) Handling Multiple Image Sizes in DiT / Clarification on $w_d$ in Eq. (3)

We appreciate your feedback on enhancing the clarity of our descriptions in Sec. 3. To address this, we have clarified DiT's mechanism for handling multiple resolutions below and will incorporate this into our revised version.

DiT [1] handles multi-aspect/size noisy images by operating on a sequences of tokens. Given a noisy image $\mathbf{x}\_{t}\in \mathbb{R}^{H\times W\times C}$, DiT first divides it into patches of size $l \times l$, creating a total of $T = (H/l) \times (W/l)$ patches. Each patch is then transformed into a token with a hidden dimension of $D$ through linear embedding. A 2D sine-cosine positional embedding is applied to the sequence of tokens. The detailed formulation of positional embedding is given in Eq. (3) of our main paper, where $w_d$ follows original definition of positional embedding: $w_d = 1/10000^{(4d/D)}$ with $d$ running from $0$ to $D/4$. Consequently, when images with different aspect ratios or sizes are input into DiT, the primary difference is the token sequence length, which is managed by assigning appropriate positional embeddings to each sequence.

(6) Simultaneous Update in a Single Reverse Diffusion Process

Please note that a single denoising step of our GrounDiT consists of the two-stage sequence: "Stage 1 $\rightarrow$ Stage 2". During Stage 1, a global update is applied to the noisy image $\mathbf{x}\_t$. In Stage 2, precise local control is provided for each bounding box $b_i$ using our proposed shared sampling technique. We found it reasonable to perform the shared sampling based on the output $\hat{\mathbf{x}}\_t$ obtained from Stage 1, as this allows to leverage the improved alignment resulting from the global update.

[1] Scalable Diffusion Models with Transformers, Peebles et al., ICCV 2023

We greatly appreciate your review, acknowledging that our paper is “well-written” and the proposed method is “simple and intuitive”. Here, we address the concerns and questions that have been raised.

(1) Additional Comparisons on Prompt Fidelity

Here we provide further quantitative comparisons on the prompt fidelity between our GrounDiT and baseline methods. We will include the results in the revised version.

[Prompt Fidelity]

The table above shows the comparison of CLIP score, ImageReward and PickScore with the baselines. Since PickScore measures the preference among two images, we present their difference in the above table. In CLIP score and ImageReward, our GrounDiT outperforms the baselines. In PickScore, GrounDiT outperforms all baseline except PixArt-R&B, while remaining comparable.

(2) Relation between Grounding and Prompt Fidelity

Additionally, we observed that precise local control provided by our GrounDiT method leads to the presence of target objects that are often missing in baseline methods, thereby increasing the prompt fidelity of the image. Please refer to Fig. S1 in the attached PDF of our general response, where we provide qualitative examples of these cases along with the corresponding CLIP scores for each image.

(3) Additional Quantitative Comparisons including MS-COCO

We further conducted quantitative comparisons on two additional benchmarks: a subset of the MS-COCO and a new custom benchmark.

[MS-COCO Subset]

Following R&B, we conducted an experiment on a subset of MS-COCO. Since we could not obtain the exact subset utilized in R&B from the authors, we similarly prepared a subset from MS-COCO.

For this, we first filtered out the image-caption pairs in the MS-COCO 2014 validation set where either the bounding box target objects were not specified in the image captions, or duplicate objects were present in the grounding conditions. Subsequently, we randomly selected 500 pairs to use for evaluation. Following R&B, we measured the mIoU (mean IoU) for each bounding box condition.

As shown in the table above, our GrounDiT achieves the highest grounding accuracy on the MS-COCO subset. Specifically, GrounDiT outperforms R&B by 0.021, representing a 5.1% increase, and PixArt-R&B by 0.014, representing a 2.2% increase. Note that the average number of bounding boxes in the MS-COCO subset is 2.06, making it a relatively easy task even for baseline methods. Since the main advantage of GrounDiT over R&B and PixArt-R&B is its robustness when there is a higher number of bounding boxes, we provide further comparisons with a higher average number of bounding boxes below.

[HRS-Spatial Benchmark]

The above table provides the mIoU comparisons on the Spatial subset of the HRS benchmark, which was also used in Sec. 5.2. The average number of bounding boxes in this benchmark is 3.11, which is larger than MS-COCO subset by 1.05. For mIoU, GrounDiT outperforms R&B by 0.046, indicating a 14.1% increase, and PixArt-R&B by 0.038, indicating a 11.4% increase. Compared to the 5.1% and 2.2% in the MS-COCO subset, respectively, the higher percentage of increase in mIoU demonstrates the robustness of GrounDiT on more complex grounding conditions.

[Custom Benchmark]

Lastly, the above table shows the comparisons on a new benchmark consisting of 500 layout-text pairs, generated using the layout generation pipeline from LayoutGPT [1]. The average number of bounding boxes in this benchmark is 4.48. Here, GrounDiT outperforms RnB by 0.052, representing a 26.3% increase and PixArt-R&B by 0.044 representing a 21.4% increase. This is in line with the trend in the above benchmarks, further highlighting the robustness and efficacy of our approach in handling a higher number of grounding conditions.

(4) Details on Computation Time

We further provide the exact computation time for each method.

The numbers in the table below are in seconds.

While our method results in an increase in inference time, the rate of increase is not significant. For three bounding boxes, the inference time is 1.03 times that of R&B and 1.33 times that of PixArt-R&B for three bounding boxes. Even with 6 bounding boxes, the inference time is 1.41 times that of R&B and 1.90 times that of PixArt-R&B.

[1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models, Feng et al., NeurIPS 2023

We sincerely appreciate your review, acknowledging that our work is solving a “relevant” problem of zero-shot guided generation through an “effective” method. Here, we address the concerns and questions that have been raised.

(1) Clarifications on the Main Method

We would like to draw your attention to our general response above, where we provide detailed clarifications regarding our method.

(2) Clarification on Grounding Loss $\mathcal{L}\_{AGG}$

Please note that we provide clarifications on the definition of the grounding loss in our general response. We will include this in the revised version.

(3) Details on Evaluation Metrics

We provide further details on the evaluation metrics from HRS-Bench [2]. We will also include this in the revised version.

• Spatial: Object detection is applied to the generated image. Based on the detection results, the spatial relations specified in the text prompt are checked. If the detected objects maintain the correct spatial relations as described in the text prompt, the image is classified as correct; otherwise, it is classified as incorrect.
• Size: Similarly, based on the detected boxes, it is checked whether the size relations between the boxes follow the relations specified in the text prompt (e.g. “bigger than”).
• Color: After obtaining a semantic segmentation for each object, the average hue color value within each segment is computed. The derived color is then compared with the color of the object specified in the text prompt to determine accuracy.

(4) Quantitative Comparisons on Additional Benchmarks

We further conducted quantitative comparisons on two additional benchmarks: a subset of the MS-COCO and a new custom benchmark.

[MS-COCO Subset]

Following R&B, we conducted an experiment on a subset of MS-COCO. Since we could not obtain the exact subset utilized in R&B from the authors, we similarly prepared a subset of MS-COCO.

We first filtered out the image-caption pairs in the MS-COCO 2014 validation set where either the bounding box target objects were not specified in the caption, or duplicate objects were present in the grounding conditions. Then we randomly selected 500 pairs for evaluation. Following R&B, we measured the mIoU (mean IoU) for each bounding box condition.

In the table above, our GrounDiT achieves the highest grounding accuracy on the MS-COCO subset. Specifically, GrounDiT outperforms R&B by 0.021, representing a 5.1% increase, and PixArt-R&B by 0.014, representing a 2.2% increase. Note that the average number of bounding boxes in the MS-COCO subset is 2.06, making it a relatively easy task even for baseline methods. Since the main advantage of GrounDiT over R&B and PixArt-R&B is its robustness when there is a higher number of bounding boxes, we provide further comparisons with a higher average number of bounding boxes below.

[HRS-Spatial Benchmark]

The above table provides the mIoU comparisons on the Spatial subset of the HRS benchmark, which was also used in Sec. 5.2. The average number of bounding boxes in this dataset is 3.11, which is larger than MS-COCO subset by 1.05. For mIoU, GrounDiT outperforms R&B by 0.046, indicating a 14.1% increase, and PixArt-R&B by 0.038, indicating a 11.4% increase. Compared to the 5.1% and 2.2% in the MS-COCO subset, respectively, the higher percentage of increase in mIoU demonstrates the robustness of GrounDiT on more complex grounding conditions.

[Custom Benchmark]

The above table shows the comparisons on a new benchmark consisting of 500 layout-text pairs, generated using the layout generation pipeline from LayoutGPT [1]. The average number of bounding boxes in this benchmark is 4.48. Here, GrounDiT outperforms RnB by 0.052, representing a 26.3% increase and PixArt-R&B by 0.044 representing a 21.4% increase. This is in line with the trend in the above datasets, further highlighting the robustness and efficacy of our approach in handling a higher number of grounding conditions.

(5) Additional Qualitative Comparisons with R&B

Please refer to Fig. S1 of the attached PDF in the general response, where we provide additional qualitative comparisons between our method and the baselines, R&B and PixArt-R&B, with detailed analysis of each result.

(6) Advantage of Shared Sampling over Directly Transferring the Patches

Please refer to our general response for a clarification on the effect of shared sampling:

We can think of each patch as an image on its own and pass it through DiT with the corresponding text condition. However, this is not feasible in practice with existing DiT, as it has a certain set of preferred image resolutions. This is due to the fact that the training data does not perfectly cover all resolutions. Please see Fig. S2 of the attached PDF in our general response for an illustration. But since DiT's Transformer architecture allows flexible token sequence length, we can perform shared sampling, which transfers the desired semantic features into the patch.

[1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models, Feng et al., NeurIPS 2023

[2] HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models, Bakr et al., ICCV 2023