Thanks for constructive comments. We would like to response as follows.

Q1: The paper is written on a quite high level with many details left out or in appendix.

A1: We focused on content shift and how generation techniques affect it. Some details were thus omitted in the main body due to the space limitation, which might be confusing. We will revise according to your advice.

For pipeline, please refer to global question GQ1.
For experimental details, we show part of refinement below due to space limit.

We extract features at $t=50$.

• Semantic Segmentation
  • Convolutional features:
    • One with fine-grained prompt "a highly realistic photo of the real world. It can be an indoor scene, or an outdoor scene, or a photo of nature. high quality" plus all category labels (ADE20K).
    • Two features: same prompt as above, ControlNet and two different LoRA weights.
  • No attention features are used, as they do not bring benefit in complex scenes.

Q2: The technical contribution is limited, combining previous techniques.

A2: Perhaps due to presentation, it is a pity to leave you this impression. In fact, our main contribution is the pioneering discovery of content shift, a widespread, harmful, yet hidden behavior of diffusion features. We systematically investigated its cause and thus found that existing generation techniques are enough to mitigate such shift effectively. Hence, we choose these techniques for easier implementation and extension. The key point is: among numerous techniques, which can serve for discrimination tasks? Our GATE guideline bridges this gap and inspires the necessary adaptation to these techniques. We hope all these observation, analysis, guidance, and adaptation can benefit future work.

Q3: More experimental details and ablation study.

A3: For more evaluation and details, please refer to GQ4 and your Q1. Fig.10 serves as ablation. For your convenience, we list results below, which validate the effectiveness of each technique and their combination.

Q4: Provide necessary details for pipeline and experiments in the main content.

A4: We will refine manuscript according to your suggestions. The pipeline in GQ1 will replace Fig.2 in Preliminary. The experimental details in Q1 will be added.

Q5: What noise levels were used? Is it possible to combine different noise levels?

A5: In fact, the choice of noise level is orthogonal to our work. Early diffusion feature work [2] systematically analyzed this issue and suggested $t=50$ as a general choice. We observe similar results and thus follow this practice. As shown below, we test ControlNet with other noise levels on a single Horse-21 split, where GATE outperforms baseline across noise levels, especially when the noise level is stronger.

It is reasonable to combine different noise levels, which is actually the focus of some prior arts [23, 56], including a semantic correspondence SOTA. Nevertheless, The results in Tab.1 show that amalgamating different generation techniques brings stronger enhancement.

Q6: A more systematic way of quantifying content shift.

A6:
We also developed a quantitative metric. We set feature extracted at $t=0$ as reference $FEAT_{ref} \in \mathbb{R}^{c\times h\times w}$ as it is less affected by noises. We use Laplacian operator to evaluate the contour difference between feature $FEAT$ and the reference:

$$diff=\sum_{i,j}^{h\times w}|\left\|Laplacian(FEAT_{ref},i,j)\right\|_2-\left\|Laplacian(FEAT,i,j)\right\|_2|$$

We then set a feature with stronger content shift as anchor and compare $diff_{anchor}$ with other features.

$$Score=\frac{(diff_{anchor}-diff)}{diff_{anchor}}\in(-\infty,1]$$

$Score=1$ means exact match, and a smaller value indicates more shift. Below are quantitative results obtained on a single Horse-21 split, showing $Score$ is consistent with mIoU performance.

In our manuscript, we use qualitative GATE guideline because it can well approximate $Score$ and is much simpler. According to your advice, we will add this metric in refinement.

Q7: How can the method generalize to more diverse types of images?

A7: Among the three techniques, ControlNet and LoRA are insensitive. For fine-grained prompt, generalization is clarified in GQ3.

Q8: How does the method perform in classification tasks?

A8: Diffusion features have much more channels (3520) than input images (RGB). Hence, prior arts prefer pixel-level task to image-level task like image classification. In Sec.4, we aim to demonstrate quality described by prompts should resemble image quality, so the classification dataset CIFAR10 is introduced for low-quality images.

According to your advice, we provide additional results on classification in GQ4.

Q9: How are clean features obtained from noisy features?

A9:

• Diffusion models learn from pre-training to reconstruct clean representations from noisy inputs. Notably, this is also related to content shift as noisy inputs cause information loss.
• Shortcut structures in UNet pass high-frequency details to upsampling stage, helping reconstruction.

Q10: Limitations are too brief.

A10: We focus on content shift and its solution. Due to space limit, we omit many implementation details, which will be revised. Besides, effectiveness of GATE relies on selected techniques, for which we propose qualitative evaluation guideline. Though we select three effective techniques and report fail cases, there still is more to explore. Furthermore, we only experiment on three tasks, so the full potential of GATE might remain under-explored.

(Part 2 of 4)

4.2 Cause of Content Shift

After the existence of content shift is confirmed in diffusion features, we next aim to find its cause. Unlike more conventional feature extractors such as ResNet \cite{he2016deep}, the inputs to diffusion models are not the original image (Figure 4(a)), but its noisy version (Figure 4(b)), as enforced by the diffusion process \cite{ho2020denoising}. The early layers of diffusion models even further add more noise to the inner representations (Figure 4(c)). However, the diffusion UNet gains the ability from vast pre-training to reconstruct clean inner representations from noisy inputs (Figure 4(d)), roughly at the middle of the UNet structure. Additionally, the shortcut structures in UNet also help the reconstruction by passing some high-frequency details. Afterward, the diffusion UNet will further predict noises based on the reconstructed representations (Figure (e)).

Notably, the diffusion features we are using are in fact the reconstructed representations, which answers why clean diffusion features can be obtained even though the inputs are noisy. Despite the reconstruction ability, however, many high-frequency details are potentially blurred out by input noises, and thus their reconstruction is mostly based on "imagination". This leads to possible drift from the original image during reconstruction \cite{Daras23consistent}. Naturally, the content shift phenomenon in extracted diffusion features reflects the drift during reconstruction. Consequently, content shift is an inherent characteristic of diffusion models and diffusion features, which suggests its broad existence across models and timesteps.

---

5.2 Quantitative Evaluation

We also propose a quantitative metric for the evaluation of generation techniques. We set feature extracted at $t=0$ as reference $FEAT_{ref} \in \mathbb{R}^{c\times h\times w}$ as it is less affected by noises. We use the Laplacian operator to evaluate the contour difference between feature $FEAT$ and the reference:

$$diff=\sum_{i,j}^{h\times w}|\left\|Laplacian(FEAT_{ref},i,j)\right\|_2-\left\|Laplacian(FEAT,i,j)\right\|_2|\in(-\infty,1]$$

We then set a feature with stronger content shift as an anchor and compare $diff_{anchor}$ with other features.

$$Score=\frac{(diff_{anchor}-diff)}{diff_{anchor}}$$

$Score=1$ means an exact match, and a smaller value indicates more shift. In this way, we can measure the extent of content shift in extracted features using different generation techniques and thus evaluate the suppression effect of techniques. Noticeably, the evaluation of this quantitative metric can be well approximated by the previously proposed qualitative evaluation, so we recommend the qualitative evaluation if efficiency is desired.

(Part 1 of 4)

In Introduction line 41-51

In pursuit of a method to suppress content shift, we need to further investigate why this phenomenon exists. We notice in Section 4 that the diffusion backbone reconstructs clean inner representations from noisy inputs in the middle of UNet before predicting noises based on the reconstructed content. The diffusion features we are using are in fact the reconstructed representations, which answers why we can obtain clean features from noisy images. However, since high-frequency details are potentially blurred out by noises in the inputs and recovered by "imagination" gained from vast pre-training, this reconstruction process inherently suffers from the risk of drifting from the original image. Content shift in diffusion features, naturally, reflects the drift during reconstruction. Consequently, to suppress content shift, we need an additional way to directly send the original clean image into the middle of UNet and steer the reconstruction towards the original image. To our delight, we notice that many off-the-shelf image generation techniques for diffusion models \cite{zhang2023adding, mou2023t2i, ye2023ip} also work by sending additional information into UNet and thus steering the reconstruction. Hence, it is possible to select some techniques that can directly satisfy the goal of suppressing content shift, which eases the implementation and extension of our method. Although the huge number of generation techniques might pose difficulty to the selection, we propose a guideline in Section 5 to bridge the gap. This method is denoted as GenerAtion Techniques Enhanced (GATE) diffusion feature and its effect is also shown in Figure 1.

---

3 Preliminaries: Diffusion Feature

Diffusion models consist of a neural network module and a diffusion scheduler. The network is an end-to-end network, which can be formally denoted as $\epsilon_\theta$, where $\theta$ is the parameters. The diffusion scheduler is the core of diffusion models. While previous generative models such as GAN expect to generate an image directly in one step, the diffusion scheduler turns the generation process into a timeline comprised of many timesteps. Generally, we use a smaller / larger timestep to indicate less / more noises, and $t=0$ / $t=T$ for clean images / total noises.

Next, we will explain how convolutional features and attention features are extracted using a common pipeline for diffusion features, with the visual illustration in Figure 2. Given an input image $x\in\mathbb{R}^{3\times h \times w}$, where $h, w$ are height and width, the extraction process has the following steps: (i) A pre-trained VAE encodes the input image into the latent space, inducing $x_0\in\mathbb{R}^{4\times h' \times w'}$, as a common practice presented in \cite{rombach2022high, podell2023sdxl}. (ii) $x_0$ acts as the input of the forward diffusion process, i.e., timestep $t=0$. As suggested in \cite{baranchuk2021label}, it is beneficial to extract diffusion features at non-zero timestep. Following this practice, we set $t=50$ and get $x_t$. (iii) $x_t$, along with the timestep $t$, a textual prompt $c$, is sent into the pre-trained diffusion UNet $\epsilon_\theta$, i.e., $\epsilon_\theta(x_t,t,c)$. The generation techniques selected by our method are also applied here to modify $\theta, c$. (iv) Convolutional and attention features are gathered during the computation of the backbone. For convolutional features, we gather the output activations of each resolution in the upsampling stage of the UNet. For attention features, we obtain the mean value of the similarity maps between query and key in all cross-attention layers.

Thanks for your constructive comments, and we would like to make the following response.

Q1: Hence, it is recommended to provide more failure cases to validate the negative impact of the content shift.

A1: We have included another failure case, IP-Adapter, in the rebuttal. For details, please refer to the global response GQ2.

Q2: To be specific, which results are implemented by the authors or borrowed from previous works could be more explicitly stated. Besides, some results in Table 1 and Table 2 are missing. The authors should make a further explanation.

A2: Thanks for your patience and advice. We will refine these details further and make some explanation here. In Table 1, Baseline for semantic correspondence and GATE for both tasks are implemented by us, and all the other results are borrowed from previous studies [1-4, 12, 23, 26, 30, 36, 39, 53]. In Table 2, MaskCLIP and ODISE results are borrowed from previous studies [8, 45]. As they do not provide results on CityScapes and it is hard to extend the codes on this dataset, we do not have CityScapes results for the two methods. Both our GATE and VPD are implemented by us. Specifically, the original results reported by VPD are based on full-scale fine-tuning of diffusion UNet, which is not fair as we do not train the diffusion model in our setting. Therefore, we re-evaluated VPD with the diffusion UNet frozen and reported our results.

Q3: Besides, some typos and minor errors found in the paper.

A3: Thanks for your patience. We will pay attention to these typos and errors during refinement.

Thanks for your constructive comments, and we would like to make the following response.

Q1: I wonder how to design these prompts. Does it require us to design prompts for each input?

A1: We observe the common characteristics of the training images and design a general prompt for all images. In this way, we do not need to design prompts for each input nor see the images of the testing set. More advanced methods can also be utilized, such as a pre-trained image captioner. We do not use image captioners in the manuscript, but some additional results taking this approach are listed in the global response GQ3.

Q2: The paper focuses on empirical validation and lacks a thorough theoretical analysis of why and how the proposed techniques effectively suppress content shift. It is recommended to provide more insights or explanations.

A2: Thanks for this constructive suggestion! We focus this paper on enhancing the actual performance and thus leave thorough theoretical analysis to future work. Nevertheless, we have made some preliminary discoveries. As shown in Fig.4, from pre-training, diffusion models obtain the ability to reconstruct clean inner latent representations from noisy inputs. However, noises may make some details blurry, thus making the reconstruction drift from the input image, which leads to content shift in diffusion features. The three selected techniques can suppress content shift due to two factors. On one hand, they introduce additional information into this reconstruction process. On the other hand, the techniques are designed to steer the reconstruction towards the introduced information. The two factors combined together can guide the reconstructed latent representation towards the input image, thus suppressing content shift.

Thanks for your constructive comments, and we would like to make the following response.

Q1: Please provide a more detailed explanation and additional diagrams to clarify how attention features are extracted from the diffusion model in Fig.1 and Fig.2.

A1: Modern diffusion models use cross-attention between the latent image and prompts to enable user interaction. We extract the similarity scores between query (latent image) and key (prompt) and average them over all layers as attention features. For more details, please refer to the global response GQ1 and the attached PDF.

Q2: Please provide details of the overall process of the proposed method.

A2: Perhaps due to the omitted details of Extract Features with Techniques in Fig.7, the overall pipeline is a bit confusing. In fact, this step follows the practice in the prior arts [2]. We first encode an input image with VAE, then add noises to it, and finally send the image into diffusion UNet to extract features.

For more details, please refer to the global response GQ1 and the attached PDF.

Q3: A broader range of techniques should be evaluated to strengthen the claim of its general applicability.

A3: We have also evaluated two techniques, classifier-free guidance and IP-Adapter. As the results presented in Appendix and global response GQ2 show, the two techniques fail to suppress content shift and do not achieve significant performance gain. These results again strengthen the general applicability of our GATE guideline.

Q4: More detailed and extensive comparative analyses with additional evaluation metrics and qualitative evaluation are needed to show the advantages of the proposed method.

A4: According to your suggestion, we have provided more comparative analyses with additional metrics, including mIoU, aAcc, and mAcc. Please refer to the global response GQ4 for more details.

For more qualitative results, besides the visualization in Fig.10, we also include additional feature visualization in the PDF attached to the global response. From the visualization, we can observe that:

i) The attention features become clearer and closer to the input image when more generation techniques are applied according to GATE, showing the suppression effect on content shift. ii) Notably, for the second image where a person is riding a horse, the baseline attention feature fails to follow the instruction, i.e., attending only to the horse and ignoring the person. In contrast, generation techniques applied according to GATE help attention features attend to the correct object. iii) From convolutional features, we can see the application of generation techniques brings stronger diversity.

Q5: How does the proposed method perform on tasks involving different types of images or datasets with varying levels of noise and complexity?

A5: Among the three generation techniques, only fine-grained prompts can be sensitive to image types and noise levels. For simplicity, we tackle this challenge via general prompts, whilst a pre-trained image captioner is also an option.

For more details, please refer to the global response GQ3.

Dear Reviewer z9si,

As the rebuttal deadline approaches, could you please let us know if our newly added response has addressed your concerns? Thank you once again for your time and valuable feedback!