Diffusion Models are Few-shot Learners for Dense Vision Tasks

We appreciate the reviewers for their valuable feedback.

---

Q1: Noise added to the input

We found that adding noise helps extract better representations from the pre-trained diffusion model, as demonstrated in [1]. In [1], features extracted at $t=234$ encode better visual correspondences compared to those at lower noise levels (e.g., $t=1$). During training, we randomly sampled $t$ in the $[5, 200]$ range as an augmentation strategy to prevent overfitting.

---

Q2: Risk of overfitting

Although we only used 10 or 20 training examples, we did not observe significant overfitting. This is partly because we applied several augmentation strategies, such as adding noise at different levels. Additionally, the number of parameters we fine-tuned is relatively small. Furthermore, each training example includes pixel-level annotations, where every pixel has a corresponding ground truth label. This means the number of annotations is much larger compared to datasets where each image has a single class label.

However, we did notice overfitting when the codebook size was too large, so we carefully controlled the codebook size. There is still some degree of overfitting, as increasing the number of training samples tends to improve validation performance even when the training loss remains similar. Nonetheless, the overfitting is not severe, especially compared to other methods.

---

Q3: Clarification regarding Figure 3

The vertical axis of Figure 3 represents the evaluation metric computed on the validation set. We compared the impact of using classification loss during training on validation metrics. Since the evaluation metric for continuous dense tasks (RMSE) coincidentally matches the regression loss, referring to it as "regression loss on the validation set" caused ambiguity. This has been clarified in the revised version.

---

W1: Explanation of the rows in Figure 2

Each row in Figure 2 contains two tasks. For each task, there are five images from left to right: the input, ground truth, VTM prediction, VPD prediction, and our method's prediction.

---

[1]. Emergent Correspondence from Image Diffusion
Luming Tang*, Menglin Jia*, Qianqian Wang*, Cheng Perng Phoo, Bharath Hariharan

We thank the reviewer for the feedback.

---

W1, W2: Issues with Presentation

We apologize for the inconvenience caused by the presentation. In the revised version, we have made extensive changes to the methodology section. We corrected the notations to ensure consistency and clarity, and refined the descriptions of operations to make the logic flow more coherent, improving the overall readability of the paper.

Specifically, in our method, we first discretize the D-dimensional continuous ground truth values of continuous dense tasks into N categories. Then, we optimize N concept embeddings and combine the cross-attention and self-attention maps to obtain a probability distribution over N categories for each pixel in the input image.

For continuous-output dense tasks, we further map the category labels back to continuous values. Since this label-to-value mapping is not necessarily unique and may vary with different inputs, we model the label-to-value mapping as a random variable and learn a codebook containing K different label-to-value mappings. Each label-to-value mapping has a size of N × D, representing the D-dimensional values corresponding to N categories.

We then combine the K mappings based on the CLIP features of the input image to obtain an input-dependent label-to-value mapping of size N × D. Finally, we combine this input-dependent mapping with the previously learned pixel-wise probability distribution to compute the output value for each pixel.

---

W3: Novelty of Our Method

Unlike prior works that adapt diffusion models by prompting without altering their original structure, our work focuses on enabling diffusion models to few-shot adapt to general dense vision tasks.

First, most previous works on adapting diffusion models focus on tasks like image synthesis or image editing. Works that adapt diffusion models for perception often target specific tasks such as semantic segmentation. However, the question of whether diffusion models can adapt to various dense prediction tasks has received relatively little attention, with VPD being a notable exception. Yet, methods like VPD require a large amount of labeled data for each downstream task during adaptation.

To our knowledge, no prior work has investigated whether diffusion models can be adapted to general dense prediction tasks using only few-shot examples. This is particularly challenging due to the significant differences between dense prediction tasks—depth estimation requires understanding geometric information, while semantic segmentation focuses on semantic and grounding information.

Our work is the first to explore how to leverage the diffusion prior for few-shot adaptation to general dense vision tasks. Dense tasks are a crucial category of vision problems, and a sufficiently general vision model should be able to handle diverse dense tasks. Furthermore, as new task definitions continue to emerge, it is impractical to annotate every potential task exhaustively. Therefore, the ability to adapt to entirely new tasks, unseen during training, using only a few-shot examples, becomes essential.

---

W4, W5: Typos in the Paper

We sincerely apologize for the incomplete sentences, unclear phrasing, and grammatical errors in the paper. In the revised version, we have carefully corrected all these typos and thoroughly reviewed the manuscript to ensure clarity and accuracy.

I would like to thank the authors for their response. However, the revised manuscript still contains critical issues and cannot meet the standard of ICLR.

• Plagiarism Issue: As noted by Reviewer 6p3N, the paragraph Background on Diffusion (L172-L185) in this paper is a direct copy-paste from a recent paper [1]. This constitutes plagiarism, which severely undermines the credibility of the work. Regrettably, the authors have failed to respond to this issue and have left the plagiarized paragraph unchanged in the revised manuscript.

• Poor Presentation: Regarding the presentation quality (also mentioned by Reviewer 6p3N), I do not observe significant improvements in the revised manuscript. The clarity and conciseness still do not meet the high standards expected at ICLR. I strongly recommend that the authors carefully revise the manuscript to enhance its readability and ensure that the content is presented in a more structured and clear manner.

• Duplicated Sections: Sec 3.1 (L189-L199) and Sec 3.2 (L203-L215) contain 100% duplicated content with the same heading "Reframing Dense Vision Tasks as Classification".

• Limited Novelty: In their response, the authors claim their novelty solely on the new task setup -- adapting diffusion models in few-shot manner to dense tasks. Technical contribution to the proposed few-shot generalization to dense tasks is NOT well justified.

---

Given the issues outlined above, I believe that this manuscript is NOT yet ready for publication, particularly at a top-tier conference like ICLR.

[1] Fan, Xiang, Anand Bhattad, and Ranjay Krishna. "Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion." arXiv preprint arXiv:2403.14617 (2024).

Dear Reviewers,

This is a kind reminder that the discussion phase will end tomorrow. If you have any remaining concerns or questions, please feel free to share them. We welcome further discussion and truly appreciate your efforts!

Thank you!

Overall, we believe this reviewer might be overreacting slightly.

---

First, we would like to point out that the description of diffusion models in the background section is not the main contribution of this paper, and similar descriptions of diffusion models can be found in many other works. For example, the following passage from Diffusion Self-Guidance for Controllable Image Generation (NeurIPS 2023):

"Diffusion models learn to transform random noise into high-resolution images through a sequential sampling process. This sampling process aims to reverse a fixed time-dependent destructive process that corrupts data by adding noise. The learned component of a diffusion model is a neural network $\epsilon_\theta$ that tries to estimate the denoised image... A common choice for $\epsilon_\theta$ is a U-Net architecture with self- and cross-attention at multiple resolutions to attend to conditioning text in $y$."

And the following from Generative Powers of Ten (CVPR 2024):

"Diffusion models generate images from random noise through a sequential sampling process. This sampling process reverses a destructive process that gradually adds Gaussian noise on a clean image $x$... A diffusion model is a neural network $\epsilon_\theta$ that predicts the approximate clean image $\hat{x}$ directly, or equivalently the added noise $\epsilon_t$ in $z_t$... A standard choice for $\epsilon_\theta$ is a U-Net with self-attention and cross-attention operations attending to the conditioning $y$."

Both contain descriptions of diffusion models that are very similar to ours. We believe that the description in the background section does not detract from the uniqueness of our main contribution.

Furthermore, we have informed the authors of VideoShop about this concern raised by the reviewers. The authors of VideoShop do not consider this to be plagiarism and have stated that the references to diffusion models in our paper are entirely appropriate. They do not regard our work as infringing on or misappropriating theirs in any way. The reviewers are welcome to directly reach out to the authors of VideoShop, who can confirm this in a manner that preserves the double-blind review process.

---

Additionally, only this reviewer has consistently raised concerns about presentation issues. None of the other four reviewers seem to have encountered difficulty understanding our paper. Below are excerpts from their reviews:

• Reviewer 6yKP: "The paper is generally well-written, and addresses an important task with growing interest in the field (how to use diffusion models for discriminative tasks). The figures are both easy to understand and aesthetically pleasing."
• Reviewer bVz2: Presentation: 3: good.
• Reviewer u6mW: Presentation: 3: good.
• Even Reviewer 6p3N, who provided substantial suggestions for improvement, stated that "The paper is mostly easy to read. The authors provide a very clear visualization of their approach in Figure 1, which nicely accompanies their explanations."

It appears that only Reviewer 66Ct feels that our presentation needs comprehensive revision.

---

The repeated section "Reframing Dense Vision Tasks as Classification" in the revised version was simply a typo, which can easily be corrected in the camera-ready version.

---

Regarding the concerns about novelty, the official guidelines on strengths for papers are very clear:

"We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results."

Novelty based on the new task setup as an application to a new domain should not be seen as diminishing the paper's overall novelty.

We thank the reviewer for the feedback.

---

Q1: Clarification on "input" in the method

Apologies for the confusion in our description. Our paper has two parts. The first focuses on a scientific discovery to demonstrate the potential of diffusion models for few-shot dense tasks. In this part, we only optimize the $N$ Concept Embeddings without modifying the diffusion model’s core UNet, which remains fixed across tasks. The optimization focuses solely on the computational graph's leaf nodes, and we show that differences between dense tasks can be identified through changes in the model's input. In the second part, we additionally optimize the linear layer and codebook embeddings to further improve performance on few-shot dense tasks. These two components are not input variables.

Thus, our statement should be "we find that optimizing only the input variables is sufficient to unleash the potential of pre-trained diffusion models on few-shot dense tasks." We have comprehensively revised this statement in the updated version to avoid misunderstandings.

---

Q2: Claim about VTM's reliance on dense annotations

VTM’s setup involves pre-training on multiple dense tasks (e.g., using 8 dense tasks out of 10) with meta-learning and then fine-tuning on few-shot samples from the remaining two unseen tasks. The pre-training phase requires extensive dense annotations across multiple tasks, which limits its scalability. In contrast, Stable Diffusion is pre-trained on datasets like LAION, where each image is paired with a text annotation, making it far easier to scale the dataset size. We have further clarified this distinction in the revised version.

---

Q3: Comparison with AdaBins

Our method, like AdaBins, involves converting continuous values into discrete buckets. However, our primary motivation lies in exploring the potential of diffusion models to adapt to new dense tasks using few-shot samples. Additionally, we optimize concept embeddings as input variables, whereas AdaBins integrates them as part of the task-specific head. In the revised version, we added a section in the future direction explaining why adapting through input variable optimization aligns better with in-context adaptation. The introduction has also been updated to further highlight the differences between our approach and AdaBins.

---

Q4: Comparison with DINOv2

In recent experiments, we found that the internal feature quality of DiT architectures (e.g., SD3) is inferior to UNet architectures (e.g., SD2), and this area warrants further exploration.

Regarding DepthPro, its notion of zero-shot adaptation refers to adapting to unseen depth scales rather than entirely new tasks. We believe there are two distinct types of adaptation: adaptation to new domains and adaptation to new tasks. DepthPro falls into the former category, as it leverages DINOv2, which is trained on substantial depth estimation data, to adapt well to different depth estimation domains. However, it struggles with entirely different tasks, such as surface normal prediction.

Our focus is on adapting to entirely new tasks without any prior exposure to such task data. As for the comparison between DINOv2 and diffusion models in adapting to new tasks, [1] indicates that diffusion models achieve comparable adaptation performance with fewer data (reduced by orders of magnitude). However, DINOv2 has a higher ceiling when adapting with more task-specific data.

---

Q5: Definition of K, N, and D in the codebook

We clarified these definitions in the revised manuscript. D represents the dimensions of the continuous output, and N refers to the N discrete categories resulting from discretization, where each category corresponds to a D-dimensional value. Each label-to-value mapping has a size of N × D. We assume the label-to-value mapping is a random variable with K possible values, resulting in K sets of label-to-value mappings.

---

Q6: Adaptation with a full training set

This involves two questions. First, how much data is required for adapting to new tasks? Our work focuses on adapting to new tasks with few-shot examples, as we aim to enable a vision system adapt quickly to new tasks with minimal data like humans. This is critical because new task definitions continue to emerge, and it is impossible to annotate every potential task exhaustively.

Second, how do we evaluate adaptation performance? This can be done on in-domain data or out-of-domain data. While out-of-domain evaluation primarily measures adaptation to new domains, our focus is on adaptation to new tasks, so we evaluate performance on in-domain data. However, in Taskonomy, the adaptation and evaluation data come from different indoor buildings, which can also be considered out-of-domain evaluation to some extent.

---

[1] GeoBench: Benchmarking and Analyzing Monocular Geometry Estimation Models, Ge et al.

Dear Reviewers,

This is a kind reminder that the discussion phase will end tomorrow. If you have any remaining concerns or questions, please feel free to share them. We welcome further discussion and truly appreciate your efforts!

Thank you!

We thank the reviewer for the feedback.

---

Q1: Discretizing continuous output space

In this paper, our primary focus is on demonstrating the potential of pre-trained diffusion models to adapt to entirely new, unseen dense prediction tasks. Discretizing continuous spaces is key to unlocking this potential. We have further highlighted this point in the revised version's introduction and discussed the differences with AdaBins [1] in more detail. More importantly, unlike AdaBins, which incorporates concept embeddings as part of the final output head, we optimize these concept embeddings as part of the model’s input. In the revised version's discussion of future work, we explain why we prioritize modifying input parameters for adaptation. This enables task differences to be defined through inputs, which is closer to in-context learning and eliminates the need to load separate model weights for different tasks.

---

Q2: Performance under different numbers of training samples

Thank you for the suggestion. We tested our method on all 12 tasks using 10, 20, 50, and 100 training examples and compared the performance with VPD. Results are visualized in Figure 4 of the revised version. The findings demonstrate that our method consistently outperforms VPD when fewer than 100 training examples are used, showcasing that our approach better unlocks the potential of diffusion models for dense prediction tasks.

---

Q3: Out-of-place sentence

We sincerely apologize for the typo and any confusion it caused. We have removed the sentence in the revised version and thoroughly reviewed the manuscript to ensure no other unrelated or erroneous typos remain.

---

Q4: Mistake in Table 3 caption

Apologies for the confusion. It should have been "cross-attention mask only." This has been corrected in the revised version. However, having only the self-attention mask (Msa) is not feasible because the number of channels in the cross-attention mask (Mca) directly corresponds to the number of bins obtained by discretizing the continuous output, while the channel count of Msa always equals $H \times W$. Therefore, Mca is necessary.

---

Q5: The impact of Msa on discrete dense tasks

We evaluated the impact of using Msa on semantic segmentation in Taskonomy, with results as follows:

The results indicate that self-attention is also beneficial for discrete tasks.

---

Q6: Clarification regarding Figure 3

The vertical axis of Figure 3 represents the evaluation metric computed on the validation set. We compared the impact of using classification loss during training on validation metrics. Since the evaluation metric for continuous dense tasks (RMSE) coincidentally matches the regression loss, referring to it as "regression loss on the validation set" caused ambiguity. This has been clarified in the revised version.

---

Q7: Insights and limitations

We have added a discussion of limitations and future directions in the revised version.

We aim to enable adaptation via in-context learning, where no parameter updates are needed—models learn tasks from few examples. While challenging, our work moves toward this by modifying only input parameters during adaptation, which aligns task differences with inputs rather than heads or internal parameters.

Our findings show diffusion models can adapt with minimal parameter changes, supporting their use as general-purpose dense decoders. Future directions include feedforward mapping from few-shot examples to task vectors and combining diffusion with autoregressive models.

---

Q8: Typos and grammatical mistakes

We sincerely apologize for the inconvenience caused. The manuscript has been thoroughly revised to eliminate all typos and grammatical errors.

---

Q9: Differences between continuous and discrete tasks

Due to differences in output formats, it is indeed difficult to make operations for continuous and discrete tasks completely identical. However, we have ensured that their model structures are nearly identical, especially when not using a codebook for label-to-value mapping. By "custom decoder," we refer to designing different structures for different tasks, not only due to output format differences but also because different tasks require different priors. In contrast, we use the same diffusion prior for all tasks, eliminating the need for custom structural designs.

---

Q10: Adding arrows in tables

Thank you for the suggestion. We have added arrows to each table in the revised version.

---

Weakness: Writing quality

We apologize for the issues in presentation quality. Many sections have been rewritten and revised in the revised version to make statements more precise and the paper easier to follow.

---

[1] AdaBins: Depth Estimation using Adaptive Bins
Shariq Farooq Bhat, Ibraheem Alhashim, Peter Wonka

Dear Reviewers,

This is a kind reminder that the discussion phase will end tomorrow. If you have any remaining concerns or questions, please feel free to share them. We welcome further discussion and truly appreciate your efforts!

Thank you!

I'd like to thank the authors for their response and the clarifications.
Some of my concerns have been addressed, but there are still quite a few remaining.

E.g. Re Q6: "Figure 3 & corresponding results: The authors state they calculate the ‘regression loss’ on the validation set, which drops faster when using regression+classification loss than it does with regression-only; -> However, this raises the question whether the two have been using the same hyperparameters or have been independently ‘optimised’; -> Note: When adding the ‘additional’ classification loss, the loss will generally likely be larger, and hence gradients will likely be larger as well; This might by itself cause a faster convergence, which is mainly caused by the magnitude and not necessarily the nature of the losses; I’d suggest the authors take a look at their gradient and loss magnitudes, and check if the same holds when the learning rate is adjusted accordingly (or losses are scaled accordingly)
$\textrightarrow$ Note that your current answer does unfortunately not address the underlying question I raised here, which essentially results in: "Is the conclusion of faster convergence even valid? "
$\textrightarrow$ Please correct me if you think there is a misunderstanding here!

Re Q1 (minor): My intent of raising this was more to draw attention to the fact the many works in the dense prediction space have used a similar technique, and research has been conducted there from which some useful lessons/inspiration could be drawn (e.g. how to discretize, etc). For depth estimation, this might included works like Deep Ordinal Regression Network for Monocular Depth Estimation by Fu et al. (CVPR2018), etc;
$\textrightarrow$ My intention was to simply raise the awareness that such works exist, and that the strategy followed in Sec 3.1 might just be 'one choice', but not necessarily the best one. (In this sense, including some references might be helpful for the reader who might want to follow up)

---

Re Quality of the manuscript: I appreciate the authors' efforts in reformulating and rewriting sections, and the quality has been improved. However, there are still various aspects that don't 'live up' to the standards of a well-formulated manuscript yet, including for instance:

• Identical sentence repeated within 2 paragraphs: Sec 3.1 & Sec 3.2 start with the exact same sentence, word-by-word.
• Using "metric" as the y-axis label in Fig 4., in addition to no indication which sub-figure represents which task;
• ...

EDIT: Addition after 66Ct's rebuttal answer:
I also went back to compare the previously mentioned work by Fan et al. side-by-side to this manuscript in terms of the background section on latent diffusion models, and have to agree with reviewer 66Ct: Although you have changed some variable names, the remainder is almost 'verbatim' copied from their work -- which is not acceptable, and will require rewriting of these parts!

---

I do appreciate the work the authors have put into their rebuttal, but given the previously mentioned aspects and serious concerns, the manuscript still doesn't pass the bar of acceptance for me.
I therefore still have to recommend 'rejection', but would like to encourage the authors to take the constructive reviews into consideration and further refine/rewrite their work, as there are valuable aspects to it that are of interest to the community.

We sincerely appreciate the reviewer’s valuable feedback and the effort put into reviewing our paper. We believe the issues raised by the reviewer are highly detailed ones, mostly related to improving the precision of captions or ensuring typos are entirely eliminated. These are issues that can be quickly and easily resolved in the camera-ready version. Below, we provide our responses to these points:

---

First, we would like to point out that the description of diffusion models in the background section is not the main contribution of this paper, and similar descriptions of diffusion models can be found in many other works. For example, the following passage from Diffusion Self-Guidance for Controllable Image Generation (NeurIPS 2023):

"Diffusion models learn to transform random noise into high-resolution images through a sequential sampling process. This sampling process aims to reverse a fixed time-dependent destructive process that corrupts data by adding noise. The learned component of a diffusion model is a neural network $\epsilon_\theta$ that tries to estimate the denoised image... A common choice for $\epsilon_\theta$ is a U-Net architecture with self- and cross-attention at multiple resolutions to attend to conditioning text in $y$."

And the following from Generative Powers of Ten (CVPR 2024):

"Diffusion models generate images from random noise through a sequential sampling process. This sampling process reverses a destructive process that gradually adds Gaussian noise on a clean image $x$... A diffusion model is a neural network $\epsilon_\theta$ that predicts the approximate clean image $\hat{x}$ directly, or equivalently the added noise $\epsilon_t$ in $z_t$... A standard choice for $\epsilon_\theta$ is a U-Net with self-attention and cross-attention operations attending to the conditioning $y$."

Both contain descriptions of diffusion models that are very similar to ours. We believe that the description in the background section does not detract from the uniqueness of our main contribution.

Furthermore, we have informed the authors of VideoShop about this concern raised by the reviewers. The authors of VideoShop do not consider this to be plagiarism and have stated that the references to diffusion models in our paper are entirely appropriate. They do not regard our work as infringing on or misappropriating theirs in any way. The reviewers are welcome to directly reach out to the authors of VideoShop, who can confirm this in a manner that preserves the double-blind review process.

---

Q1: Clarification regarding "faster convergence" in Figure 3 caption
It is true that the scales of the two optimized losses are different, but we would like to emphasize that the comparison is based on the RMSE metric on the validation set, i.e., validation performance. Validation performance is generally considered independent of the loss scale. Therefore, even with different loss scales, if the validation performance improves more quickly, describing it as "faster convergence" is not unreasonable.

That said, we understand the reviewer’s concern for more precise phrasing. We will revise "faster convergence" to "Validation performance improves faster and achieves higher final validation performance." We believe this simple adjustment to the caption will address any concerns regarding the interpretation of Figure 3.

---

Q2: Missing reference
Apologies for overlooking the paper Deep Ordinal Regression Network for Monocular Depth Estimation during the rebuttal phase. We acknowledge that many prior works have explored how to discretize for dense tasks, and these approaches could be complementary to our work and potentially integrated in a plug-and-play manner. We do not claim that our method of discretization is optimal, and we believe this is an area worth further exploration. We will include citations to these works in the "Future Directions" section and welcome additional suggestions for related references.

---

Q3: Typos
We sincerely apologize for the newly identified typos. Due to the extensive revisions and additional experiments required for the manuscript, the limited time available has made it challenging to catch all such errors. We hope for your understanding.

1. Section duplication: Sections 3.1 and 3.2 are completely repeated. We will remove one of them.
2. Figure 4 improvements: We will label the specific metric names and task names in Figure 4. Currently, the tasks in Figure 4 are as follows:
   • First row (left to right): EucDepth, Z-depth, 2DEdge, 3DEdge.
   • Second row (left to right): 2DKeypoint, 3DKeypoint, Reshading, Curvature.
   • Third row (left to right): Normal, SemSeg, NYUDepth, NYUNormal.

We will ensure the captions in the figure are more complete and precise.

We thank the reviewer for the feedback.

---

W1, Q1: Clarification on Performance Improvements and SD2.1 Backbone

Although the original VPD used SD1.5, in this paper's experiments, we compare with VPD using SD2.1 as the backbone. All VPD experimental results were rerun in our setting using SD2.1. We have clarified this in the revised version.

---

W2: Importance of Few-Shot Adaptation

We believe that each method has its own applicable scope. For example, VPD underperforms even VTM in the few-shot setting, indicating that VPD fails to fully exploit the potential of pre-trained diffusion models. One of our main contributions is exploring whether pre-trained diffusion models can effectively handle few-shot dense prediction tasks.

Why is few-shot adaptation important? First, we want to highlight that large datasets have two primary uses: (1) for pre-training general vision models that can adapt to different tasks, especially unseen ones, and (2) for learning task-specific models that are tailored to fixed tasks. Our primary goal is the first: exploring whether a general vision model can adapt to entirely unseen dense tasks using only a few examples, akin to human learning. Stable Diffusion itself was pre-trained on large-scale image-text datasets before adapting to dense tasks, but the adaptation stage relies on few-shot examples.

Few-shot adaptation is crucial because new task definitions continue to emerge. Regardless of how large vision datasets become, it is impossible to exhaustively define every potential task. Additionally, annotating large datasets for every new task is impractical. We envision an AI system with lifelong learning capabilities that can use minimal examples to master entirely new tasks.

We also fine-tuned with varying numbers of samples across all 12 tasks and observed that when fewer than 100 samples are used, VPD cannot adapt to new dense prediction tasks as effectively as our method. Due to time constraints, we could not precisely identify the threshold where VPD starts to outperform, but at least in the few-shot setting, our method is superior to VPD. Visualized results are provided in the revised version (Fig. 4).

---

Q2: Clarification Regarding Figure 3

The vertical axis of Figure 3 represents the evaluation metric computed on the validation set. We compared the impact of using classification loss during training on validation metrics. Since the evaluation metric for continuous dense tasks (RMSE) coincidentally matches the regression loss, referring to it as "regression loss on the validation set" caused ambiguity. We have clarified this in the revised version.

---

Q3: Role of Different Components

We agree that the classification transformation is the most critical aspect. Our primary goal is to demonstrate the ability of pre-trained diffusion models to perform few-shot dense prediction tasks and to use the codebook to further enhance performance. By reframing dense vision tasks as classification problems, we show that diffusion models can effectively adapt to unseen dense tasks.

---

Q4: VPD with More Parameters Frozen

We compared VPD adapted using LoRA on two NYU datasets but found it did not outperform our method. We set the rank to 4, and the results are shown below:

Furthermore, we emphasize that we aim for adaptation without modifying internal model parameters. Instead, we achieve this by modifying input variables, which facilitates deployment.

---

Q5: Why Optimizing Inputs Facilitates Deployment

Adapting to different tasks by modifying internal model parameters requires loading a separate model for each task—even when using methods like LoRA. In contrast, modifying only the input allows a fixed black-box model to be shared across tasks without requiring separate models. This is a key advantage of prompt tuning, as demonstrated in both NLP [1] and vision [2] tasks.

---

Typo in Line 451: “We can see We can observe that both.”

Thank you for pointing this out. We have corrected it in the revised version.

---

[1]. Prefix-Tuning: Optimizing Continuous Prompts for Generation
Xiang Lisa Li, Percy Liang

[2]. Visual Prompt Tuning
Menglin Jia*, Luming Tang*, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim