ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generation

Thank you for your positive and constructive feedback. We appreciate your recognition of the clarity and coherence of our method, the effectiveness of our reward-adaptive sampling strategy, and the strength of our visual and experimental results. We address your comments in detail below:

(1) Difference with motion-control approaches.

As “motion control approaches” may refer to a broad range of domains (e.g., camera trajectory control, object animation), we would appreciate it if the reviewer could kindly clarify which specific line of work is being referenced. We would be glad to address the comparison more directly. In general, our method focuses on grounding the static 3D object orientations during image generation from scratch, rather than manipulating their motions over time. As such, we believe our problem formulation and methodology are distinct from works that address temporal motion control.

(2) Scope and stability of zero-shot approaches.

Scope. Note that spatial grounding in image generation is one of the most active research fields [1, 2, 3, R1, R2]. While the existing spatial grounding methods were limited to 2D position grounding, ORIGEN makes a significant contribution by extending the grounding modality to 3D orientation, which has only been explored in limited forms in prior works.

Zero-shot approaches. We respectfully disagree with the claim that zero-shot methods are generally less stable than training-based methods. As discussed in the paper, training or fine-tuning a generative model conditioned on object orientation is non-trivial due to the lack of a large-scale dataset containing (prompt, image, orientation) samples. Consequently, existing fine-tuning-based approaches such as [18] exhibit clear limitations in both orientation controllability and realism, as evidenced by our comparisons in Tables 1 and 6 in the paper. In contrast, our method does not require training a base model and can fully leverage the rich prior knowledge embedded in large pre-trained text-to-image models. This allows for the generation of high-quality images across diverse prompts, object categories, and orientations, making our approach both practical and effective.

(3) More comparative baseline methods.

Following your suggestion, we performed additional baseline comparisons with state-of-the-art multi-view diffusion models more recent than Zero-1-to-3 — SV3D (ECCV 2024) [R3] and MV-Adapter (ICCV 2025) [R4] — on the ORIBENCH-Single benchmark. As shown in the table below, our method outperforms all these additional baselines, demonstrating the effectiveness of the proposed Reward-Guided Langevin Dynamics and Reward-Adaptive Time Rescaling. We thank your suggestion and will include these additional results in the revision.

---

Table. Comparison on ORIBENCH-Single. SV3D and MV-Adapter generate 6 and 21 views respectively; the one generated image with target azimuth is selected for evaluation. All models use identical prompts and target azimuths. Best results are shown in bold, second‑best are italics.

(4) Targeted problem scope (vs. image editing).

While we agree that image editing with object orientation control is a potentially useful application, we did not initially prioritize this task, as spatial grounding itself is considered a stand-alone application in the literature, and the related prior works [1, 2, 3, R1, R2] also do not necessarily address image editing. We believe the nature of these two tasks differs: grounded image generation focuses on generating images from scratch while aligning with grounding conditions, whereas image editing aims to modify an existing image while preserving its original content.

Nevertheless, we agree that image editing with orientation control is another valuable direction. Our method could potentially be extended to this task by integrating personalized generation techniques or incorporating reward terms that encourage content consistency with a reference image [R5, R6]. We promise to include these additional image editing application results in the revision.

(5) Additional generated examples.

Although we cannot include additional examples due to the NeurIPS policy prohibiting the inclusion of images or videos in the rebuttal, we promise to include additional qualitative results—covering more diverse object categories, captions, and orientations—in the revision. We thank your valuable suggestion.

---

References

[R1] BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion, Xie et al., ICCV 2023
[R2] R&B: Region and Boundary Aware Zero-shot Grounded Text-to-Image Generation, Xiao et al., ICLR 2024
[R3] SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion, Voleti et al., ECCV 2024
[R4] MV-Adapter: Multi-view Consistent Image Generation Made Easy, Huang et al., ICCV 2025
[R5] Improving Editability in Image Generation with Layer-wise Memory, Kim et al., CVPR 2025
[R6] DreamBooth: Fine-Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, Ruiz et al., CVPR 2023

Thank you for your comment and for engaging in discussions with us. Indeed, we believe that our method can potentially be extended to support 3D generation. Since 3D generation requires orientation-grounded image generation along with content preservation, our orientation grounding method can be combined with a personalized image generation technique [R1] to produce multi-view images with consistent content, which can then be used to fit a 3D representation (e.g., NeuS, NeRF) to enable 3D generation. (Here, we assume a 3D generation scheme similar to that of MV-Adapter or SV3D, which reconstructs 3D objects based on the generated multi-view images.)
While our method can be extended for 3D generation in this way, we respectfully reiterate that our targeted task—grounded image generation—is considered a different scope in the existing literature [18, 19, 1, 2, 3], as it aims to generate images from scratch (unlike the mentioned baselines, which require a reference image as input) while aligning with grounding conditions. In these prior works [18, 19, 1, 2, 3], grounded image generation has generally been treated as a standalone application, without necessarily addressing additional applications.
Nevertheless, we agree that 3D generation can be a valuable application of our method, and we promise to include the corresponding results in the revision.

---

References
[R1] DreamBooth: Fine-Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, Ruiz et al., CVPR 2023

Thank you for your thoughtful and encouraging review. We appreciate your recognition of our method’s novelty and strong empirical results. Below, we provide detailed responses to your comments:

(1) Limited 3D control.

While our model is fully capable of controlling the complete 3D orientation—including variations along the z-axis—we designed our experimental setup within the xy-plane to enable comparisons with the baseline methods, which do not support full 3D controllability (L244-246). While we are unable to show additional image results here due to the NeurIPS policy prohibiting the inclusion of images in the rebuttal, we promise to add more such examples in the revised version to better showcase the versatility of our approach beyond the xy-plane.

(2) Ablation study on Langevin Dynamics.

Please note that we have already included experiments validating both our Reward-Guided Langevin Dynamics and Reward-Adaptive Time Rescaling in Table 1 in the paper. To evaluate the effectiveness of Reward-Guided Langevin Dynamics, we included ReNO [22], which uses gradient ascent with noise regularization, as a baseline in the table.

For a more comprehensive comparison, here, we show additional comparisons with a naïve gradient ascent method (i.e., ReNO without noise regularization) in the table below. Our method achieves superior results compared to all these baselines, demonstrating the advantages of Reward-Guided Langevin Dynamics in both orientation alignment and text-to-image alignment.

To validate the effectiveness of Reward-Adaptive Time Rescaling, we already provided an ablation in Table 1 in the main paper, comparing ORIGEN* (without time rescaling) and ORIGEN (with time rescaling), where our final method with time rescaling achieves clearly improved performance.

*Best results are shown in bold, second‑best are italics. *

(3) Other reward function experiments.

Thank you for your thoughtful suggestion regarding the potential extensions of our work. As you kindly pointed out, our method is reward-agnostic and can be potentially extended to other reward alignment tasks such as spatial grounding, depth, or pose. To demonstrate its generality, we conducted additional experiments on spatial grounding (i.e., bounding-box-conditioned 2D position control) and depth-conditioned image generation. In both setups, we used FreeDoM [35] and ReNO [22] as baselines to highlight the effectiveness of our sampling strategy.

---

Spatial Grounding. For this experiment, we randomly sampled 100 examples from the HRS-Bench dataset [R1]. We used GroundingDINO [55] as the reward model to guide object positioning based on bounding-box conditions. As evaluation metrics, we report spatial accuracy using mIoU (measured by SalienceDETR [R2]) and HRS score, and text-image alignment using CLIP similarity, VQA score, and PickScore.

---

Depth-Conditioned Generation. For this setup, we randomly sampled 100 image-caption pairs from the ORIBENCH-Single dataset and generated pseudo-ground-truth depth maps using DepthAnything-V2 [R3]. We then used DepthAnything-V2 as a reward model to guide image generation and evaluated the results using Absolute Relative Error (AbsRel).

These results demonstrate that our method is reward-agnostic and effectively generalizes to different types of guidance signals, such as 2D spatial constraints and depth cues, without requiring task-specific model modifications. While this was originally raised as a limitation of our method, we believe it instead highlights our method’s strength—its generalizability and potential applicability to broader tasks in the future.

(4) Discussion on failure cases and text prompts with explicit orientation cues.

Failure cases. Generating images of objects with inherently ambiguous orientations (e.g., objects with little distinction between front and back) can indeed be challenging to handle in our framework. We will make sure to clarify these limitations in the revised version of the paper.

Text prompts with explicit orientation cues. We conducted additional ablation experiments using text prompts augmented with explicit orientation cues. We considered two settings: (1) adding text prefixes with abstracted orientations (e.g., “{object} is viewed from front-right”), and (2) adding text prefixes with both abstracted and numerical orientations (e.g., “{object} is viewed from front-right, low-angle (azimuth 315°, polar 61°, rotation 90°)”). As shown in the table below, these settings result in negligible effects in orientation grounding performance on the ORIBENCH-Single benchmark, suggesting that orientation grounding cannot be trivially improved by simply adding explicit orientation phrases to the prompt—reinforcing the necessity of our reward-guided sampling approach. We will include this discussion in the revised version of the paper.

---

References

[R1] HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models, Bakr et al., ICCV 2023
[R2] Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement, Hou et al., CVPR 2024
[R3] Depth Anything V2, Yang et al., NeurIPS 2024

We sincerely appreciate your recognition of our work as the first zero-shot framework for 3D orientation grounding. Below, we respond to your concerns in detail:

(1) Reliance on the OrientAnything reward model.

Although the neural reward model (OrientAnything [20]) might not be perfect and could potentially affect our framework’s performance, our experimental results nevertheless demonstrate that our approach significantly outperforms existing orientation grounding methods (Table 1 in the paper).

We also note that, in related works on reward alignment for diffusion models [R1, R2, R3], most methods rely on neural reward models [R4, R5, R6] to enable alignment with objectives that are difficult to analytically define (e.g., text-to-image alignment, aesthetic preference). A similar trend is also widely observed in LLM reasoning, where neural reward models are used to evaluate and guide multi-step reasoning processes [R7, R8]. We similarly follow this paradigm to use a pretrained neural model to define a reward function tailored to our task.

(2) Sampling ~50 steps, making the one-step model narrative less compelling.

As discussed in Sec. 3.1 (Why Based on a One-Step Generative Model?), our main purpose for using a one-step generative model is to enable clearer and more reliable reward estimation, rather than to speed up image generation. Note that our reward model (OrientAnything [20]) was trained on, and expects, a clear image as input to compute accurate rewards. However, images generated by multi-step generative models in early steps are often blurry due to the high variance in their marginal distribution. For further discussion on this particular point, we kindly refer the reader to (5) below.

Although efficiency was not our primary motivation for adopting a one-step generative model, we also provided a computation time analysis in Sec. 4.7 in the paper. There, we discussed that our method’s number of function evaluations (NFEs) was set to 50 to perform fair comparisons with the multi-step baselines to match the number of their denoising steps. In practice, our method requires only an average of 12.8 NFEs—approximately 4× fewer than the 50 NFEs used in our reported experiments—to achieve superior orientation alignment than all the baseline methods. Please refer to Sec. 4.7 for the full analysis.

(3) In-depth discussion about Reward-adaptive Monitor Function.

In our proposed monitor function, $\mathcal{G}(x) = s_{\min} - \tanh (k x) \cdot (s_{\max} - s_{\min})$, hyper-parameters include $k $ controlling the slope (how the step size shrinks as the reward rises), and $s_{\min}, s_{\max} $ bounding the step size values. Below, we present additional experimental results to assess the sensitivity of these hyper-parameters; note that hyper-parameters are set to their default values in the main paper unless otherwise specified.

(i) Ablation on $k $

(ii) Ablation on $s_{\min}$, $s_{\max}$

*Best results are shown in bold, second‑best are italics. *

Take-away. Our performance is robust to all three hyper-parameters: changing $k, s_{\min}, s_{\max} $ has negligible impact on all evaluation metrics. Our defaults were chosen by a coarse grid search, not to finely tune peak scores, confirming that ORIGEN does not rely on fragile hyper-parameter selection. Moreover, all variants with the reward-adaptive monitor consistently outperform the setting without it (Table 1 in main paper), highlighting its overall effectiveness.

(4) Comparison with RL-based fine-tuning baseline.

Following your suggestion, we provide additional comparison results with a well-known RL-based fine-tuning method, DDPO [50], which fine-tunes a diffusion model via policy gradient updates on a learned reward signal. We note that existing RL-based diffusion fine-tuning methods are designed for either unconditional or text-conditional generation [48, 50, 51]. Thus, to enable orientation conditioning while leveraging existing code of DDPO, we opt to appending orientation phrases to the input text prompts (e.g., “~. {object} is viewed from front-right, low-angle (azimuth 315°, polar 61°, rotation 90°)”). To fine-tune with our orientation grounding reward, we constructed a single-object text–orientation paired training dataset using 50k orientations from the OrientAnything training set and captions from the Cap3D dataset [R9, R10]. For other fine-tuning setups (e.g., hyper-parameters), we primarily followed the original DDPO configuration to ensure fair comparisons, and we will release our full implementation.

As shown in the table below, DDPO-based fine-tuning demonstrates limited improvement of the orientation alignment over the base model (FLUX.1-dev). Moreover, it also results in substantial degradation in image quality, as we described in L106–109. In contrast, our method achieves the highest performance in both orientation and text-to-image alignment, while preserving image fidelity.

(5) Lack of analysis about multi-step guided generation methods

Please note that we already included comparisons with multi-step diffusion-based methods, which are DPS, MPGD, FreeDoM in Tables 1 and 5 of the paper, where we experimentally validated their suboptimal performance. We respectfully clarify that the phenomenon of "blurry early-stage outputs and minimal updates in late stages" is widely discussed in the existing literature [R11, R12, R13, R14]. For visual evidence, please refer to Fig. 5 and Fig. 6 in DDPM [R15], and Fig. 1 in EDM [R16], which illustrate the blurry Tweedie estimations in the early timesteps.

For the additional baselines suggested in the review (InitNO, D-Flow), both approaches are either infeasible (InitNO) or lack accessible implementations (D-Flow):

InitNO [45] was originally designed for an attention score reward, which can be computed after very few reverse steps (typically 1-step). However, the orientation grounding reward in our task must be evaluated on the clean image, requiring backpropagation through much larger timesteps—making it infeasible given current GPU memory limitations.

D-Flow [43] addresses the memory issue via heavy gradient checkpointing, but no public code is available. We contacted the authors to request the implementation but did not receive a response. Aside from this code accessibility issue, we note that D-Flow reports a computation time of 3–40 minutes per prompt, making it orders of magnitude slower than our method.

---

References

[R1] Diffusion Model Alignment Using Direct Preference Optimization, Wallace et al., CVPR 2024
[R2] Test-time Alignment of Diffusion Models without Reward Over-optimization, Kim et al., ICLR 2025
[R3] Training Diffusion Models with Reinforcement Learning, Black et al., ICLR 2024
[R4] Laion Aesthetic Predictor, Schuhmann et al., 2022
[R5] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation, Xu et al., NeurIPS 2023
[R6] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, Kirstain et al., NeurIPS 2023
[R7] Training verifiers to solve math word problems, Cobbe et al., arXiv 2110.14168
[R8] Let's Verify Step by Step, Lightman et al., ICLR 2024
[R9] Objaverse: A Universe of Annotated 3D Objects, Deitke et al., CVPR 2023
[R10] Scalable 3D Captioning with Pretrained Models, Luo et al., NeurIPS 2023
[R11] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers, Balaji et al., arXiv 2211.01324
[R12] Prompt-to-Prompt Image Editing with Cross Attention Control, Hertz et al., ICLR 2023
[R13] Perception Prioritized Training of Diffusion Models, Choi et al., CVPR 2022
[R14] R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation, Xiao et al., ICLR 2024
[R15] Denoising Diffusion Probabilistic Models, Ho et al., NeurIPS 2020
[R16] Elucidating the Design Space of Diffusion-Based Generative Models, Karras et al., NeurIPS 2022

Thank you for your thoughtful and encouraging review. We sincerely appreciate your recognition of the novelty and technical soundness of our approach. We’re also grateful for your positive feedback on our reward-guided sampling, benchmark contributions, and the clarity of our presentation. Below, we provide detailed responses to your comments:

(1) Comparison with naïve gradient‑ascent method.

As a kind reminder, we note that we already included comparisons with ReNO [22] (Tables 1 and 5), which is the naïve gradient ascent with noise regularization. The suggested comparison setting—the naïve gradient ascent method—can be interpreted as a simplified version of ReNO without noise regularization, and was not initially prioritized in our experiments, as we considered it an ablation on different types of regularization of ReNO.

Nevertheless, following your suggestion, we conducted an additional comparison with this naïve gradient ascent baseline. As shown in the table below, this method (denoted by Naïve GA) performs the worst in both orientation accuracy and text-to-image alignment. These results further highlight that the stochastic exploration inherent in our Langevin dynamics not only mitigates deviation from the original distribution but also improves alignment with the target reward signal. We thank your comment and will include these results in the revision.

*Best results are shown in bold, second‑best are italics. *

(2) Video visualization of continuous orientation change.

Thank you for your helpful suggestion. We agree that adding a video visualization of continuous orientation change would be informative. Although we cannot directly show our video results due to the NeurIPS rebuttal policy that prohibits the inclusion of images or videos, we promise to add these video visualizations in the revision.