Preventing Shortcuts in Adapter Training via Providing the Shortcuts

We thank reviewer fa9d for their constructive comments and feedback. We are delighted that they found our solution "counterintuitive but effective," which precisely reflects the sentiment we aimed to convey with our title.

Weakness 1: Other confounding factors like “background” and “lighting”, or “complex scenes”

Motivated by the suggestion to test SR on additional attributes and the practical observation that lighting often "leaks" from the input image, we conducted a new experiment. When prompted to generate an image with artistic lighting, this effect is frequently diminished in the personalized model, as the input image typically features more ordinary, everyday lighting. Based on this, we experimented with shortcut rerouting for lighting.

To establish a lighting shortcut, we trained an IPA where the input image was the target image with the human masked out. This adapter is designed to encode all target image information except the human. We then used this adapter as a shortcut to train our personalization adapter (IPA w/ SR-LG). Our preliminary results indicate that this mechanism effectively factors out lighting and improves adherence to stylistic prompts during generation. Due to the inability to include qualitative results in the rebuttal, we used the LPIPS between the generated and prior images as a proxy for lighting preservation. Additionally, we conducted a user study across all testing prompts where raters viewed the prompt, prior image, and a generated image, and answered: “Is the lighting harmonized with the prompt?”

These results demonstrate that SR effectively factors out attributes such as "lighting" and improves adherence to the prior. We will include this new experiment and its qualitative results in the revision.

Weakness 2: Limited improvements in qualitative results

Our primary contribution is "shortcut rerouting training." We aim to validate its effectiveness in eliminating shortcuts during inference, specifically "expression," "pose," and "distribution shift." Figures 5 and 6 qualitatively demonstrate these improvements, showing that SR-CN results more closely resemble "Prior" generations than baselines. "Prior" generations were produced by the same T2I model without the adapter, using identical prompts and seeds. An ideal adapter, free of shortcuts, should optimally maintain the pose and structure of this "prior" image. We will make this point more prominent in the revised writing.

Q1: Does the SR-IPA mentioned in this paper have this decoupling ability for any input image after training, or does it simply need to be trained for each case, which means that each case corresponds to a specific SR-IPA weight?

Once trained, the SR-IPA operates in a feedforward manner and generalizes to unseen input identities without requiring retraining. The adapter is trained to factor out specific shortcut cues (e.g., lighting, background, pose, expression), and this decoupling behavior holds for new test images as long as the shortcut definition remains the same. If users wish to factor out a different type of shortcut, a new adapter must be trained accordingly. In other words, the decoupling generalizes across identities, but not across shortcut types.

Q2: How to define these confounding attributes? So far, this paper focuses on the decoupling of face identity information and its pose or background. Is there any other decoupling ability? For example, for face only, can we decouple makeup, hair color/style, glasses, and other attributes?

We came to the choice of these confounding attributes–“expression”, “pose” and “distribution shift”--from practically trying to build a useful appearance adapter. We seek a “face” (or “fullbody”) adapter that can generate anybody in any imaginary scene. Yet, in practice, we found that existing adapters failed to control the character’s expression through prompt, and failed to adapt the characters' pose through prompt. These severely limited the kind of images we can generate.
Your suggestion of decoupling makeup, hair, and accessories sounds interesting! “Hair” and “accessories” can likely be masked out by segmentation masks to a large extent. “Makeup” sounds like a good attribute to factor out given a dataset that isolates “makeup”.

Q3: This paper uses a text-to-image generation model. How to input the prior examples given in this paper? Is it to directly use some existing tools, such as ip-adapter, to encode and input images? Or enter the text prompt of the corresponding image? If a text prompt is entered, how can to ensure the consistency with the prior image?

The prior images in the paper are generated using the same base text-to-image model (FLUX.1 Dev) with the same prompt and a fixed random seed— just without using any image encoder or IP-Adapter. These "prior images" are not used as inputs to our method. Instead, they serve as a reference to visualize what the model would produce without any adapter conditioning. Their purpose is to illustrate how much the adapter alters the layout, lighting, pose, or expression. An ideal "appearance" adapter should preserve these attributes (e.g. layout of IPA generation should remain same as in prior image). As shown in Figures 5 and 6, our method produces images that stay more faithful to the prior compared to baseline methods.

Q4: In the experiment section, it is declared that a dataset has been collected. Will the dataset be open-accessed?

We are currently exploring options for responsibly sharing the dataset. In the meantime, we emphasize that our contribution, Shortcut Rerouting training, is simple in both concept and implementation. To support reproducibility, we provide detailed methodological descriptions and include additional experiments that demonstrate the generality of our approach. Given the minimal requirements of our method, we anticipate that it can be readily reproduced using similar data.

We believe our responses have addressed your concerns and clarified any confusion. We respectfully request you to consider updating your rating, as the additional analyses and clarifications strengthen the case for acceptance. We are available for further dialogue and to refine our submission should any questions arise during the discussion period.

We thank the reviewer for the encouraging feedback and thoughtful comments. Below, we address the main concern raised:

Limited number of tested attributes

We agree that expanding the range of evaluated attributes would further strengthen our claims.

We investigated specific shortcuts due to their practical implications. During IPA development and testing, we observed that generated images often failed to align with the prompt, as pose and expression from the input image "leaked" through (i.e., pose/expression shortcuts). Additionally, the aesthetic quality of the trained model deteriorated compared to the base FLUX.1-Dev model (i.e., distribution shift). These observations motivated our study of these attributes.

Motivated by the suggestion to test SR on additional attributes and the practical observation that lighting often "leaks" from the input image, we conducted a new experiment. When prompted to generate an image with artistic lighting, this effect is frequently diminished in the personalized model, as the input image typically features more ordinary, everyday lighting. Based on this, we experimented with shortcut rerouting for lighting.

To establish a lighting shortcut, we trained an IPA where the input image was the target image with the human masked out. This adapter is designed to encode all target image information except the human. We then used this adapter as a shortcut to train our personalization adapter (IPA w/ SR-LG). Our preliminary results indicate that this mechanism effectively factors out lighting and improves adherence to stylistic prompts during generation. Due to the inability to include qualitative results in the rebuttal, we used the LPIPS between the generated and prior images as a proxy for lighting preservation. Additionally, we conducted a user study across all testing prompts where raters viewed the prompt, prior image, and a generated image, and answered: “Is the lighting harmonized with the prompt?”

These results demonstrate that SR effectively factors out attributes such as "lighting" and improves adherence to the prior. We will include this new experiment and its qualitative results in the revision.

In conclusion, we hope our responses have sufficiently addressed concerns. We respectfully request you to consider updating your rating, as we believe the additional analyses and clarifications strengthen the case for acceptance. Should any further questions arise during the discussion period, we would be happy to engage in continued dialogue and refine our submission accordingly.

We respectfully address the criticism raised by the reviewer.

1. “Title” – We acknowledge that the title can appear informal. It was a very deliberate decision, and understandably risky. Our intent was to use a conversational tone to highlight the counterintuitive insight that “providing the shortcuts” can prevent shortcut learning. We are happy to discuss the possibility of revising the title, and removing the “ .. well ..”.
2. “Format error and typo” – Thanks for pointing out the two syntax errors. We will remove the line break in the abstract, and remove the straying “fig” on page 2.
3. “Math” – Our formalism aims to outline the core idea of shortcut rerouting in adapter training – by explicitly routing confounders through separate modules, we change the optimization path and effectively guide the adapter toward disentangled representations.
4. “.. approach appears to be a straightforward combination of existing techniques, namely LoRA and ControlNet” – We would like to highlight that our contribution is the “shortcut rerouting” training scheme, and the use of LoRA and ControlNet are just to provide the shortcut. We use these existing tools in very unintended (and novel) ways. In fact we argue that using well-established tools as part of our method adds to the usability of the method, not diminishes it. We respectfully refer the reviewer to Strength 2 highlighted by reviewer fa9d, which emphasizes the advantage of leveraging mature tools to enable shortcut rerouting.
5. “have been generated entirely by an AI large model” – We are disheartened to see such a comment directed towards our project. First, as indicated in our authors’ checklist, we acknowledge using LLMs for editing our paper. Second, if the paper were entirely generated using an LLM, it would likely pick a much more conventional title and not cause the concern you had.

We thank reviewer wPqP for the constructive and thoughtful feedback on our submission. We appreciate your recognition of the clarity of our presentation, the intuition behind shortcut rerouting, and the strong qualitative results demonstrating improved identity and attribute preservation. Your comments have been very helpful, especially the suggestion to use the EMOCA model for expression evaluation. We address the concerns raised below.

Q1: “Did you consider learning other shortcuts using existing encoders? E.g. expression parameters from EMOCA, CelebA attributes, head-pose from HopeNet ...”

We are grateful for the suggestions to explore shortcuts using existing encoders like EMOCA or HopeNet. Our method establishes pose and expression shortcuts via ControlNet. Achieving a similar outcome with EMOCA/HopeNet would require learning an adapter compatible with their outputs, which would be less user-friendly than the presented approach.

We believe our extensive experiments sufficiently demonstrate the effectiveness of SR training in human-centric generation tasks. While our SR training strategy is inherently task-agnostic and broadly applicable, extending it to non-human subjects is beyond the scope of this work. As a team focused on human-centric personalization, we currently lack the resources (e.g., data and compute) and domain expertise to train adapters for general objects. Nevertheless, we hope our findings can inspire researchers in related areas to explore this direction further. For instance, training a background adapter (SR-BG) and incorporating it into object adapter training may help mitigate background and lighting leakage—potentially enhancing generalization in non-human scenarios.

Q2: Usage of FaceNet vs. ArcFace/AdaFace

We appreciate the suggestion to use ArcFace for identity evaluation. However, we note that competing baselines such as InifinteYou and PuLID use ArcFace as their image encoder. To avoid coupling and biasing the evaluation, we chose FaceNet (i.e. an encoder not used by any of the methods) to ensure a neutral comparison. FaceNet was also used as a standard metric in recent studies on face swap [Huang et al., 2024] (where face similarity is the focus).

Q3: Evaluation for identity preservation and expression control

We included LM-as-a-judge for several reasons.

Existing encoder-based face identity metrics (e.g., FaceNet, AdaFace, ArcFace) are limited to evaluating only facial features. They fail to assess other critical aspects of appearance, such as head shape and hair. To overcome this limitation and achieve a more comprehensive identity evaluation, we adopted LM-as-a-judge.

Furthermore, LM-as-a-judge has been previously employed in papers requiring custom metrics for controlled generation, as exemplified by Stylus [Luo et al., 2024], a work cited in our submission from the last NeurIPS.

For evaluating expression, using LM-as-a-judge followed a similar thought process. Yet, we really appreciate the suggestion of using existing blendshape prediction models like EMOCA for evaluation. We evaluated the similarity of expressions by inputting both prior and generated images into the EMOCA model. We then calculated the cosine similarity of their respective EMOCA outputs. Prior images were created using the base Flux model, without personalization. A higher cosine similarity indicates a greater resemblance in expressions. Ideally, a personalization adapter should achieve a high value, demonstrating that it does not leak expression from the input image. Below are the table of results:

The result of EMOCA similarity shows that IPA trained with SR-CN (i.e., factoring out the 2D keypoints) indeed achieves a higher similarity with the expression in the prior model. In terms of downstream impact, this indicates that such a model is more effective in adhering to the expression prompt, hence allowing users to create more diverse generations without being limited by a personalization adapter with expression leakage.

Q4: "Evaluation set".

Our evaluation set includes 32 identities from real-world photos, encompassing both studio images and in-the-wild selfies, as well as 146 diverse prompts covering 5 different test cases (see Appendix G for details). We thank the reviewer for the suggestion to evaluate on cartoon or drawing-style inputs, which is out of our training distribution—we view this as a valuable and interesting direction for future work (to include stylized inputs in training).

That said, we have included 10 stylized prompts with cartoon/drawing aesthetics to validate the capacity in generating stylized images with photorealistic input faces. In these cases, our method (IPA with SR-CN) significantly outperforms both the baseline IPA (without SR-CN) and other state-of-the-art methods, InfU and PuLID. The baseline IPA consistently fails to produce stylized outputs, generating photorealistic images regardless of the prompt. While InfU and PuLID are capable of generating stylized images, they struggle to preserve identity. In contrast, our IPA with SR-CN successfully follows the stylized prompts and produces cartoonized images while maintaining identity fidelity. As images are not allowed in rebuttal, we conducted a user study in which participants were asked to select their favorite generation among all methods:

Q5: “Qualitative results for the SR-LoRA method is not shown in neither the main paper nor the supplementary materials”

We would like to clarify that SR-LoRA qualitative results are included in the supplementary material (see Appendix Fig. 3, 4, and 5 with the column name “IPA w/ LoRA”). We apologize if this was not sufficiently highlighted in the main text. We will rename it as “IPA w/ SR-LoRA (Ours)” and make clearer references in the revision to guide readers to these results.

Additional Comment: Potential negative societal impact

We thank the reviewer for pointing out that there was no explicit discussion regarding the potential negative societal impact. We will add the following paragraph in the revision:

As our method improves identity preservation and expression control in personalized generation, it naturally raises concerns about misuse, particularly in the creation of hyper-realistic synthetic identities or deepfakes. We acknowledge that enhanced controllability and realism may lower the barrier for malicious use. To mitigate such risks, we advocate for responsible deployment practices, such as model watermarking, usage restrictions, and alignment with ethical frameworks for generative media.

In conclusion, we hope our responses have sufficiently addressed your questions and clarified the contributions and design choices of our work. We respectfully request you to consider updating your rating, as we believe the additional analyses and clarifications strengthen the case for acceptance. Should any further questions arise during the discussion period, we would be happy to engage in continued dialogue and refine our submission accordingly.

References

• [Luo et al., 2024] Luo, Michael, et al. "Stylus: Automatic adapter selection for diffusion models." Advances in Neural Information Processing Systems 37 (2024): 32888-32915.
• [EMOCA] https://github.com/radekd91/emoca
• [Huang et al. 2024] Huang, Ziyao, Fan Tang, Yong Zhang, Juan Cao, Chengyu Li, Sheng Tang, Jintao Li, and Tong-Yee Lee. "Identity-preserving face swapping via dual surrogate generative models." ACM Transactions on Graphics 43, no. 5 (2024): 1-19.
• [InfU code reference of using ArcFace] https://github.com/bytedance/InfiniteYou/blob/5fff05b3cba6c119857d62d9cc27a90bcc501187/pipelines/pipeline_infu_flux.py#L269