FreeControl: Efficient, Training-Free Structural Control via One-Step Attention Extraction

Rebuttal to Weakness 1: Quantitative Analysis of Ablations

We appreciate Reviewer 4’s call for deeper quantitative analysis of the ablations — this is a fair point and will strengthen the paper. Our initial focus on qualitative illustrations in Sec. 5.2 and Fig. 5 was driven by two considerations:

1. The shifts caused by hyperparameters (layer depth, σ, timestep condition) produce very visible changes — such as structure tightening or relaxing and granularity change — rather than subtle metric tweaks. We believed showing these striking differences directly would help readers intuitively understand how each setting affects the result.

2. Because these parameters can combine in many ways, exhaustive metrics for every combination would create a prohibitively large experiment grid.

That said, we fully agree that numerical backing is important. We have already run a minibatch of quantitative ablations and include those results below, with more comprehensive runs planned for the camera‑ready.

Table A: Mini‑Batch Quantitative Ablation Results

Table B: Mini‑Batch Quantitative Ablation Results on Stylized COCO

On timestep 661: the discussion in L166 and the sweep in Sec. 5.2/Fig. 5 already show that we evaluated the entire range, and 661 provided the best balance for Flux. There is also a clear trend from both directions — from timestep 0 moving up toward 661 and from timestep 1000 moving down toward 661 — where structural alignment and generation quality steadily improve, peaking around the optimal region near 661. The partial quantitative results are reflected in Table A, and a more complete version will be provided later.

When we tested FreeControl on other backbones, the exact numbers shifted — as expected — but the principles for selecting these hyperparameters and how they affect control remained consistent. These principles are carefully embedded into the method section so they can guide future applications of FreeControl to new models.

We thank the reviewer again for highlighting this — it prompted us to expand our quantitative experiments and commit to adding more rigorous analysis from multiple dimensions in the paper and appendix to fully justify these design choices. An additional analysis regarding matrix similarity is also provided in our response to Reviewer T9YS, which may be referenced for further insights.

Rebuttal to Weakness 2: Baselines

We sincerely thank Reviewer 4 for emphasizing the importance of comparing FreeControl against recent Flux/DiT‑based control methods and for providing helpful recommendations. This was a valuable point, and we acted on it directly.

Using released code, we ran In‑Context Edit and Taming Rectified Flow, adding their results in Table A below. The other suggested methods currently have no public implementations, but we will include them in the camera‑ready version once code becomes available.

As shown in Table A, FreeControl performs comparably to — and in several cases surpasses — Taming Rectified Flow across key metrics. In‑Context Edit (ICEdit), built on the FLUX.Fill model for image completion (filling areas in existing images), scores strongly on similarity metrics because its base model is highly conservative in retaining existing content, but is performing less optimally on CLIP-T on both Table A and B. Visual results from the experiments in Table B further highlight its limitations: despite favorable metrics, ICEdit frequently produces unnatural and low-quality outputs that appear unrealistic or visually jarring—such as disjointed objects, missing body parts, or incoherent scenes. In contrast, our method demonstrates significantly greater stability and consistently generates higher-quality, more natural, and human-acceptable results

We chose Flux intentionally to demonstrate FreeControl’s effect on a strong, modern backbone — but stronger generative quality alone does not guarantee easier structural control. That is why we prioritized adding new baselines built on Flux itself. These comparisons show that our method performs comparably or better than other Flux‑based methods on metrics, and that those methods often produce generations with severe visual flaws. In contrast, FreeControl consistently produces stable, natural‑looking images, which is essential for applicability.

We also note that Table 1 of the paper already includes ControlNet‑Flux, the latest Flux/DiT ControlNet adaptation, ensuring FreeControl is also evaluated against modern Flux‑based pipelines.

Table A: Mini‑Batch Expanded Baseline Comparison Results

Table B: Mini‑Batch Expanded Baseline Comparison Results on Stylized COCO

Rebuttal to Weakness 3: Facial Identity

We thank Reviewer 4 for pointing out the identity issue in some of our human examples. The reviewer is correct that some samples appear looser — this was intentional, as in Fig. 1 we used relaxed control settings (σ/timestep) to illustrate FreeControl’s ability to support different levels of structural strength and granularity, such as a comic‑style look. When identity retention is required, FreeControl supports much tighter guidance through the same hyperparameter adjustments described in the method section.

We have prepared additional facial examples to demonstrate this, but unfortunately, since rebuttals cannot include new images or figures, we are unable to show them directly here. You could refer to the wolf in the middle‑left of Fig. 1 for a rough approximation of this fine‑grained control — though we note that FreeControl actually supports even higher control of details when the hyper-parameters are tuned for identity‑sensitive tasks. We will provide a comprehensive discussion on this topic, accompanied by a complete quantitative ablation study in the full paper.

Overall, as long as the prompt itself does not explicitly request facial changes, FreeControl could maintain identity features under higher control. For example, in one prepared (but not attachable) sample, adding a “cyberpunk face implant” preserved the subject’s facial details to a high extent while incorporating the requested implant — but if the prompt describes a more direct change to global facial identity, then in this case the facial identity is not guaranteed to be preserved unless control is pushed to an extent where it begins to limit the creative freedom of other aspects of the image (and since rebuttals cannot include new images, we explain this nuance here in words rather than figures).

Thank you to the authors for the detailed responses, which have largely addressed my concerns. I hope the authors will consider incorporating the qualitative and quantitative comparison experiments discussed in the rebuttal into future revisions, as doing so would significantly strengthen the paper’s overall clarity and persuasiveness.

Rebuttal to Reviewer 3

We sincerely thank the reviewer for the careful read and valuable feedback. We address your points in depth below and clarify areas that may have caused confusion.

(1) Baseline Scope (ControlNet vs. Inversion Methods)

Initial baseline choice
Inversion-based methods like DDIM Inversion or Null‑Text Inversion perform very different tasks: they reconstruct the full latent trajectory to edit an image, which makes them perform less optimally on structure‑controlled regeneration. Because they must run the entire reverse denoising process, they are typically 2–3× slower than FreeControl and are designed to alter what’s already present, rather than enforcing where new content should appear.

By contrast, ControlNet‑style approaches are explicitly designed for structural guidance, using external maps (edges, depth, etc.) to align generation. This makes them the most relevant comparison point for FreeControl’s goal of test‑time structural control.

Added methods and new runs
That said, we agree with the reviewer that additional baselines improve fairness and completeness. We have now added In‑Context Edit and Taming Rectified Flow to our experiments.

We have already run a mini‑batch comparison with these methods, and their preliminary results are reflected in the rebuttal tables. We will expand this evaluation with full runs for the final paper and appendix, ensuring FreeControl is compared against a broader range of recent methods as requested.

As shown in Table A, FreeControl performs comparably to, and often surpasses, Taming Rectified Flow across several metrics. ICEdit, built on the FLUX.Fill model for image completion, benefits in similarity metrics due to stronger content retention. However, its CLIP-T score is lower, and it often struggles with stability and controllability when following editing instructions. Visual results from the experiments in Table B further highlight its limitations—despite favorable metrics, ICEdit frequently produces unnatural, low-quality outcomes that appear unrealistic or visually jarring, such as disjointed objects, missing body parts, and incoherent scenes. In contrast, our method is significantly more stable and consistently produces higher-quality, more natural results.

Table A: Mini‑Batch Expanded Baseline Comparison Results

Table B: Mini‑Batch Expanded Baseline Comparison Results on Stylized COCO

(2) Timestep Condition (“Magic Number 661”) and LCD

We appreciate the chance to clarify this, as some confusion comes from mixing two different timestep concepts:

– t in Eq. 1 refers to the timestep in the forward simulation used to create the latent for query extraction.
– Timestep condition is the timestep value fed into the diffusion transformer when queries are extracted.

In Section 3.1, they are treated in sync — t and the timestep condition have the same value, as diffusion models normally do. In Section 3.2, however, LCD decouples the latent and the timestep condition, so that in L166 it points to the timestep condition, not the t in the forward process, and the equation around L160 produces the latent in a way that no longer takes t as a parameter (sorry for the missing formula numbering — since the equation was unnumbered, we can only refer to it by this line number).

Section 3.2 then walks through how these two inputs interact: the latent determines what information can be extracted, while the timestep condition shifts how the model interprets it.

The reviewer notes that 661 appears with no context in L154 — but in fact it is first introduced and explained earlier, where the forward process for query extraction is described.

The number 661 is not arbitrary — it was empirically selected by sweeping across candidate timesteps to find the best balance of structural fidelity and flexibility. We have also run a mini‑batch quantitative ablation specifically showing how different timestep condition choices (including 661 and nearby values) affect structural alignment and generation quality, and those results are included in Table C. We also include a quantitative ablation study on the Stylized COCO dataset; please refer to the corresponding table provided in our responses to other reviewers.

Table C: Mini‑Batch Quantitative Ablation Analysis

Sadly, we cannot include the full sweep plot here because rebuttals do not allow image attachments, but the middle of Fig. 5 offers a similar reference. It is not identical to the sweep — the latent there is fixed (derived from the forward process around L160), whereas the initial findings in Eq. 1 used matching latents (latents created at the same timestep as the timestep condition). This still provides a reasonable approximation of how the timestep condition influences structural control.

(3) Ablation Depth and Design Choices

We acknowledge the reviewer’s point that the ablation section would benefit from stronger quantitative backing. Section 5.2 already examines the key factors — injection depth, σ (noise factor), and timestep condition — and Fig. 5 shows how each parameter affects generation. We initially leaned on qualitative comparisons because the changes are immediately and clearly visible — for example, deeper injection tightens structure but dulls colors, and adjusting the timestep condition shifts how rigid the layout becomes — and these visualizations were considered the most direct way to convey how each design choice influences the results.

That said, we agree that numbers are needed to complement the visuals. We have already run a mini‑batch of quantitative ablations as in Table C. We are extending this work and will add fuller quantitative results and a more detailed analysis from multiple dimensions in the paper and appendix to justify these design choices more completely.

Responses to the reviewer’s specific questions

1. Fig. 1 clarity
We agree Fig. 1 needs clearer labeling. In the revised paper, we will provide an updated version that explicitly marks Reference Image, Prompt, and Generated Output so the input–output flow of FreeControl is obvious at a glance.

2. Missing related work
We agree that additional related work should be acknowledged. Citations to RB‑Modulationand RL‑based editing will be added. We will also further review the literature and include additional relevant references to strengthen the related work section.

3. Eq. 1 “linear interpolation”
Eq. 1 uses the forward process from a Euler-discretized FlowMatch scheduler. It applies the forward process at timestep t to simulate a noised latent for query extraction. We omit the calculation of the sigma value here for simplicity, as the t-dependent sigma is later replaced by a fixed value. In LCD mode, we remove the stochastic term to avoid speckle artifacts and improve stability.

4. Ablations around L133
The ablations the reviewer flagged as “missing” are in fact already included in Sec. 5.2 (L274) and illustrated in Fig. 5. That section varies layer injection depth, σ, and timestep condition, showing how each choice affects control strength and image fidelity. We have also added quantitative ablations, as discussed above, to provide numerical backing for these choices.

5. “Mid‑to‑late layers” explanation
This phrase refers to layers closer to the model’s output side — beyond the very early layers that model color and texture. Injecting queries here provides strong semantic and structural guidance without disturbing early‑stage appearance modeling, as shown in Fig. 5.

6. Question about Eq. 160 and timestep (L166)
There is a misunderstanding here: the t in Eq. 1 is not the same as the timestep condition. In Eq. 1, t is used in the forward process to generate the latent. In Section 3.2, we decouple these by introducing LCD — the equation around L160 generates the latent in a way that no longer depends on t. The timestep mentioned in L166 is the timestep condition passed to the diffusion transformer during denoising, not the t used in the forward process.

Rebuttal to Reviewer 2

We sincerely thank the reviewer for their thoughtful and constructive feedback. Below, we respond point by point and summarize both new and existing evidence.

(1) Novelty and Conceptual Contribution
Prior works on attention injection largely use attention matrices “as it is” — the matrices they extract, and the control they deliver, tend to be vague, unstable, and time‑consuming to apply because they rely on multi‑step extraction. FreeControl changes this by reducing the process to one carefully chosen timestep, which isolates a single, stable query signal and allows us to identify and manipulate the key factors embedded in that signal.

This one‑step simplification is not just a convenience — it enables something fundamentally new by reducing the problem to one query matrix: we could now easily manipulate the content encapsulated in the query matrix itself, giving FreeControl the ability to deliver precise, flexible, and interpretable structural control. Because the process collapses to a single controllable factor, we can:

• adjust the latent input directly (noise vs. no‑noise) to improve quality,
• provide a clean lever for structural granularity via LCD, and
• outperform training‑based methods on structural control despite being a test‑time augmentation method with only ~5% inference overhead.

(2) Generalization Beyond DiT and FLUX
FreeControl is designed to operate purely at the attention level, making it architecture‑agnostic. We have implemented it on U‑Net–based models (SD1.5, SDXL) and observed strong structural control behavior after only minimal adjustments to fit the model. It would, honestly, take some time to tune the hyperparameters and carefully check the details to achieve stable and robust performance.

Our method:

• is not DiT‑specific, and
• works even on models without RoPE, further confirming that FreeControl’s structural adherence does not rely on positional encoding quirks.

(3) Understanding One‑Step Extraction: Similarity, Timestep Intuition, and Trade‑offs
The reviewer requested a deeper look at one‑step extraction, and we provide several perspectives:

Layer‑wise Similarity, Continuity, and LCD Alignment

We extracted both the query matrices from LCD and from the optimal key timestep without LCD (as defined in Sec. 3.1) and compared each against the query matrices from every timestep of the multi‑step extraction variant. A few key points emerged from this analysis.

Table A: One‑Step vs. Iterative Q‑Matrix Similarity

First, the layer‑wise similarity shows a clear low‑to‑high trend: shallow layers tend to have lower similarity, while deeper layers converge more. This aligns with our layer‑aware injection choice in Sec. 3.1 — early layers focus more on appearance elements rather than structure, diverging more across timesteps and contributing less to shared structural signals.

Second, we observed strong cross‑timestep continuity. Each query matrix is most similar to those from adjacent timesteps and steadily diverges from those further away. This continuity reinforces that the structural signal evolves gradually along the denoising trajectory, validating our full sweep across timesteps that identified ≈ 661 as the optimal point, and showing how control quality predictably falls off as we move away from that region.

Third, LCD adds an important refinement. When we compared LCD‑generated query matrices to the multi‑step variant, similarities ranged 0.4–0.7 and peaked around the same optimal timestep region. As timesteps drift further from that point, similarity drops. This indicates that LCD sharpens the structural guidance precisely where it is most effective, aligning closely with the paper’s observation that LCD improves on the “raw” optimal point while maintaining separation from less informative regions.

Timestep‑by‑Timestep Injection

We extracted Q from every timestep in the multi‑step extraction process and used each as the one‑step query‑matrix injection candidate, injecting it across the denoising process. The reviewer can refer to the middle of Fig. 5 for a comparable trend (the actual changes across timesteps are even more pronounced). As seen there, the queries from very early timesteps are too coarse, those from very late timesteps overfit to fine details, and only those from the middle range provide balanced, usable structure.

Trade‑offs and Additional Analysis

To our knowledge, there is not a major trade‑off between our one‑step method and multi‑step extraction in most applications. Breaking this down across speed and quality: in terms of speed, as shown in Table 2, one‑step extraction provides a dominant advantage — even more so when compared to inversion‑based approaches. On quality, one‑step extraction combined with LCD offers the ability to isolate and adjust the injected attention matrix as needed, tuning granularity, semantic focus, and strength. This flexibility produces cleaner, less noisy outputs and often better visuals than the multi‑step variant. While multi‑step extraction has one unique edge — by collecting attention at different stages of denoising it can sometimes capture richer information from the reference image for highly specialized cases like blending or computational photography — the raw restoration achievable under LCD can already be tuned to a very satisfying state, further narrowing that gap while preserving the one‑step method’s flexibility and efficiency.

The trade‑offs between timestep condition and latent changes are originally visualized in Fig. 5, offering intuitive understanding of the design choice. We have also run a mini‑batch quantitative ablations of these settings in Table B and Table C, and will expand this into a more comprehensive study in the paper and appendix. We attempted to visualize the model’s internal representations, but found DiT features difficult to reduce into meaningful visuals. Instead, we rely on qualitative examples (as in Fig. 5) combined with new quantitative results and matrix‑similarity analysis to provide a more rigorous discussion.

Table B: Mini‑Batch Quantitative Ablation Analysis

Table C: Mini‑Batch Quantitative Ablation Results on Stylized COCO

(4) RoPE Clarification
The Q matrices FreeControl extracts are captured before RoPE is applied, so the injected queries contain no positional encoding — they are entirely image‑driven. While RoPE still affects K and V during generation, it does not alter what FreeControl injects. Furthermore, FreeControl works identically on U‑Net architectures (which do not use RoPE), showing that structural consistency stems from the extracted queries themselves, not from positional priors.

To directly confirm this point, we ran a controlled test by removing RoPE entirely from the FLUX model. As expected, the base model collapsed into near‑random noise, since it was never trained to operate without positional encoding. Crucially, when we applied FreeControl under the same no‑RoPE setup, the one‑step injection still imposed clear, image‑driven structure on the output. The result looked like “structured noise” faithfully echoing the condition image’s layout — strong evidence that FreeControl’s guidance originates from the injected queries themselves, not from RoPE.

Rebuttal to Reviewer 1

We sincerely thank the reviewer for highlighting areas where additional clarification and analysis would strengthen the paper. We address each concern directly below.

(1) Failure cases for complex or ambiguous compositions
We agree that it is important to analyze the limitations of FreeControl. To this end, we explicitly tested cases involving highly complex or semantically entangled references, and conducted an extensive pressure test under extreme compositional setups to evaluate its robustness.

We began with object occlusion cases, ranging from mild overlaps — where only small regions are covered — to extreme situations where objects are almost entirely obscured with only small portions remaining. In both settings, FreeControl handled the occlusion well, producing results with clear semantic retention and structure preservation.

We then escalated testing to scenes with multiple objects (including humans) crowding into one image, each with different conceptual depths in the scene. Even under these conditions, semantic meaning and structure remained well‑preserved across several trials.

In the most extreme test, where two humans were heavily overlapped and the person behind had only partial limbs visible, we observed that under medium control strength, the model naturally merged them into a single person—producing outputs that remained visually plausible and coherent, with no distorted or unnatural artifacts. However, by slightly increasing the control strength, FreeControl was able to correctly disentangle and restore the separate individuals.

It is worth noting that even in these highly challenging and intentionally destructive cases, FreeControl never produced distorted or unrealistic outputs — for example, no unnatural extra limbs or warped bodies. Across all tests, the results remained natural‑looking and visually coherent. We honestly believe the stability of the method is in a very good state; it remains fairly robust even under stress. (Unfortunately, we can only describe these outcomes in words rather than include images in the rebuttal.)

(2) Scalability beyond FLUX, COCO, and dataset scope
We attempted and implemented FreeControl on other diffusion backbones, including SDXL and SD1.5. As expected, the generation quality is bounded by the underlying model, but the structural control effect remains prominent, demonstrating that the method generalizes effectively.

FreeControl is architecturally agnostic because it operates entirely at the attention level. Porting to new backbones only requires trivial code changes and light adaptation to the new model structure. It would, admittedly, take more time to tune the hyperparameters and carefully check the details to bring the method to a stable and robust state—particularly with models like SD 1.5, which are inherently less stable. We also note that the appendix shows adaptation to LoRA and fine‑tuned models, further underscoring generality.

As for the dataset, we initially consider the COCO dataset to be a sufficiently large dataset that could fairly evaluates the overall performance. To provide additional angles on flexibility, the appendix also includes experiments on a stylized prompt version of COCO. During our experiments on various other data sources, our method delivered stable outcomes (even on roughly assembled compositional images with artifacts).

We recognize that explaining these points explicitly is key to showing the method’s applicability, and we will add a new section in the appendix detailing these cross‑model experiments, dataset considerations, and engineering notes, making the portability and robustness of FreeControl clearer to readers.

On the other hand, if your concern is less about generalization and more about benchmarking on other datasets to improve representativeness, we are already working on such an evaluation and will be happy to provide it once it is complete.

Additionally, we have included new minibatch experiments with expanded baseline comparisons and quantitative ablations in response to another reviewer’s request. A complete version with formal discussion will be included in the full paper. Please feel free to check the current results below and the discussions could be found in the responses to other reviewers — we’d be happy to hear your thoughts!

Table A: Mini‑Batch Expanded Baseline Comparison Results

Table B: Mini‑Batch Expanded Baseline Comparison Results on Stylized COCO

Table C: Mini‑Batch Quantitative Ablation Results

Table D: Mini‑Batch Quantitative Ablation Results on Stylized COCO