Pin the Tail on the Model: Blindfolded Repair of User-Flagged Failures in Text-to-Image Services

We thank you for noting the strengths of our paper, namely:

• Significant novelty and practical orientations
• Addressing a pressing problem with real-world challenges
• Clever optimizations

We fixed all typos and respond below to all questions raised.

paper is not very friendly to those who are not familiar with cryptography

Thank you for your suggestions! We will add more crypto preliminaries in the text. Here's an intuitive summary of how privacy is preserved and the related concepts.

Secure Multi-Party Computation (MPC): MPC enables multiple parties to jointly compute an agreed-upon function of their inputs. The protocol guarantees that no information beyond the output is revealed. 2PC (Two-Party Computation) is the specialized case for two participants. We use a 2PC protocol for the Matching Phase. The 2PC matching function takes the provider's matching key (semantically, “fix gender bias in nurse”) and the institute's set of database keys as inputs. It only reveals the index of the correct match to the service provider. The index is later used to retrieve the fix matrix via OT.

Oblivious Transfer (OT): Oblivious Transfer is a protocol between a sender and a receiver. The sender has multiple messages, and the receiver wants to retrieve one of them. OT’s privacy guarantee ensures that (1) the sender learns nothing about which item the receiver chose, and (2) the receiver learns nothing about the items they didn't choose. Our protocol uses OT in the Model Repair phase. The service provider (receiver) privately retrieves the specific $W_{fix}$ matrix it needs from the repair institute's (sender's) database without revealing which one it picked.

Semi-Honest and Malicious Security: A semi-honest (or honest-but-curious) adversary correctly follows the protocol specification but may attempt to infer additional information. A malicious adversary may arbitrarily deviate from the protocol. We implement and evaluate our protocol with semi-honest security, but our protocol can also be extended to malicious security without significant overhead. We estimated the maliciously secure version of our protocol and present the numbers in the table below.

How privacy is preserved: Our protocol works in two main steps: (1) private matching via 2PC and (2) private fix retrieval via OT. The privacy is preserved directly through the security properties of 2PC and OT primitives. Our protocol is formally proven secure under the standard simulation paradigm. The core idea is to first define a hypothetical, ideal-world trusted party that handles the entire model repair (Figure 2). By definition, this process is perfectly secure. Then, the proof shows that our real-world protocol “behaves the same” as the ideal-world trusted party. Since the ideal functionality is secure by definition, it follows that our protocol is just as secure and leaks no unwanted information.

Why service providers can't be repair institute? What is the challenge of the model repair task?

The primary challenge in model repair is not the computational cost of applying a fix but the accurate diagnosis of a failure from limited user feedback. We argue it is challenging for service providers to repair their model for several key reasons:

1. Insufficient Feedback for Diagnosis: Modern generative models typically rely on simple feedback mechanisms like a "thumbs up/down" button[Ref. 2]. This single bit of information is ambiguous and lacks the context needed to identify a specific failure. Without extensive expertise of user reports from the repair institute, interpreting a "thumbs down" is guesswork.
2. Lack of Specialized Domain Knowledge: Even with feedback, service providers cannot possess the deep domain expertise required to pinpoint subtle errors. They may not be familiar with the nuances of niche topics (e.g., the inaccurate details of a sports car), the evolving standards for evaluating social bias, or the specifics of a factual inaccuracy. They can't fix a problem they don't have the expertise to understand.
3. Inability to Keep Knowledge Current: A model's implicit knowledge can become outdated overnight due to news, cultural events, or even a celebrity changing their hairstyle, or other significant events after its knowledge cutoff. It is not the core business of a service provider to track these real-world changes, making it difficult for them to identify and repair obsolete information.

For example, consider a user who prompts for an image of the "latest iPhone" shortly after a new model is released and gives a "thumbs down" to the generated image of an older model. A service provider, seeing a plausible modern smartphone, would be unable to diagnose the error from this vague feedback. They may lack the knowledge to distinguish the subtle design changes between models, and it is not their core business to track real-time product releases. In contrast, a specialised repair institute would immediately recognize that the model's knowledge is outdated and provide the correct fix.

Can this method be applied to erasing method that does not have a closed-form solution? Can this method be applied to methods with very complex closed-form solution?

Our method cannot be directly applied to iterative methods, as our focus is on creating a protocol that is both crypto-friendly and highly efficient. While such methods could theoretically use generic secure computation, our baseline shows that even simple edits become prohibitively inefficient, making iterative approaches infeasible in practice. However, we note that our framework is highly versatile and supports advanced, state-of-the-art editing methods. Closed-form editing solutions handle erasing tasks very well, and our protocol supports highly complex closed-form solutions. The core principle is that as long as the final weight update can be expressed as $W’ = WW_{fix}$, our protocol can apply it, regardless of the complexity involved in deriving $W_{fix}$. The follow-up work on TIME titled "Unified Concept Editing in Diffusion Models" (UCE) [Ref 1] is the perfect example of this. UCE provides a more complex closed-form solution that is capable of repairing multiple targets in a single edit, which contains a set for desired fixes and a specific “preservation term” to further prevent undesirable editing artifacts. We observed that the closed-form formula of UCE can also be refactored, and the fix can be encapsulated into a fix matrix $W_{fix}$ of the same size. Therefore, our protocol can directly support it without additional overhead.

Specifically, UCE gives the following closed-form update formula (equation 7 in the paper): $W =\Bigl(\sum_{c_i\in E} v_i^* c_i^\top+\sum_{c_j\in P} W_{{old}} c_j c_j^\top\Bigr)\Bigl(\sum_{c_i\in E} c_i c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)^{-1}$ where $E$ and $P$ are sets of editing target and preserving target and $v^* = W_{old}c_i^*$. We can refactor the formula in a similar way by plugging in $v*$ $W = W_{{old}}\Bigl(\sum_{c_i\in E} c_{i}^* c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)\Bigl(\sum_{c_i\in E} c_i c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)^{-1}$ , where $W_{fix}$ is the terms in brackets. Notice that, like TIME, computing the $W_{fix}$ is independent of $W_{old}$, and therefore can be prepared by the repair institute before any interaction.

Effectiveness of the proposed method: does the encrypted editing method (TIME) still works? Need qualitative and quantitative results

Our method is instantiated by TIME and can also support its more powerful follow-up UCE as discussed above. Crucially, our method does not change the underlying editing algorithm itself. In our experiment, all numbers are represented using standard single-precision floating point, so the numerical accuracies will be identical as well. Therefore, the utility and effectiveness of an edit applied via SURE are identical to those of the original algorithm, and the comprehensive evaluations of the original TIME and UCE paper can be directly adopted to conclude the effectiveness of SURE, which we briefly summarize below.

Qualitatively, the TIME algorithm is shown to work on a wide range of tasks. It can change object attributes like making roses blue, update factual information such as making a character blond, and apply various stylistic changes. Quantitatively, TIME demonstrates high efficacy in making the intended edit and strong generality in applying that change to related concepts. While there is a slight trade-off, it largely maintains specificity, leaving unrelated images untouched. Importantly, standard image quality metrics like FID and CLIP scores remain stable, which means that the overall model quality is preserved. Furthermore, our protocol supports the more advanced UCE algorithm, which extends these capabilities to complex tasks. UCE can perform multi-attribute debiasing for gender and race and is powerful enough to erase up to 100 artistic styles simultaneously without significant degradation to the model.

[Ref 1] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.

[Ref. 2] Collins et al. (2024). Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation.

Thank you for your review and valuable feedback! We want to clarify a crucial point that addresses your primary concerns (Questions 1-4). Our work introduces SURE, a novel, privacy-preserving protocol for applying model edits, which does not alter the underlying mathematical editing algorithm. Therefore, the utility and effectiveness of an edit applied via SURE are identical to those of the original algorithm. We will add a discussion to clarify this in our manuscript. To address your concern regarding the effectiveness of TIME's “one fix fits all” nature and demonstrate how our flexible protocol can benefit from future algorithmic improvements, we consider the follow-up work Unified Concept Editing (UCE) [Ref. 1]. UCE generalizes TIME's closed-form formula to handle multiple, distinct fixes in a single update. Our protocol fully supports this multi-target editing without loss of efficiency, as the resulting $W_{fix}$ matrix prepared by the repair institute is the same size. Specifically, UCE gives the following closed-form update formula (equation 7 in the paper): $W =\Bigl(\sum_{c_i\in E} v_i^* c_i^\top+\sum_{c_j\in P} W_{{old}} c_j c_j^\top\Bigr)\Bigl(\sum_{c_i\in E} c_i c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)^{-1}$ where $E$ and $P$ are sets of editing target and preserving target and $v^* = W_{old}c_i^*$. We can refactor the formula in a similar way by plugging in $v*$ $W = W_{{old}}\Bigl(\sum_{c_i\in E} c_{i}^* c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)\Bigl(\sum_{c_i\in E} c_i c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)^{-1}$ , where $W_{fix}$ is the terms in brackets. Notice that, like TIME, computing the $W_{fix}$ is independent of $W_{old}$, and therefore can be prepared by the repair institute before any interaction.

Given that our protocol faithfully applies these algorithms, the extensive evaluations from the TIME and UCE papers directly answer your questions (1-4) about the editing method's performance. Both papers provide extensive quantitative and qualitative evaluations for (1) efficacy, which measures how effective the editing method is on the editing prompt, (2) generality, which measures how the editing method generalizes to other related prompts, and (3) specificity, which measures the ability to leave the generation of unrelated prompts unaffected. We summarize these results below.

[Ref 1] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.

Q1: Generality of one fix fits all.

Both TIME and UCE show extensively that this closed-form formula is very effective for a wide range of repair tasks (including object removal and style alteration). For example, it could

• modify object attributes: (e.g., changing a rose's color, a pizza's shape)
• completely remove concepts (removing a "garbage truck" from the model's knowledge)
• apply style alteration (erase artistic styles (removing the influence of a specific artist)
• handle complex, multi-attribute debiasing.
• perform combinations of these edits simultaneously without a loss in effectiveness.

Q2: Impact on Unrelated Generation Tasks

Modifying all layers does not harm unrelated images. The editing is highly targeted to only a small fraction of the model's parameters (2.2% for Stable Diffusion 1.4). In both TIME and UCE, evaluations confirmed that:

1. Modifying specific concepts does not affect others. For example, editing "roses" to be blue does not change the colour of "poppies".
2. TIME does not degrade the overall generative quality of the model after editing (wrt FID and CLIP Score)
3. UCE achieves high fidelity and minimal distortion to generations for unaddressed tasks, even after extensive batch edits.
4. UCE’s editing formula contains a specific “preservation term” to further prevent undesirable editing artifacts.

Q3: Lack of In-depth Qualitative Evaluation/Repair Examples

The TIME and UCE paper presents extensive before-and-after image comparisons demonstrating its effectiveness, generalizability, which includes changing object attributes ("a pack of roses" to "blue roses"), context ("a cow" to "a cow on the beach"), and correcting biases. They also demonstrate that edits generalize logically to related prompts (e.g., editing "house" also affects "an oil painting of a house"). They also provide visual examples that confirm that unrelated prompts remain pristine (e.g., "poppies" are unchanged when "roses" are edited). We are happy to add them to our paper if you think it is useful.

Q4: Insufficient Verification of Failure Type Diversity

The effectiveness and generalisation abilities of TIME and UCE directly apply to SURE. Their evaluations and visual examples show that the editing generalises very well beyond bias correction while having little undesirable artifacts on unaffected prompts.

1. Outdated information: TIME fixes outdated information, such as when a celebrity changed their hairstyle. They edit the Harry Potter Franchise character "Hagrid" to become "Blond Hagrid". It shows that it generalizes well for prompts containing the concept of Hagrid (e.g. “a painting of Hagrid”) and doesn't affect other Harry Potter characters.
2. Factual Inaccuracies: Both TIME and UCE can override models’ inaccurate "implicit assumptions" about object attributes and align them with desired factual alterations, including altering colors, shape, materials, and completely erasing the concept of an object.
3. Stylistic Changes: TIME can induce stylistic shifts by modifying conceptual attributes (e.g., changing the material of a house). UCE can remove specific artistic styles, and experiment shows it can erase up to 100 artists simultaneously before significantly impacting image fidelity or CLIP scores on unrelated concepts.

Q5: costs with scaling repair database: construction, efficient retrieval, and when multiple $W_{fix}$ entries apply or conflict?

Regarding scalability, we note that the offline costs for constructing the database are generally negligible compared to the cryptographic online cost, as they only involve standard plaintext matrix operations, which are inexpensive and highly parallelizable on modern hardware.

As for the online cost, the database size will not affect the efficiency of retrieval much (model repair phase in Figure 4). The retrieval only requires $\mathcal{O}(\log n)$ number of oblivious transfers: concretely, for 1 billion entries, it will only require 30 oblivious transfers, although the final running time will depend on the bandwidth, as communication unavoidably will scale linearly with the database size. Database size will mainly affect the cost of the matching phase, but our protocol is modular to handle this: for practical, domain-specific databases of moderate size, a simple linear scan is concretely efficient. For massive-scale databases, the matching component can be swapped with advanced sub-protocols like Private Information Retrieval to achieve sub-linear performance that can easily handle millions of records.

For conflict resolution, we break ties by entry index, mainly for technical consistency. In practice, for a well-curated database with a precise similarity metric, it is unlikely that equally similar entries have significant conflicts in fixes. Furthermore, if a conflicting or low-quality fix were applied, the service provider could readily detect the resulting model degradation through local benchmarking.

Q6: "Blindfolded Repair" Context and Limitations: the scope of privacy and adversarial models, especially concerning malicious attackers or sensitive PII in user feedback.

The title "Blindfolded Repair" refers to the privacy guarantee of our protocol: the repair institute provides fixes without learning the model weights or which specific fix a service provider is requesting, and the service provider obtains a needed fix without learning anything else about the repair database or the underlying repairing algorithm.

Our implementation is under the semi-honest threat model (line 122) and we specified our assumption for each party in Section 3. Regarding sensitive PII, we assume the user voluntarily hands their feedback to the service provider, but the service provider must not disclose this to any third party (in practice, this is reasonable under privacy regulations). For security against malicious adversaries, our protocol can be easily extended to a malicious setting by (1) replacing cryptographic primitives (2PC and OT) with their malicious secure counterparts and (2) a lightweight consistency check between the matching output and the OT input to ensure a Service Provider correctly uses the index from the Matching phase in the subsequent steps. We estimate the cost of a maliciously secure version of SURE and compare it with a maliciously secure baseline (implemented using the MASCOT protocol from MP-SPDZ). We summarize the running time in the table below.

Malicious security increases the total runtime by ~9x. This overhead comes almost entirely from the Matching phase executed in malicious 2PC. However, in applications where the Matching phase is not required (as noted in footnote 3), our protocol bypasses the costly 2PC. In this case, the consistency check can also be avoided, and the $\mathcal{O}(\log n)$ numbers of malicious OT adds virtually no overhead.

Thank you for noting the strengths of our paper:

• well-motivated and real-world problem;
• end-to-end efficient solutions with tight security guarantees.

We address your questions below.

The paper does not discuss malicious-security guarantees or expected change of cost (compared to semi-honest setup) but this kind of setting might be required for a real-world use case. How would the protocol/runtime/cost have to change to change to a maliciously secure setting?

Our protocol’s modular design allows for a direct extension to the malicious security setting. The overall protocol structure will largely remain the same and consists of two minor changes: (1) replacing the semi-honest cryptographic primitives with their maliciously secure counterparts and (2) adding a lightweight consistency check to ensure the potentially malicious Service Provider will provide the same index it retrieved from the matching phase to the OT functionality. Because the matching circuit and number of oblivious transfers are both very small, switching to their malicious version will not blow up the overall runtime. Additionally, the consistency check has a constant cost independent of the database size.

We estimate the running time of malicious secure SURE using Malicious OT and 2PC subprotocol. We also report the malicious version of the baseline (using the mascot protocol variant from the MP-SPDZ). Maliciously secure SURE increases the total runtime by ~9x compared to our semi-honest implementation. Note that the overhead is almost entirely incurred by the Matching phase, which runs in a malicious 2PC. In contrast, the Model Repair phase, which only requires a few oblivious transfers, incurs very little additional cost. We comment that malicious security overhead can be avoided in scenarios where the Matching phase is not needed (as we discussed at Line 329, footnote 3). In such cases, the protocol only needs a few malicious OTs and can skip the expensive 2PC and the consistency check; the cost of malicious security introduced by O(logn) malicious OT becomes negligible.

Stable Diffusion v1.4 has a significant gap between more recent generation models that are typically deployed and might need repairs in a real-world setup. Runtime analysis for larger models such as SDXL or SD3 might be necessary to demonstrate generalizability.

We note that our work, SURE, is the first privacy-preserving protocol to support complex, real-world diffusion models. We view identifying repair algorithms that can be made crypto-friendly as one of our main contributions, as it's rare for cryptographic protocols to scale to non-trivial, real-world machine learning models of this size without resorting to approximations that would degrade performance. Our protocol achieves this efficiently, performing repairs in seconds. We chose Stable Diffusion v1.4 specifically because it is a widely-used, large-scale model that serves as a common benchmark in both academic research and deployed products, and it is the benchmark model used in the foundational editing literature (TIME, UCE) that our protocol implements. This allows us to directly validate that SURE applies edits with zero loss of utility compared to the original, non-private algorithms. While Stable Diffusion v1.4 serves as our validation target, our protocol is designed to scale efficiently to even larger models. Its scalability can be analyzed by two factors:

Models with more layers: A key advantage of our protocol is that its cost is independent of the number of layers being edited. The fix matrix $W_{fix}$ is retrieved once through the interactive protocol, and the service provider can then apply this same fix to all relevant layers of the model locally, without any further interaction. In contrast, the baseline edits each layer with a separate execution of a secure 2PC protocol for matrix operations, which makes the total cost scale linearly.

Models with larger weight matrices: Larger models could have higher-dimensional weight matrices, which increase the size of the corresponding $W_{fix}$ . In our protocol, this only impacts the communication cost. The total number of Oblivious Transfer (OT) operations remains the same, but the size of the payload within each OT message increases linearly with the size of the $W_{fix}$.

Have you run any quantitative or qualitative studies to show that a patched model actually corrects the user-flagged error without degrading other outputs?

The unique feature of TIME is that it does not change the underlying Model editing logic/operation at all. Although we did clear refactoring to the editing algorithm, it only makes it more cryptography-friendly while maintaining the identical effect. Therefore, our method provides the exact same utility as the original algorithm. The quantitative and qualitative studies demonstrating that TIME successfully corrects errors without significant degradation to other outputs can be found in the original paper and its follow-up work.

Is there a possibility of a stronger choice of baseline other than a full linear-scan MPC?

As we discussed in Appendix C (line 828), our baseline already includes significant optimizations for matrix operations to be a fair point of comparison. Specifically, we allow the repair institute to locally pre-compute parts of the repair formula. This strategy eliminates the most expensive matrix inverse operations and reduces the number of matrix multiplications inside the 2PC protocol. Despite these optimizations, the secure matrix computation remains the overwhelming bottleneck: for a database of 500 entries, the secure edit takes around 167 hours while the linear scan only takes ~500 seconds. This shows that even if we were to use a more customized protocol like Private Information Retrieval to make the matching phase faster, the baseline would remain impractical.

Is it possible to apply several fixes to one prompt at a time? Does the cost increase in this case?

Thanks for the great question. Yes, our protocol directly supports multiple fixes through a single protocol execution, and the core cryptographic cost does not increase. The only potential cost increase relates to the size of the repair database, which is a manageable trade-off.

Our protocol is compatible with follow-up work on TIME, “Unified Concept Editing in Diffusion Models” [Ref 1], which generalizes the closed-form solution to incorporate multiple editing targets into a single update. Their formula can also be refactored using our optimisations as $W’ = W_{old} W_{fix}$ in a similar way, which makes it compatible with our protocol. Specifically, UCE gives the following closed-form update formula (equation 7 in the paper): $W =\Bigl(\sum_{c_i\in E} v_i^* c_i^\top+\sum_{c_j\in P} W_{{old}} c_j c_j^\top\Bigr)\Bigl(\sum_{c_i\in E} c_i c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)^{-1}$ where $E$ and $P$ are sets of editing target and preserving target and $v^* = W_{old}c_i^*$. We can refactor the formula in a similar way by plugging in $v*$ $W = W_{{old}}\Bigl(\sum_{c_i\in E} c_{i}^* c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)\Bigl(\sum_{c_i\in E} c_i c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)^{-1}$ , where $W_{fix}$ is the terms in brackets. Notice that, like TIME, computing the $W_{fix}$ is independent of $W_{old}$, and therefore can be prepared by the repair institute before any interaction.

When applying UCE using our protocol, each $W_{fix}$ encodes multiple editing targets. The cost can be analyzed in two parts: The core cryptographic cost of our protocol involves securely retrieving one $W_{fix}$ matrix. Since the multi-edit $W_{fix}$ ​has the same size as a single-edit one, the computation and communication for one run of the protocol do not change at all. Increased database size: The potential cost increase comes from the growth of the repair database. To support combined edits, the database must store pre-computed entries for those combinations. For example, a simple database might have entries for fixing gender bias for “nurse”, “engineer”, and “cook”. But to support multiple edits, the database would need to store all possible combinations of these three professions. However, our protocol is modular to make this manageable. For extremely large databases, the default matching component can be swapped with advanced solutions like Private Information Retrieval (PIR), which reduces the matching cost to be sub-linear in the database size.

[Ref 1] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.

We thank you for noting the strengths of our paper:

• interesting and well-motivated problems;
• novel and practical solutions.

We respond below to all questions raised.

What are the assumptions on the editing algorithm to fit this framework to reduce the overhead?

Can this framework support more general repair requests, like align a prompt with a (batch of) images, and general repair methods, like a fine-tuning based method? How can another editing method be plugged-in this framework? What are the requirements?

TL;DR: Our protocol assumes the editing can be expressed as a linear transformation of the model’s weight. These editing methods can all be made crypto-friendly through our optimisation. Our protocol directly supports advanced, state-of-the-art editing algorithms like UCE [Ref.1], which can perform batch alignment.

In theory, it is feasible to execute any editing algorithm securely in 2PC. However, as we show in our baseline, even for TIME, directly plugging it into a 2PC protocol incurs prohibitive costs. To make this practically feasible and mitigate the substantial computational overheads inherent to both 2PC and knowledge editing algorithms, we show that optimisations on both machine learning and cryptography fronts are essential: i) designing a crypto-friendly editing algorithm; ii) developing a customised secure protocol tailored to it.

In this paper, we instantiate one such category of crypto-friendly algorithms using a closed-form repair formula that satisfies two key properties: Enabling expensive operations to be performed offline, entirely eliminating matrix operations within the 2PC protocol; Being agnostic to the number of layers, allowing the same fix to be reused across all layers.

As formalized in Definition 1, this class includes any editing algorithm that can be refactored as applying a fixed linear transformation to a layer. In particular, we refactor TIME editing algorithm (state-of-the-art in knowledge editing as it is highly efficient and offers a closed-form solution) such that its repair modifies the weight matrix by right-multiplication it with a “fix matrix”: $W’ = WW_{fix}$.

Without incurring any additional costs, our framework can support multiple edits at the same time using "Unified Concept Editing in Diffusion Models" (UCE) [Ref 1]. UCE is capable of repairing multiple targets in a single edit: it can erase up to 100 artists simultaneously before significantly impacting image fidelity or CLIP scores on unrelated concepts. UCE has a more complex closed-form solution that is capable of repairing multiple targets in a single edit, which contains a set for desired fixes and a specific “preservation term” to further prevent undesirable editing artifacts. We observed that the closed-form formula of UCE can also be refactored, and the fix can be encapsulated into a fix matrix $W_{fix}$ of the same size. Therefore, our protocol can directly support it without additional overhead.

Specifically, UCE gives the following closed-form update formula (equation 7 in the paper): $W =\Bigl(\sum_{c_i\in E} v_i^* c_i^\top+\sum_{c_j\in P} W_{{old}} c_j c_j^\top\Bigr)\Bigl(\sum_{c_i\in E} c_i c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)^{-1}$ where $E$ and $P$ are sets of editing target and preserving target and $v^* = W_{old}c_i^*$. We can refactor the formula in a similar way by plugging in $v*$ $W = W_{{old}}\Bigl(\sum_{c_i\in E} c_{i}^* c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)\Bigl(\sum_{c_i\in E} c_i c_i^\top+\sum_{c_j\in P} c_j c_j^\top\Bigr)^{-1}$ , where $W_{fix}$ is the terms in brackets. Notice that, like TIME, computing the $W_{fix}$ is independent of $W_{old}$, and therefore can be prepared by the repair institute before any interaction.

While both TIME and UCE algorithms use a single fix matrix for all edited layers, our protocol easily and naturally generalises to support layer-specific fix matrices with only negligible additional cost. Specifically, the Oblivious Transfer (OT) cost will slightly increase with the number of matrices, but the dominating key matching phase will remain constant. This means the total overhead remains low even when handling multiple distinct fix matrices.

We will summarize and incorporate this clarification into the paper accordingly.

Can the protocol be extended so that the service provider does not lose access to prior repairs even if the model repair institutes stopped the service? For example, allowing the service provider to store the database and do the database matching offline?

That is an excellent point regarding the long-term availability of repairs. Our protocol inherently supports this need, although through a different mechanism than the one suggested. While the repair database itself is the intellectual property of the repair institute and cannot be shared (see Section 3, line 141), our framework offers a more direct solution. Once the service provider securely obtains the fix through our protocol, they can store it locally and retain it indefinitely. This stored fix can then be reapplied to the current model at any time. Moreover, it can even be applied across multiple different future models that require the same correction, without any further interaction with the repair institute. Therefore, even if the repair institute were to stop its operations, the service provider would not lose access to any previously acquired repairs. This is in contrast to the generic baseline, which requires a costly interactive 2PC execution for every single model repair, even if the fix has been previously applied and the model is the same. We appreciate the insight and have highlighted this important property in our problem description.

Minor presentation suggestions

Thank you for your careful reading and helpful suggestions regarding the presentation of our paper. We have incorporated both presentation suggestions.

[Ref 1] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.

I still have concerns about the applicability of the proposed decomposition for most of the model repair methods, which largely limits the proposed decomposition to be a general framework. The proposed decomposition requires the repair method to be expressed as a closed-form parameter update, where most of the computation is in a matrix multiplication that can be decomposed. Such assumption excludes all gradient-descent-based repair methods, and limits the type of repair requests/specification to the ones that are supported by a subset of methods that satisfies this assumption.

Furthermore, the submission is written specifically for TIME, lacks description and discussion as a method-agnostic general framework as well as evaluation on common methods.

Therefore, I suggest the authors to rephrase this work as an efficient 2PC adoption of the prior TIME method instead of a general framework.