“...editing and inverting have been widely studied in the field (see Weaknesses)” and “These methods have covered the scenarios including latent sampling, editing, and inverting and thus should be discussed” and “The methodological level design is also not entirely new and intriguing.”

We agree that latent sampling, editing, and inverting has been studied extensively. However, we believe there might be a misunderstanding. We explore these applications on model weights, not input noise or latents. weights2weights space can be thought of as a meta-latent space, with “applications analogous to those of a generative latent space – inversion, editing, and sampling – but producing model weights rather than images” (L316-318 in paper). We agree that the works you bring up are relevant, and we will include them in the related works. However, they focus on image manipulation, operating on the image manifold. As pointed out by reviewers rzPz and wMSF, we model and operate on the “model weight manifold,” manipulating the models themselves in order to produce new models. To the best of our knowledge, we believe this idea of a meta-latent space over model weights is novel. Furthermore, we add to this contribution by finding semantic weight subspaces with PCA and linear classifiers. Our analysis of the model weights shows that we can get valid identity-encoding diffusion models via linear interpolation among the weights of existing diffusion models spanning the subspace.

We contrast sampling, editing, and inverting in the traditional sense with our idea of operating on model weights in Figure 2 of our paper. 1) Sampling in the traditional sense will produce a latent that results in a single image. Sampling from w2w space produces a new diffusion model that can generate an infinite number of different images of the same identity. 2) Editing an input such as a latent results in one edited image. In contrast, edits in w2w space result in a new model with edited weights such that attributes of the identity that this model generates are changed. Once the edit is conducted, we can generate new images from the model and the edited identity is consistent (e.g., now always has a beard). Throughout the paper, we show results of generated images before and after the model weight edits and observe that the generated images are largely unchanged except for the identities. This is to show that after editing the weights, the model is largely unaffected except its identity. 3) Inversion into the input space in generative models reconstructs a single image. However, inversion into w2w space produces a new model based on the identity in the input. We can then indefinitely sample new images from the model which consistently generates that identity. Furthermore, with the OOD projection, we are distilling a realistic identity from the input image into a new model.

“... resulting in inaccurate claims on ‘Alas, in multi-step generative models, like diffusion models, such a linear latent space is yet to be found.’”

Thank you for pointing this out. We agree that this sentence is poorly phrased. The point we meant to get across is nicely summarized in paper [a] that you brought up: “In contrast to a single unified latent space in GANs [the multistep process of diffusion models] imposes extra difficulties for studying the latent spaces of DDMs.” Still, our motivation for studying the weight space and modeling its manifold remains. The fact that personalization approaches can insert concepts into the model weights without disrupting the prior of the model suggests that we can find such subspaces. This inspires us to develop the weights2weights as a meta-latent space controlling model weights. We will clarify this in the manuscript.

“The authors put much more emphasis on the empirical investigations on the interpretability of latent spaces in DMs, but neglect the dimension of diffusion time…”

We agree that analysis of diffusion timesteps is interesting with regards to the latent or noise spaces in DMs. However, we are analyzing the weights, which are fixed across timesteps. The identity and attribute subspaces we find are in the weights, and the applications such as inversion, sampling, and editing produce new weights parameterizing a new model which is fixed across timesteps.

“... it would be helpful to perform some user studies as subjective evaluations…”

Thank you for your suggestion! We conducted user studies for identity inversion in rebuttal PDF Table 2 and identity editing in rebuttal Table 3. We discuss the details of these in global responses 3 and 4. Overall, users have a strong preference towards weights2weights for identity inversion and editing compared to baselines.

“...other than identity context…it may strengthen the paper if the authors could extend their application scenarios to a wider context.”

We agree with your suggestion. Following your comment, we create a weights2weights weight subspace for dogs and present the results in rebuttal PDF Figure 3. We conduct Dreambooth fine-tuning for 120 different dog breeds and train linear classifiers to define separating hyperplanes in weight space for different semantic attributes, such as “large” or “wavy fur.” In rebuttal Figure 3, we sample two models from this space and edit the weights to change attributes of the encoded breeds.

"...I acknowledge that using model weights as datasets is interesting…‘scaling up’... one of the biggest contributions to me.”

Thank you for acknowledging the scale of our weight space analysis! We will open source our code and release the weights for the community.

“The construction of w2w spaces needs over [60,000 models]...The compute resources should be provided…”

Training a single identity LoRA with rank 1 for our dataset of models requires ~8GB VRAM, and takes 220 seconds on a single A100 GPU. Please refer to Appendix C for details.

“And is there a contained solution to balance computing resources and performance.”

Thanks for the great question. We ablate the number of models used to construct w2w space in Figure 9 of the paper, and find that a subspace spanned by 10,000 models retains most of the performance in ID preservation with a score of 0.53 versus 0.65. The score of 0.65 is achieved by using more than six times the number of models. We highlight that although creating this space requires a notable amount of compute, using the space for the variety of applications is lightweight. For instance, inverting an identity only requires one image and uses 1/10 the number of parameters (just 10000) and 1/4 the time (under 55 seconds) for typical Dreambooth LoRA finetuning (see Appendix C).

“The paper uses several evaluation metrics…nuanced attributes related to human faces”

We agree with your point on the limitations of the reported metrics, acknowledging it in L226-227: “...score can be noisy...too coarse to describe attributes as nuanced as those related to the human face.” We utilize these metrics since they are commonly adopted for evaluating edits and identity preservation. Following your point, we present user studies for identity inversion in Table 2 and identity editing in Table 3 of the rebuttal PDF. Overall, w2w is largely preferred over baselines. We discuss this further in Global Responses 3 and 4.

“...are the first m principal components identity attributes…?”

That is an interesting question! Thank you for bringing it up. We provide a visualization of traversals along a set of principal components in Figure 4 of the rebuttal. The principal components change attributes of the identity, although various semantic attributes are entangled. For instance, the first PC seems to be changing age, hair color, and hair style. The second PC appears to change gender and skin complexion. The third PC seems to change age, skin complexion, and facial hair. This motivates our use of linear classifiers to find separating hyperplanes in weight space and disentangle these attributes.

“...some recent works present the way of generating identity…given only one reference image…provide the comparison between this paper and these works.”

Thank you for your suggestion. In the rebuttal PDF Table 1, we compare against Celeb-Basis [1] and IP-Adapter FaceID [2], following the same evaluation protocol used for Table 2 in the main paper. We further conducted a user study comparing with [1] and [2], which is presented in Table 2 of the rebuttal PDF. We show qualitative comparisons against these two methods in rebuttal PDF Figure 6. Details are provided in Global Responses 2 and 3. Across all these metrics, w2w performs stronger than Celeb-Basis and IP-Adapter FaceID. Our results indicate that operating in the weights2weights weight subspace is highly expressive as it is able to faithfully capture identity without overfitting to the input image. This enables diverse generation of the identity.

“LoRA training often affects the performance of the base model to some extent…”

We are aware of this phenomenon known as “drift” as introduced in Dreambooth [4]. We agree with your concern of LoRA training affecting the base model, but we believe this is not necessarily an issue of weights2weights itself, but fine-tuning in general. Section 4.3 in Dreambooth [4] and Section 4.4 in Custom Diffusion [5] already run experiments indicating that the use of prior-preservation regularization limits the undesirable change of the base model. That is why we use the prior preservation loss introduced by Dreambooth to avoid losing the priors of the base model. Furthermore, the w2w models inherit the rich priors of the base model as shown by composition of the identities in novel contexts such as posing with a famous figure in Figures 6, 7 of the paper. These identities can also be converted into new styles with prompts like “statue” or “painting” as seen in Figures 7, 8 in thr paper. More examples demonstrate this compositionality throughout the qualitative examples in the appendix.

“The overall idea of this paper is more similar to [1]...I suggest the authors could discuss more about the differences between this paper and [1].”

Although these two papers share the use of PCA and the application to personalization, there are fundamental differences. Many works have found subspaces in the input space, such as text embeddings [1], to produce meaningful linear interpolations. But as you point out, we use PCA to discover a subspace within the weights themselves, suggesting that weights can also be interpretable spaces. Our analysis of the model weights shows that we can get valid identity-encoding diffusion models via linear interpolation among the weights of existing diffusion models spanning the subspace. For instance, our w2w inversion essentially finds the best linear combination of existing diffusion model weights to define a new customized diffusion model, which, in turn, can generate an unlimited number of identity-consistent images of that novel person.

Furthermore, beyond personalization, which is the main focus of [1], we demonstrate two additional applications of this subspace. We also develop methods for sampling and editing identities, which “expands the scope of manifold applications” (rzPz). We show how to sample new models each of which encodes a novel identity (Sections 3.2, 4.2), and edit the models (Sections 3.3, 4.3), which as a result, edits the identities encoded in them. As such, weights2weights space can be thought of as a meta-latent space as its applications are “analogous to those of a generative latent space – inversion, editing, and sampling – but producing model weights rather than images” (L316-318 in paper). We will make sure to clearly highlight these differences with Celeb-Basis in the related works section of the paper, where we briefly introduced Celeb-Basis.

“...similar phenomenon to mean person in [1]?”

w2w has a mean person, as shown in rebuttal Figure 5. These images are generated by a “mean model” using the averaged weights of models spanning w2w space.

“Have the authors studied the effect of PCA on the generation of results?”

Thank you for the suggestion. We present an ablation on the number of principal components used during w2w inversion in rebuttal PDF Figure 1. We vary the number of principal components used during identity inversion, and measure the average ID score (higher is better) over 100 inverted FFHQ evaluation identities. The ID score monotonically increases until 10,000 principal components, after which it starts to decrease. We qualitatively visualize this phenomenon in rebuttal Figure 2, where each column represents a fixed seed. 20,000 PCs overfits to the input image and pose, with varying seeds producing only face shots with artifacts which degrade the identity. 1000 PCs underfits the original identity, but shows diversity across different generation seeds. 10,000 PCs fits the identity without overfitting.

...I am concerned about real-world applications… what are the advantages of w2w compared to existing personalization efforts…”

We present comparisons against variants of Dreambooth in Table 2 in the main paper, showing how given only a single image, w2w can achieve competitive performance with a method which uses multiple images. Following your suggestion, we compare quantitatively against other personalization efforts, Celeb-Basis [1] and IP-Adapter FaceID [2], in rebuttal Table 1. We follow the same evaluation protocol used for Table 2 in the main paper. We further conduct a user study presented in Table 2 of the rebuttal PDF. We show qualitative comparisons against these two methods in rebuttal PDF Figure 6. Details are provided in Global Response 2 and 3.

Overall, across all these metrics, w2w performs stronger than Celeb-Basis and IP-Adapter FaceID. Our results indicate that operating in our weight subspace is highly expressive and flexible compared to text embedding space as it is able to faithfully capture nuanced identity without overfitting to the input image. Based on the user study, users found w2w generations capture identity better while also generating more diverse images that better align with the prompts. Compared to other learning based approaches like Celeb-Basis, w2w is more lightweight as training takes less than a minute with 8GB on a single NVIDIA A100 GPU, while Celeb-Basis takes 21GB and trains a costly MLP in three minutes on a single A100. We present details on the efficiency of w2w in Appendix C.

We would like to note that previous personalization methods such as IP-adapter [2] or Celeb-Basis [1] only demonstrate identity inversion. As we mentioned earlier, we create a unified space for identity-encoding weights with applications of inversion, editing, and sampling identity-encoding models.

“I’m curious if style information…” and “How does w2w perform on multi-concept generation?”

Thank you for suggesting these creative applications! Since the models used to create w2w space are fine-tuned on realistic identities, style information, such as cartoon or painting, would not be encoded in any principal components. However, models from w2w space can be merged with other personalized models for multi-concept generation. In Figure 7 of the rebuttal PDF, we train a LoRA with Dreambooth for Pixar style, and merge it with a w2w model to showcase multi-concept generation ability.

Thanks to the authors for their efforts during the rebuttal. The author's response solved part of my problem. But I still have a lot of concerns that haven't been clarified.

1. The biggest concern is that collecting a large number of weights to build the w2wk space is far more complex than collecting a large number of images. In addition, in terms of the number of parameters, Celeb-Basis only has 1024 learning parameters.

2. And for question 1, what guarantees that information like cartoon drawing is not encoded in the principal components? For example, the weight information represents a Buzz Lightyear. Wouldn't Pixar style and Buzz Lightyear images appear at the same time? Then, why is the new image created based on the weight information not affected by the Pixar style?

3. For question 2, perhaps my question is ambiguous and I am sorry for this, I am actually concerned with the generation of multi-subject, whether the identity information of each person can be maintained when generating multiple characters at the same time (I only saw two case shows in Figure 6 and 7 in the paper).

Thank you for clarifying the doubts. Most of my concerns have been addressed. I will update my rating. However, I still think the use of weights does introduce a large amount of difficulty in preprocessing. Also how to merge multiple LoRa is a big challenge now, and the token-based approach can easily generate multiple roles.