Idempotent Generative Network

Motivation: “Several different consequences are described in the introduction (IGN provides outputs in a single step, yet allows additional refinements, etc.) Each of these properties is attractive, but already possessed by an existing mature method (as is pointed out in the paper). “
IGN has some unique properties that do not exist in other models. In addition, IGN has some properties that exist in other models but do not co-exist together. For instance, let us consider the single step + refinement property the reviewer mentions. There are Diffusion based methods with fewer steps, such as consistency models. However IGN always does one step full generation after which it is possible to apply it again for refinement. Another example is a consistent latent space. Surely, GANs have that, but IGN allows smooth interpolation across all iterations ($f(z), f(f(z)), f(f(f(z)))$...). (See video in supplementary). Also, Unlike GANs and Diffusion models, editing of real images in IGN does not require any type of inversion. Simply inputting a real image in a feed-forward fashion into IGN allows manipulation on it (see new figures 9,10 in revised manuscript). Lastly there is the projection property addressed below in response to your specific question.

“IGN can be used to project corrupted images … already a range of methods for doing this (e.g., via image denoising with a diffusion model).”
While many methods handle certain degradations or modifications of images, they will typically be trained for it specifically (e.g., Diffusion-Denoising, Cold Diffusion, image enhancement methods, image-to-image methods), or they will require some manipulation at test time (adding noise, guidance in diffusion and some tricks in GANs). IGN is not trained for specifically on modified images. IGN only sees pure noise and real images during training. No different type of inference is needed. For example, no noise is added to bring the input into distribution, as would be in, for instance, SDEdit. Note that it is not only for corrupted images but also image-to-image translation such as sketches. It can be thought of as a generic ``make-it-real" button.

“ $\mathbf{f^k(z) \quad k\rightarrow\infty}$. Does this converge or diverge?“:
We added fig.16 in the appendix of the revised manuscript which demonstrates applications of MNIST trained IGN up to $300000$ iterations. For the first few tens of iterations, the system seems to be stable but around 100 it diverges from the result of the first application. We hypothesize that the size of the model which influences the ability to get low reconstruction loss determines the stability.

“This reviewer did not understand the modified GAN network architecture described in Section 2.3.”
In the revised manuscript, we added Fig.15 to illustrate the modification of DCGAN architecture to be used for IGN on CelebA. The exact architecture is in Table.1 in the appendix. DCGAN has a generator $G$ and a discriminator $D$. In training $D(G(z))$ is applied. $D$ produces a $1\times1\times1$ result per image. we modify $D$ to end with a $512\times1\times1$ tensor instead, denoted as $\tilde{D}$. Our model in terms of DCGAN is $G(\tilde{D}(z))$.

Tightness loss clarity:
Indeed it is subtle, and challenging to convey. We added the notion of $\mathcal{S}$ to the text around fig.2 and its caption as you suggested. We updated the text about $\mathcal{L}\_{tight}$ and also added Fig.17 that offers further clarification on the two paths to enforce idempotence. We welcome any suggestions to enhance the explanation in the paper. We have provided some informal perspectives to help understand $\mathcal{L}\_{tight}$ in responses to other reviewers. please check them (not pasting here due to char limit, sorry.).

“Would it be possible to build an IGN model using the framework of a known generative approach?”, “What are the current drawbacks to scaling this method up? “
Scaling up is a current effort. Indeed, the ways explored are based also on existing methods. Among others, working in latent space as in Stable Diffusion and using transformer architectures are being tried. One challenge we have seen is the need of a bottleneck which makes the popular Unet challenging to work with due to its skip connections.

Quantitative evaluation and comparison to other methods:
We report FID=39 for IGN on CelebA. Naturally, this is not competitive with modern models, but is comparable to early models such as DCGAN (FID=34), while also capable of a wide range of conditional tasks. We updated the draft paper to include this info.

Idempotence spelled wrong: Fixed, thanks!

Code better in appendix:
The provided PyTorch code is a substitute for an algorithm pseudocode. We felt that it is the cleanest way of presenting the algorithm. That said, we can consider moving it to the appendix and replace it with the regular pseudocode notation. We will also release full code.

Dear reviewer, towards the end of the discussion phase, we trust that our response has successfully addressed your inquiries. We look forward to receiving your feedback regarding whether our reply sufficiently resolves any concerns you may have, or if further clarification is needed.

Thank you for the thoughtful consideration of the paper and constructive feedback.

The value $M$:
$M$ is the supremum of the distance metric for any pair $(y\_1, y\_2)$. There is no bounded space assumption -- M can also be infinity (which is why we did not define it as Maximum. We added a clarification in the paper). We examine both the scenario of unbounded $M$ (which is the case of MSE as used for MNIST and shown in the provided code), and the scenario of bounded $M$ which exists when using the clamp technique as introduced in eq.14. Unbounded $M$ is consistent with all the steps of the proof, under the practical assumption that $M \cdot 0 < M$.

The role of $M$ in eq.21 (formerly eq.19):
Thank you for pointing out the lack of clarity. We have revised the text before the equation (that is eq.21 now). Eq.21 describes what happens when $\mathcal{L}\_{rt}$ is minimized, according to how it is expressed in eq.20 (eq.18 second row in pre-revised version):

\theta^* = argmin\_\theta  \mathbb{E}\_{z}\bigl[ M(y) \cdot  1\_{\{
\mathcal{P}\_{x}(y) < \lambda\_t  
\mathcal{P}\_{\theta}(y) 
\}} \bigr] 

Dear reviewer, towards the end of the discussion phase, we trust that our response has successfully addressed your inquiries. We look forward to receiving your feedback regarding whether our reply sufficiently resolves any concerns you may have, or if further clarification is needed.

I read the response of the author, other reviewers, and the public commenter. I agree with the public commenter that this method is a special case of EBGAN where the generator and discriminator share the same structure. As noted by the author response, there are interesting properties of an idempotent network that EBGAN does not have and that the connection between idempotence and EBGAN is an interesting novel view that is perhaps more streamlined than the original EBGAN framework. I suggest that the authors simply use the EBGAN proof of convergence to theoretically justify their method. Overall, I believe this paper needs a major revision to incorporate EBGAN as a prior work and reframe its contributions. I will keep my score.

Thank you for the thoughtful consideration of the paper and constructive feedback.

Motivation: “When one designs a new class of generative model that is different from any previous approaches, it is necessary to clarify that it has the potential to address some limitations of previous approach.”
IGN has several potential advantages over the current methods:

1. Unlike DIffusion models, IGN generates in a single step, and then, unlike GANs, allows for sequential refinement. There are Diffusion based methods with fewer steps, but for IGN the number of reprojections can be determined adaptively (apply again if desired). A single step will always yield good results.
2. Unlike Diffusion models, IGN maintains a consistent latent space, allowing smooth interpolation across all iterations ($f(z), f(f(z)), f(f(f(z)))$...). (Please see videos in supplementary).
3. IGN can be applied to types of data that it has never ‘seen’ and project them onto the data manifold. While Diffusion models can be guided to get condition signals, IGN does that when regularly applied to degraded/modified input.
4. Unlike GANs and Diffusion models, editing of real images does not require any type of inversion. Simply inputting a real image in a feed-forward fashion into IGN allows manipulation on it (see new figures 9,10 in revised manuscript).

”I don't quite get the point of the tightness loss. Why maximize it is good for regularization? More explanation is needed.”
We appreciate the feedback and acknowledge the challenge in conveying the concept of $\mathcal{L}\_{tight}$. In the new draft, we added the notion of $\mathcal{S}$ to the caption of fig.2. We updated the text about $\mathcal{L}\_{tight}$ and also added Fig.17 to the appendix of the revised manuscript. Fig.17 offers further clarification on the two paths to enforce idempotence. We offer additional informal perspectives here for better clarity and welcome any suggestions to enhance the explanation in the paper.

Consider the metaphor of High Jump. The objective is to clear the bar. This can be achieved in two ways: jumping higher, or lowering the bar. If the athlete, $f$ trained directly for this objective, it would become an expert on lowering the bar (tends to become identity). Instead, the athlete tries to get bigger gap from the bar when jumping, but trying to get smaller gap from the bar (opposite from the objective), when setting it according to its recent jumps. We emphasize that this is the same athlete that both jumps and sets the bar.

Examining the objective $\delta\_\theta = D(f\_\theta(f\_\theta(z)),f\_\theta(z))$, we can split it into two phases:
(1) $y=f\_\theta(z)$ (jumping)
(2) $\delta\_\theta = D(f\_\theta(y),y)$ (evaluating the gap from the bar)
Both are done using the same set of parameters $\theta$. See Fig.17 in the appendix. The instance of $f$ that is in phase (1) sends gradients to $\theta$ that improve the generation (higher jump, lower drift $\delta\_\theta(y) = D(f\_\theta(y), y)$). The Gradients to $\theta$ through the instance of $f$ that is in phase (2) judge generated images and train to exclude them from the manifold (higher drift), just like setting the bar higher according to the recent jumps. This aspect can also be thought of as a GAN where the generator and discriminator are the same network.

Dear reviewer, towards the end of the discussion phase, we trust that our response has successfully addressed your inquiries. We look forward to receiving your feedback regarding whether our reply sufficiently resolves any concerns you may have, or if further clarification is needed.

Thank you for the thoughtful consideration of the paper and constructive feedback.

"I found the discussion around Figure 2 to be less clear than I would have liked."
We appreciate the feedback and acknowledge the challenge in conveying the concept of $\mathcal{L}\_{tight}$. We added the notion of $\mathcal{S}$ to the text around fig.2 and its caption. We updated the text about $\mathcal{L}\_{tight}$ and also added Fig.17 to the appendix of the revised manuscript. Fig.17 offers further clarification on the two paths to enforce idempotence. We offer additional informal perspectives here for better clarity and welcome any suggestions to enhance the explanation in the paper.

Consider the metaphor of High Jump. The objective is to clear the bar. This can be achieved in two ways: jumping higher, or lowering the bar. If the athlete, $f$ trained directly for this objective, it would become an expert on lowering the bar (tends to become identity). Instead, the athlete tries to get bigger gap from the bar when jumping, but trying to get smaller gap from the bar (opposite from the objective), when setting it according to its recent jumps. We emphasize that this is the same athlete that both jumps and sets the bar.

Examining the objective $\delta\_\theta = D(f\_\theta(f\_\theta(z)),f\_\theta(z))$, we can split it into two phases:
(1) $y=f\_\theta(z)$ (jumping)
(2) $\delta\_\theta = D(f\_\theta(y),y)$ (evaluating the gap from the bar)
Both are done using the same set of parameters $\theta$. See Fig.17 in the appendix. The instance of $f$ that is in phase (1) sends gradients to $\theta$ that improve the generation (higher jump, lower drift $\delta\_\theta(y) = D(f\_\theta(y), y)$). The Gradients to $\theta$ through the instance of $f$ that is in phase (2) judge generated images and train to exclude them from the manifold (higher drift), just like setting the bar higher according to the recent jumps. This aspect can also be thought of as a GAN where the generator and discriminator are the same network.

"Why does M:= sup(D(y1, y2)) show up in eq 19?"
Thank you for pointing out the lack of clarity. We have revised the text before the equation (that is eq.21 now). Eq.21 describes what happens when $\mathcal{L}\_{rt}$ is minimized, according to how it is expressed in eq.20 (eq.18 second row in pre-revised version):

$\delta\_\theta$ is multiplied by a given expression in the brackets that is not being optimized in this phase. To obtain a minimum, $\delta\_\theta\geq0$ needs to be as big as possible when the expression in the brackets is negative and as small as possible when positive. Thus, for any $M>0$ eq.21 holds when $\mathcal{L}\_{rt}$ obtains its minimum: $\delta\_{\theta^*}(y) = M \cdot 1\_{\{\mathcal{P}\_{x}(y) < \lambda\_t \mathcal{P}\_{\theta^*}(y) \}}$

Dear reviewer, towards the end of the discussion phase, we trust that our response has successfully addressed your inquiries. We look forward to receiving your feedback regarding whether our reply sufficiently resolves any concerns you may have, or if further clarification is needed.

Thank you for your comment and insights. EBGAN is a very relevant work. We will add it to the paper and discuss it.

Having the same model as both generator and discriminator is really the main point of IGN. The capabilities of IGN, mentioned in the paper and listed also in the responses here, all rise from exactly that and are not held by EBGAN. We briefly remind them and compare:

1. Sequential refinement of the model. EBGAN or any other GAN only do one step.
2. Projections of modified/corrupted images into the manifold. EBGAN does not demonstrate, and cannot trivially do that since it is not applied to images.
3. Editing of real images with no inversion, by simply applying the model to an input image. IGN gets images as inputs but EBGAN like other GANs cannot trivially do that.
4. Consistent latent space is the one property that we assume EBGAN also has, just like any other GAN. IGN has this consistency across sequential applications of the model.

We should also point out that the strong relation to both GANs and EBMs is acknowledged in the related work section. (Though not mentioning specifically EBGAN). In particular, we describe the full analogy of IGN to GANs:
"One could view $\delta$ as a discriminator trained using $L\_{rec}$ for real examples and $L\_{tight}$ for generated ones, while $f$ serves as the generator trained by $L\_{idem}$. Unique to IGN is a form of adversarial training we term “self-adversarial”, Put simply, $f$ is both the generator and the discriminator. This streamlined architecture affects the optimization process. Rather than alternating between two networks, a single model accumulates gradients from both discriminative and generative perspectives in each step."
We also mention the relation to EBMs in which we acknowledge the similarity, and make the distinction that having an idempotent $f$ "reduces the need for separate optimization procedures to find the energy minimum".

There are some other important implications to the difference, like the fact that there is no alternating optimization for IGN, but a single accumulation of the gradients from all losses. Another difference is the theoretical analysis. IGN has only one player, therefore cannot be formalized as a two-layer zero-sum game as GANs. Our theoretical derivation, while has similarities to that of GANs, is different.

Over all, we appreciate this comment, and indeed EBGAN should have been discussed in the paper, and such a revision will be made soon.

I would like to thank the Michael for their insights on the connections between EBGAN and IGN. However, I would also like to alert them, as they are a more senior researcher, that their tone has begun to become somewhat hostile, and I am beginning to become concerned that this might bring undo influence over the review process of this paper (particularly when received by more junior researchers).

I intend to go over the connections between EBGAN and IGN (it's been a long time since I've read that paper) in detail, and I acknowledge the connections you have laid out. However, I may still consider the work to be publishable, as the insights are still interesting and novel.

More specifically, I believe there are many ways to arrive at the same solution. And that's OK: these works should still be permitted to exist, despite they come around to a similar (or nearly the same) answer. If we are to completely discard such works, then we may unfortunately be stifling good and well-intentioned research, which will reduce the quality of this conference. Please keep this in mind: the purpose here should be to improve science, not gate-keep it.