Image as First-Order Norm+Linear Autoregression: Unveiling Mathematical Invariance

Thank you for dedicating your time and effort to provide feedback on our work. Below, we answer the questions that have been raised.

Q1: Some details of the figures are not clear.

For example, what do red and blue arrows in z(x, y) mean? The red and blue arrows in $\mathbfit{z}(x, y)$ represent the two autoregression paths of FINOLA for any given position $(x, y)$ from the center where the global pooled vector $\mathbfit{q}$ is placed. Each arrow signifies a FINOLA step, and the color denotes the order of regression. The red path involves horizontal regression followed by vertical regression, while the green path involves vertical regression followed by horizontal regression. The results of these two paths are averaged to obtain the final feature vector.

How do they affect the reconstruction? It's crucial to note that we repeat this "two-path" autoregression until the entire feature map is generated. Subsequently, the feature map goes through the decoder to reconstruct the images.

Does a single vector q mean the feature of the whole image or the specific pixel? "A single vector q" indeed denotes the feature of the entire image after pooling.

In the figure, the a high frequency in the reconstruction loss. As for the concern about the loss of high-frequency details in the reconstructed images, we acknowledge this limitation, and it is discussed in detail in Appendix B. We attribute this limitation to the choice of the $L_2$ loss function. Our intentional use of a simple loss function is to highlight that FINOLA is not heavily dependent on complex reconstruction loss functions. To improve image quality, an alternative approach could involve exploring the use of a combination of perceptual loss and an adversarial training procedure with a patch-based discriminator, as outlined in the VQGAN framework.

Q2: How to calculate the mean $\mu_z$ and the standard deviation $\sigma_z$? In Eqn. (1), how to initialize the matrix A and B? What are the constraints of the matrix A and B?

The mean $\mu_z$ and the standard deviation $\sigma_z$ are calculated per position $(x, y)$ on the feature map $\mathbfit{z}$. For each position $(x, y)$, the corresponding feature $\mathbfit{z}(x, y)$ is a vector with $C$ channels: $\mathbfit{z}(x, y)=[z_1,\dots,z_C]^T$. $\mu_z$ and $\sigma_z$ are the mean $\mu_z=\frac{1}{C}\sum z_i$ and standard deviation $\sigma_z=\sqrt{\frac{\sum (z_i-\mu_z)^2}{C}}$ of these $C$ values.

Matrices $\mathbfit{A}$ and $\mathbfit{B}$ are randomly initialized, and no specific constraints are placed on them.

Q3: The authors mainly compare VQGAN (Esser et al. (2021)) and stable diffusion (Rombach et al. (2021)) in image reconstruction. It would be better to compare recent image reconstruction methods. In addition, could you compare convolutional U-Net for image reconstruction?

Comparison with convolutional U-Net: The table below provides a comparative analysis between our method and convolutional U-Net for image reconstruction (measured by PSNR). Both approaches share the same Mobile-Former encoder and have identical bottleneck dimensions (C=1024 or 3072). In our method, FINOLA is employed to generate a 16x16 feature map, followed by a convolutional decoder to reconstruct an image with a size of 256x256. On the other hand, U-Net has a deeper decoder that utilizes convolution and upsampling for image reconstruction. Both methods boast a similar model size. The results showcase the superior performance of FINOLA over convolutional U-Net, indicating that a single layer FINOLA is more effective than multi-layer convolution and upsampling.

Comparison with ViT-VQGAN: In the following comparison, Masked FINOLA is evaluated against ViT-VQGAN ("Vector-Quantized Image Modeling with Improved VQGAN, Yu et al., ICLR 2022") using ImageNet linear probing. FINOLA achieves higher accuracy with a much more compact encoder.

For a detailed comparison in image reconstruction, we will address this in the subsequent question (Q4).

Q4: In Table 4 (a), using PSNR to compare VQGAN and Stable Diffusion is not convincing because these methods are generative methods. It would be better to use LPIPS and FID.

Below, we present a comparison of FINOLA with the first stage of multiple generative methods, assessed using both PSNR and FID metrics. While FINOLA outperforms in PSNR, it lags in FID. This discrepancy can be attributed to our deliberate use of the $L_2$ loss function. We intentionally opted for a simple loss function to underscore that FINOLA does not heavily rely on intricate reconstruction loss functions. To enhance FID, we could explore the success of perceptual and GAN losses, as seen in VQGAN, ViT_VQGAN, and Stable Diffusion.

It's essential to clarify that FINOLA does not operate as a generative model. Our primary goal is not image generation; instead, we aim to unveil mathematical properties within the latent space. Image reconstruction, one of our tasks alongside self-supervised learning, serves as a means to validate FINOLA — a first-order norm+linear autoregressive model — as a fundamental mathematical property shared by all images.

FINOLA is also implemented differently from autoregression-based generative models. The latter typically consists of two stages: the first stage involves learning an autoencoder and vector quantization in the latent space, while the second stage focuses on learning autoregression in the quantized latent space. In contrast, FINOLA operates as a single-stage model, seamlessly learning the encoder, autoregression, and decoder end-to-end, with autoregression performed on a continuous (non-quantized) latent space.

Thank you for dedicating your time and effort to provide feedback on our work. Below, we answer the questions that have been raised.

Q1: The author mentioned "this intriguing property reveals a mathematical invariance". What does invariance refer to should be further explained.

The term "mathematical invariance" refers to the consistent relationship shared by all images between feature values and their spatial derivatives on the feature map. Mathematically, for any given image, the feature values $\mathbfit{z}(x, y) \in \mathbb{R}^C$ at any position $(x, y)$ on the feature map determine its spatial derivatives as follows:

$\mathbfit{z}(x + 1, y) - \mathbfit{z}(x, y) = \mathbfit{A}\mathbfit{z}_n(x, y)$,

$\mathbfit{z}(x, y + 1) - \mathbfit{z}(x, y) = \mathbfit{B}\mathbfit{z}_n(x, y)$,

where $\mathbfit{z}_n(x, y) = \frac{\mathbfit{z}(x, y) - \mu_z}{\sigma_z}$.

It's essential to note that matrices $\mathbfit{A}$ and $\mathbfit{B}$ exhibit invariance across different images and pixels on the feature map, emphasizing the consistent mathematical relationship that holds universally.

Q2: The author mentioned "providing insights into the underlying mathematical principles" more than once, but did not provide an in-depth explanation or comparison. It is recommended to provide more details.

Thank you for the valuable suggestion. Allow us to provide a more detailed explanation. The crux of our mathematical insights lies in the understanding that the feature maps of all images adhere to two partial differential equations (PDEs):

$\frac{\partial \mathbfit{z}}{\partial x}=\mathbfit{A}\mathbfit{z}_n, \quad \frac{\partial \mathbfit{z}}{\partial y}=\mathbfit{B}\mathbfit{z}_n$

These equations reveal that the spatial rate of change, or derivatives, in the feature map is entirely determined by the feature values at the current location. It's essential to note that matrices $\mathbfit{A}$ and $\mathbfit{B}$ exhibit invariance across different images and pixels on the feature map, emphasizing the consistent mathematical relationship that holds universally.

Another mathematical insight is related to the role of masking, particularly in the context of masked FINOLA for self-supervised learning. This technique plays a crucial role in facilitating the extraction of meaningful semantic representations from the feature maps. However, it comes with an inherent trade-off, impacting the preservation of intricate image details. Notably, compared to vanilla FINOLA, masked FINOLA exhibits a substantial increase in Gaussian curvature on the surfaces of critical features. This observed increase suggests a heightened curvature in the latent space, underlining its effectiveness in capturing and emphasizing semantic information.

Q3: When validating the norm+linear approach, the authors repeated q by W × H times. In the original setting, the authors learned two matrices A and B, with more parameters. Thus, the comparison seems a bit unfair.

The objective of this comparison is to demonstrate the superiority of using norm+linear over its absence. Repetition, requiring zero additional effort, naturally stands out as a straightforward option. In this context, our intention is to specifically exclude the scenario where the zero-effort repetition achieves performance comparable to our more complex norm+linear approach. Such a scenario would contradict the essential need for norm+linear. The poor performance of repetition serves as evidence that norm+linear cannot be easily replaced by a simpler and more straightforward solution, highlighting the necessity of our chosen approach, even in the presence of additional parameters when learning matrices $\mathbfit{A}$ and $\mathbfit{B}$.

Q4: In comparable performance with Stable Diffusion, the input of the paper method is an image of the same size as the output, while the input of stable diffusion is a lower resolution image. The tasks of the two are different, so it may not be appropriate to compare them together.

Let's clarify the comparison with Stable Diffusion. Our comparison is specifically with the first stage of Stable Diffusion, known as perceptual image compression, which involves training an autoencoder to reconstruct images. We explicitly refrain from comparing with the second stage of Stable Diffusion, which includes generation of super-resolution from lower-resolution images. The comparison is thus limited to the common task of image reconstruction and does not involve tasks related to super-resolution from lower-resolution inputs.

Thank you for dedicating your time and effort to provide feedback on our work. Below, we answer the questions that have been raised.

Q1: There are previous regression-based generative models (e.g., refer to related works) in feature space, and what are the major difference and advantage of this approach compared with these models? Is it possible to compare with them for the generation quality and computational speed?

First and foremost, it's essential to clarify that FINOLA does not function as a generative model. Our primary objective is not image generation but rather the revelation of mathematical properties within the latent space. Image reconstruction, one of our tasks (alongside self-supervised learning), serves as a means to validate FINOLA — a first-order norm+linear autoregressive model — as a fundamental mathematical property shared by all images.

It's crucial to note that FINOLA is both functionally and technically distinct from autoregression-based generative models. The latter typically consists of two stages: the first stage involves learning an autoencoder and vector quantization in the latent space, while the second stage focuses on learning autoregression in the quantized latent space. In contrast, FINOLA operates as a single-stage model, seamlessly learning the encoder, autoregression, and decoder end-to-end, with autoregression performed on a continuous (non-quantized) latent space.

Given that FINOLA doesn't fall into the category of generative models, it isn't suitable for direct comparison in terms of generation quality and computational speed with models designed for such purposes.

Q2: Are the matrix A and B shared for all different images and pixels in the feature space? If it is, why learned constant A, and B can deduce good feature regression?

Yes, the matrices $\mathbfit{A}$ and $\mathbfit{B}$ are shared for all images and pixels in the feature space.

The success of using constant $\mathbfit{A}$ and $\mathbfit{B}$ for achieving good feature regression can be attributed to three primary factors. Firstly, images inherently exhibit significant spatial redundancy, leading to strong correlations between consecutive positions within the feature map. Secondly, our hypothesis regarding the existence of a feature map where spatial derivatives correlate with feature values appears to be supported. Lastly, the effectiveness of deep learning allows us to validate this hypothesis by seamlessly learning the encoder, FINOLA, and decoder components in an end-to-end manner.

Q3: The paper states that this work does not aim to achieve SoTA results, however, comparisons with SoTA regression models or other variants, e.g., using nonlinear regression instead of linear transform, should be able to better give insights to audience.

Excellent point! We have conducted a comparison between our norm+linear approach and non-linear regression, which involves replacing the linear transform with two MLP layers incorporating GELU activation in between. This evaluation was performed for both 1024 and 3072 channels. Surprisingly, our norm+linear, despite having fewer parameters, closely trails the performance of the two-layer MLP. This result underscores the effectiveness of norm+linear in modeling the transition between pixels in the feature map.

We further benchmark our method against an alternative non-linear solution: employing a multi-layer convolutional approach to upsample the vector $\mathbfit{q}$ to a 16x16 feature map. This method undergoes four upsample stages, each incorporating three 3x3 convolution layers. The results below demonstrate the superior performance of a single layer FINOLA over the multi-layer convolutional baseline.

Thank you for dedicating your time and effort to provide feedback on our work. Below, we answer the questions that have been raised.

Weakness 1: Justifications for the novel design choice (but seems heuristic) compared to the regular AR model.

Thanks for the suggestion. Below we provide justification for each item.

Predefined assignments and predicting three points in Block-wise Masked FINOLA

This design is meticulously crafted to achieve several key objectives: (a) Full coverage: ensuring that all masked positions receive attention, (b) Balance: distributing the usage of unmasked positions equally, and (c) Efficiency: utilizing each unmasked position to predict three masked positions, especially under the condition where 75% of positions are masked. Additionally, this design can be implemented in a block-wise manner to further enhance efficiency, a crucial consideration for the extended training durations involved in self-supervised learning.

Why first order?

The selection of first-order autoregression is driven by its inherent simplicity and local property, where the derivatives along the $x$ and $y$ axes are exclusively determined by the current position. This strategic choice not only simplifies the computational process but also aims to unveil valuable mathematical insights that could be shared across different images, provided the approach proves effective."

Weakness 2: Lack of interpretation of the derived PDE.

The derived partial differential equations (PDEs) in Eq. 4 provide a theoretical extension of FINOLA from a discrete grid to continuous coordinates, establishing a conceptual bridge between the discrete representations of neural networks and the continuous nature of human vision. Although our digital images and feature maps are inherently discrete, the human vision system perceives continuous signals.

This theoretical extension allows us to hypothesize about the mathematical properties of continuous feature representations in the human vision system. We posit that the proposed PDEs in Eq. 4, derived from their discrete counterparts in Eq. 1, capture the patterns inherent in continuous features within human vision.

This is related to Q2 below.

Q1: How do the coefficient matrices $\mathbfit{A}$ and $\mathbfit{B}$ directly know the spatial relationship? They do not take any of the spatial information.

The coefficient matrices $\mathbfit{A}$ and $\mathbfit{B}$ encode the relationship between a feature vector $\mathbfit{z}(x, y) \in \mathbb{R}^C$ and and its spatial derivatives, namely $\mathbfit{z}(x+1, y)-\mathbfit{z}(x, y)$ and $\mathbfit{z}(x, y+1)-\mathbfit{z}(x, y)$. This relationship is mathematically expressed as:

$\mathbfit{z}(x + 1, y) - \mathbfit{z}(x, y) = \mathbfit{A}\mathbfit{z}_n(x, y)$,

$\mathbfit{z}(x, y + 1) - \mathbfit{z}(x, y) = \mathbfit{B}\mathbfit{z}_n(x, y)$,

where $\mathbfit{z}_n(x, y) = \frac{\mathbfit{z}(x, y) - \mu_z}{\sigma_z}$.

It's essential to note that matrices $\mathbfit{A}$ and $\mathbfit{B}$ exhibit invariance across different images and positions $(x,y)$ within the feature map once they are learned from the data.

1.1 Coefficient matrices $\mathbfit{A}$ and $\mathbfit{B}$ capture channel-wise relationship.

Indeed, the difference of the channel vector between consecutive positions, such as $\mathbfit{z}(x + 1, y) - \mathbfit{z}(x, y)$, represents a linear combination of normalized channel values at the current position $\mathbfit{z}_n(x, y) \in \mathbb{R}^C$.

1.2: The pattern and correlation encoded in the channel are related to the positional information.

The correlation does NOT pertain to the positional information; instead, it is associated with the derivatives (or rate of change) along the $x$ and $y$ axes, often referred to as "spatial derivatives".

1.3: the coefficient matrices $\mathbfit{A}$ and $\mathbfit{B}$ (indirectly) capture the relationship between each position. Maybe it is trivial for some readers but it is not for me.

As discussed earlier, the matrices $\mathbfit{A}$ and $\mathbfit{B}$ explicitly capture the relationship between the feature at a given position $\mathbfit{z}(x, y)$ and its derivatives along the $x$ and $y$ axes. Consequently, this encoding implicitly extends to capturing the relationship between consecutive positions, such as from $\mathbfit{z}(x, y)$ to $\mathbfit{z}(x + 1, y)$.

Dear Reviewers,

We are writing to remind you that the author-reviewer discussion period is soon coming to an end. If you have any concerns or questions regarding our rebuttal, please do not hesitate to inform us. Your input will be valuable in helping us refine our manuscript.

Understanding the demands on your schedule, we sincerely appreciate the time you've dedicated to this rebuttal discussion. Your thoughtful feedback is immensely helpful to us.

Thank you for your attention.

Best regards,

-authors