Exploring Invariance in Images through One-way Wave Equations

We sincerely thank the reviewer for the thoughtful feedback and valuable suggestions, which have significantly improved the quality of our paper.

$\color{blue}{**[Weakness 1]:**}$

First, I don’t think the authors have discovered a meaningful image invariance. As I see it, the authors have devised an autoencoder with an unusual architecture. They claim that, since the autoencoder’s learned parameters are constant across all images, this represents some kind of invariance inherent in images. At best, I think this stretches the definition of invariance. At worst I think it is philosophically confused. The relationship to wave equations feels like an arbitrary constraint on the autoencoder's latent representations. If this characterization is unfair, feel free to debate me on this.

Thank you for the insightful comment! We appreciate you pushing us to clarify our claims about image invariance. We agree that any autoencoder will exhibit some form of invariance through its learned parameters. However, FINOLA introduces a distinct and more profound type of invariance.

Here's why FINOLA is different:

FINOLA's Unique Approach to Image Representation

To understand this distinction, let's focus on how FINOLA and the decoder represent an image. They essentially construct an image function:

$I=f_{\mathbf{A}, \mathbf{B}, \theta}(\mathbf{q})$,

where $\mathbf{q}$ is a compressed vector, $\mathbf{A}$ and $\mathbf{B}$, and $\theta$ represents the decoder's parameters.

Crucially, the decoder plays a minimal role in this representation. As shown in Table 1, FINOLA can generate a full-resolution feature map using only $\mathbf{A}$ and $\mathbf{B}$. This means the decoder (with its 1.2M parameters) is dwarfed by the complexity of $\mathbf{A}$ and $\mathbf{B}$ (e.g., $\mathbf{A}$ alone has 9M parameters). Therefore, the image function is primarily governed by these FINOLA matrices.

Unveiling Invariant Local Structure

FINOLA's core operation (Equation 1) can be expressed as:

• $\Delta_x\mathbf{z}(x,y)=\mathbf{A}\widehat{\mathbf{z}}(x,y)$

• $\Delta_y\mathbf{z}(x,y)=\mathbf{B}\widehat{\mathbf{z}}(x,y)$

These equations essentially describe how the feature map changes as you move horizontally ($\Delta_x\mathbf{z}$) or vertically ($\Delta_y\mathbf{z}$) across the image. They reveal a fundamental property: both horizontal and vertical changes in the feature map are linearly correlated to the current position (after normalization). This means that $\mathbf{A}$ and $\mathbf{B}$ capture an invariant local structure that holds consistently across the entire feature map (any x, y) and across all images.

This type of explicit, geometrically grounded invariance sets FINOLA apart. Other autoencoders, while exhibiting invariance through their parameters, do not offer this level of interpretability and explicit modeling of local image structure.

We sincerely thank the reviewer for the thoughtful feedback and valuable suggestions, which have significantly improved the quality of our paper.

$\color{blue}{**[Question 1]:**}$

What do you think your results teach us about natural images?

The results suggest that natural images possess a remarkable underlying structure that can be effectively captured by a surprisingly simple model. Here's a breakdown of the key insights:

• Local dependencies: The FINOLA equations demonstrate that the latent representation $\mathbf{z}(x,y)$ at any point $(x,y)$ can be predicted solely based on its immediate neighbors.

  $\Delta_x\mathbf{z}(x,y)=\mathbf{A}\widehat{\mathbf{z}}(x,y)$

  $\Delta_y\mathbf{z}(x,y)=\mathbf{B}\widehat{\mathbf{z}}(x,y)$

  This localized nature implies that, despite the complex and globally correlated structures in natural images, there exists a transformation (through our encoder) that maps them into a latent space where essential information is encoded in consistent local dependencies.

• Simple, first-order relationships: These local dependencies are characterized by remarkably simple, first-order linear relationships (matrices $\mathbf{A}$ and $\mathbf{B}$) after normalization. This simplicity is surprising, given the complexity of natural images.

• Efficient autoregressive process: The combination of local dependencies and simple linear relationships enables an efficient autoregressive process for reconstructing images. This means that we can predict the value of $\mathbf{z}(x,y)$ at any point in the latent space by considering only its immediate neighbors, making the reconstruction process highly efficient.

$\color{blue}{**[Question 2]:**}$

How do you think the matrices A and B encode spatial information? Have you examined the learned matrices?

Thank you for this question, which is also raised by Reviewer PCAc. Understanding the meaning of matrices $\mathbf{A}$ and $\mathbf{B}$ is key to grasping FINOLA's core mechanism.

What do $\mathbf{A}$ and $\mathbf{B}$ represent?

Matrices $\mathbf{A}$ and $\mathbf{B}$ model the local structure within the latent vector space ($\mathbf{z}(x,y)$). They essentially describe how the feature map changes as you move horizontally or vertically across the image.

We've analyzed these learned matrices and confirmed they are full rank and diagonalizable. This allows us to express FINOLA as:

• $\Delta_x\mathbf{z}(x,y)=\mathbf{A}\widehat{\mathbf{z}}(x,y)=\mathbf{P}_A\mathbf{\Lambda}_A\mathbf{P}^{-1}_A\widehat{\mathbf{z}}(x,y)$

• $\Delta_y\mathbf{z}(x,y)=\mathbf{B}\widehat{\mathbf{z}}(x,y)=\mathbf{P}_B\mathbf{\Lambda}_B\mathbf{P}^{-1}_B\widehat{\mathbf{z}}(x,y)$

where $\mathbf{P}_A$ and $\mathbf{P}_A$ are the eigenvector matrices, and $\mathbf{\Lambda}_A$ and $\mathbf{\Lambda}_B$ are the diagonal eigenvalue matrices of $\mathbf{A}$ and $\mathbf{B}$, respectively.

Interpreting $\mathbf{A}$ and $\mathbf{B}$

Imagine each point in the latent space as having a set of "springs" attached to it, pulling it horizontally and vertically. Matrices $\mathbf{A}$ and $\mathbf{B}$ define the stiffness of these springs, while the eigenvectors ($\mathbf{P}_A$ and $\mathbf{P}_B$) represent the directions of these springs.

• Diagonalization: Diagonalizing $\mathbf{A}$ and $\mathbf{B}$ helps us understand these "springs" in a simpler way. It's like projecting the latent space so that each dimension (eigenvector) corresponds to an independent spring with a specific stiffness (eigenvalue).

• Horizontal and vertical changes: After this projection ($\mathbf{P}^{-1}_A$ and $\mathbf{P}^{-1}_B$), the horizontal change ($\Delta_x\mathbf{z}$) is simply a scaling of the current position along each independent dimension by the corresponding eigenvalue in $\mathbf{\Lambda}_A$. Similarly, the vertical change ($\Delta_y\mathbf{z}$) scales the position by the eigenvalues in $\mathbf{\Lambda}_B$.

In essence, $\mathbf{A}$ and $\mathbf{B}$ encode the directional "stretching" or "compressing" factors that govern how the feature map changes locally. The eigenvalues represent the strength of these factors along specific directions in the latent space.

We sincerely thank the reviewer for the thoughtful feedback and valuable suggestions, which have significantly improved the quality of our paper.

$\color{blue}{**[Weakness 1]:**}$

The paper lacks a logical flow, making it challenging to read. While each individual section is well-written, the overall structure makes it difficult for readers to follow a coherent narrative. Here are some specific points:

$\color{blue}{**[1.1] Focus of the Paper:**}$

The title and much of the abstract emphasize "Exploring Invariance," with a focus on demonstrating invariance in images using wave equations. However, the paper appears to focus on FINOLA as the main contribution, focusing on its performance and characteristics.

Thank you for raising this point. We acknowledge that the title and abstract emphasize exploring invariance, while the paper focuses significantly on FINOLA. This is because FINOLA is the key method we developed to explore and demonstrate the empirical invariance in images.

More specifically:

1. FINOLA reveals the invariance: By applying FINOLA to various images, we discovered consistent patterns in the learned matrices $\mathbf{A}$ and $\mathbf{B}$, which represent the invariance property. This empirical observation led us to further investigate the underlying principles of this invariance.

2. Generalization to wave equations: Through theoretical analysis, we generalized the observed invariance from FINOLA to a broader mathematical framework, represented by the one-way wave equation. This connection provides deeper understanding the observed invariance and its implications for image analysis.

Therefore, FINOLA serves as a bridge connecting the empirical observation of invariance in images to the theoretical framework of the wave equation. This allows us to explore invariance in a more principled and comprehensive manner.

We will revise the paper to better articulate this connection and ensure that the focus on FINOLA is properly framed within the broader context of exploring invariance.

$\color{blue}{**[1.2] Purpose and Structure of Sections::**}$

If the primary goal is to demonstrate invariance through FINOLA, then sections on Parallel Implementation and Importance of Norm+Linear relate more to the efficient training of FINOLA and the utility of norm+linear components. These sections feel tangential to the paper’s message and might be better suited to the experiments section or supplementary material.

Thank you for the feedback. We agree that the sections on Parallel Implementation and Importance of Norm+Linear might seem tangential at first glance. However, they play a crucial role in supporting our primary goal of demonstrating invariance through FINOLA. Here's why:

1. Reproducibility and validation: The ease of implementation and efficiency of parallel implementation facilitate reproducibility of our results. This is crucial for others to validate our findings about the invariance property and build upon our work.

2. Connection to the wave equation: The importance of Norm+Linear section highlights how the specific design choices in FINOLA, particularly the norm+linear components, are essential for deriving Equation 2: $\Delta_x\mathbf{z}=\mathbf{AB}^{-1}\Delta_y\mathbf{z}$. This equation is a key step in connecting the observed invariance to the one-way wave equation, which provides a foundation for our findings.

Therefore, these sections are not merely about efficient training or utility of components; they are crucial for establishing the credibility and theoretical grounding of our claims about invariance. We will revise the paper to better articulate these connections and ensure that the relevance of these sections to the main message is clear.

Thank you for raising this important question. You're right to point out that relying solely on PSNR might not be sufficient to justify the FINOLA series. While improved PSNR is a key indicator, we also considered the impact on parameter efficiency.

FINOLA series (or multi-path FINOLA) not only improves PSNR with the same number of parameters but also achieves comparable or better PSNR with a significant reduction in the size of matrices $\mathbf{A}$ and $\mathbf{B}$. This reduction in parameters provides an important validation.

For example, in Table 1, multi-path with 4 paths and 1024 channels outperforms single-path with 3072 channels across all resolutions. Importantly, multi-path with 1024 channels only has 1024x1024=1M parameters in matrices $\mathbf{A}$ or $\mathbf{B}$, which is 9 times less than the single-path with 3072 channels (requiring 3072x3072=9M parameters).

Furthermore, Figure 5 shows that similar PSNR (between 25 and 26) can be achieved by using 16 paths and 256 channels, costing only 256x256=64K parameters in matrices $\mathbf{A}$ or $\mathbf{B}$ (144 times less than single-path with 3072).

Therefore, relaxing the FINOLA constraint through FINOLA series offers a favorable trade-off: comparable or better image quality with significantly reduced the model complexity (achieved by reducing the size of matrices $\mathbf{A}$ and $\mathbf{B}$). This significantly reduces the number of one-way wave equations.

In summary, while improved PSNR is a key result, the justification for FINOLA series also lies in its ability to achieve comparable performance with significantly reduced number of parameters. This highlights the potential of FINOLA series for efficient image representation.

We sincerely thank the reviewer for the thoughtful feedback and valuable suggestions, which have significantly improved the quality of our paper.

$\color{blue}{**[Weakness 1 + Question 1]:**}$

The important problem of this work is that no intuition is given, and all proposals and findings have to be taken as-they-are. If the meaning of the encoding vector is not explained (if it can be explained at all), then, the property found is of no use because one does not know how to understand the (otherwise interesting) constancy of the coefficients of the differential equation. For instance, when one derives classical autoregressive models of the images in the spatial domain, one can interpret the coefficients of these models as autocorrelation functions that describe the (prediction) relations between the luminances at different points of the images. The above sentence can be illustrated both visually and in predictive coders and in conditional probability markov models. However, what is the meaning of things here?.

Can you elaborate on the meaning/interpretation of your result? [I acknowledge this may be difficult to do within the 9 page restriction, but you can do it in the appendices. I would understand the "that is matter for further work" response, but you would understand that the reader may be not satisfied with that response. Understanding is what the reader wants].

Thank you for this insightful feedback! We appreciate you pushing us to clarify the intuition and meaning behind our work. We'll address your concerns by first comparing FINOLA to classical autoregressive models and then delving into the specific interpretations and implications of our findings.

A. FINOLA vs. Classical Augoregressive Models

You're right to draw parallels with classical autoregressive models. Here's a comparison to highlight the key difference:

• Classical autoregressive models: These models use autocorrelation functions to describe the relationship between luminances at different points in an image, essentially predicting one position based on others.

• FINOLA: FINOLA, on the other hand, focuses on the relationship between changes in the feature map (horizontal and vertical) and the corresponding local features. The coefficients of matrices $\mathbf{A}$ and $\mathbf{B}$ in Equation 1 capture this relationship.

Essentially, FINOLA transforms an image into a compressed representation ($\mathbf{q}$) where each point has an invariant local relationship with its neighbors, defined by how the feature map changes at that point.

B. Intuition and Meaning of Our Work

Our work introduces a new autoencoder architecture designed to explore an interesting invariance in images through a simplified autoregressive process.

B.1 A New Autoencoder

The core idea is to maximize FINOLA's role in image reconstruction, forcing it to capture the underlying invariance. We achieve this by introducing two constraints:

1. Single Vector Output: The encoder compresses the entire image into a single vector $\mathbf{q}$, eliminating explicit positional information. This forces FINOLA to recover spatial information from the compressed representation.

2. Minimal decoder: The decoder is intentionally kept minimal, with FINOLA responsible for reconstructing a high-resolution feature map. This further emphasizes FINOLA's role in capturing image structure.

By imposing these constraints, FINOLA has to recover a high-resolution image (e.g., 256x256) from a single vector with minimal assistance from the decoder. This can be represented mathematically as:

$I=f_{\mathbf{A}, \mathbf{B}, \theta}(\mathbf{q})$,

where the decoder's parameters ($\theta$) are dwarfed by the complexity of the FINOLA matrices $\mathbf{A}$ and $\mathbf{B}$. For instance, when using FINOLA to generate full-resolution feature map, the decoder only has three 3x3 convolutions with 1.2M parameters, significantly less than parameters in matrix $\mathbf{A}$ (9M parameters).

B.2 Validating the Invariance

Our experiments (see Table 1) demonstrate that this architecture can indeed reconstruct images accurately, even with a minimal decoder. This validates that Equation 1 captures an inherent invariance in images, encoded in the coefficients of $\mathbf{A}$ and $\mathbf{B}$.

B.3 Interpreting the Invariance

FINOLA's core operation (Equation 1), expressed as:

• $\Delta_x\mathbf{z}(x,y)=\mathbf{A}\widehat{\mathbf{z}}(x,y)$

• $\Delta_y\mathbf{z}(x,y)=\mathbf{B}\widehat{\mathbf{z}}(x,y)$

reveals that changes in the feature map (horizontal $\Delta_x\mathbf{z}$ and vertical $\Delta_y\mathbf{z}$) are linearly correlated to the current position. This means $\mathbf{A}$ and $\mathbf{B}$ capture an invariant local structure that holds consistently across all images. This invariance can be interpreted as a fundamental property of how image features change locally.

Despite some of my requests for intuitive explanations have not been fully addressed now (given the time constraint) and hence the work is still non intuitive, I still think the work is worth reading and disagree with the reviewers that just reject it. There are things to learn here.

I confirm the ACs that i'd like to see this published, and given the suggestions bay the authors in their responses (which should be pursued in the future for clearer illustrations), I raise my score form 6 to 8.

Thanks for your response: I think the introduction of the nonlinearity is crucial beyond (as you said) the technical help in the training. And that is the fundamental difference between (just technical) batch normalization and more clever normalizations over features such as Divisive Normalization or yours. I think the conection of your response with the literature I mentioned is pertinent.
Visualization of your nonlinearity depending on properties of the image may be an interesting extension of this work (in the future, I dont think it is necessary now).

$\color{blue}{**[Weakness 4 + Question 4]:**}$

You suggest that the normalization of the z features is critical for the wave equation. However, why is relevant the normalization over features? In which way is this normalization similar or different from other normalization schemes sucha as the usual batch normalization, or the (more interesting) local normalization as in the biological Divisive Normalization [Heeger Science 92, Carandini & Heeger Nature Rev. Neurosci. 12, Malo et al. J.Nonlin.Sci. 24], also used in state-of-the-art end-to-end optimized autoencoders [Ballé et al. ICLR 17] and in the local normalization in Alexnet [Krizhevsky et al. NeurIPS 12]. A possibility is that this normalization captures predictable information [Schwartz & Simoncelli Nat Neurosci. 01, Malo & Laparra Neur. Comp. 10] and hence transforms the (complicated) nonlinear relations into (easier to describe) linear relations, which are more invariant [Hernandez et al. Patt.Rec.Lett.24] and hence, it may make the linear differential formulation possible.

Why is relevant the normalization over features? Can you show reconstructions -both z and in the image domain- with and without normalization? Discuss possibly related normalizations in related work.

Thanks for raising these important points! While normalization plays a crucial role in training our FINOLA autoencoder, it doesn't directly influence the wave equation derivation.

Normalization doesn't influence the wave equation derivation

Let us clarify that using normalization or not does not affect the derivation of the wave equation. As the normalized $\widehat{\mathbf{z}}(x,y)$ is canceled out when deriving

$\Delta_x\mathbf{z}(x,y)=\mathbf{A}\mathbf{B}^{-1}\Delta_y\mathbf{z}(x,y)$

from FINOLA equations $\Delta_x\mathbf{z}(x,y)=\mathbf{A}\widehat{\mathbf{z}}(x,y)$ and $\Delta_y\mathbf{z}(x,y)=\mathbf{B}\widehat{\mathbf{z}}(x,y)$.

Impact of Normalization on FINOLA Training

We believe the normalization helps FINOLA training in two ways:

• Prevent exploding gradients: The normalization is essential for stabilizing the training process. Without it, we observed that the training simply doesn't converge due to gradient explosion. By keeping the feature values within a reasonable range, normalization helps ensure stable gradients during backpropagation.

• Introduce non-linearity: Our method performs normalization for each sample independently, by normalizing the feature values across all channels. This normalization introduces non-linearity by dividing each feature value by the standard deviation of the feature values across all channels within that sample. To demonstrate its importance, we compared it with standard batch normalization (BN), which can also stabilize training but is essentially linear during inference. We observed a substantial drop in validation accuracy (e.g., PSNR 25.1 for training, 16.3 for validation) when using BN, suggesting that the non-linearity introduced by our normalization method is crucial for the model's performance.

$\color{blue}{**[Weakness 5 + Question 5]:**}$

The comparison of the performance with other autoencoders is illustrative, however, I am curious about how this will compare with more interpretable autoencoders using not only the number of coefficients but the information they contain (i.e. entropy, as in the comparison with JPEG). what about JPEG2000 [Marcellin Kluwer 02], or end-to-end divisive-normalized optimized linear transforms [Balle et al. ICLR17].

Can you compare to more interpretable entropy autoencoders? (see above). Note that the dimension of the representation for unbounded variance/entropy is misleading.

Thank you for this insightful question. We agree that such a comparison would provide valuable insights.

However, we are currently unable to conduct a fair comparison with JPEG2000 or end-to-end divisive-normalized optimized linear transforms. Our current FINOLA implementation utilizes a much simpler pipeline:

• FINOLA uses uniform quantization per channel, which is not optimized in training.

• We have not implemented entropy coding for FINOLA.

We choose this simple pipeline as our focus is on demonstrating the core principles and effectiveness of FINOLA in capturing local image structure and achieving high-quality reconstruction, rather than optimizing for compression.

While this simpler pipeline achieves superior performance compared to JPEG, it lags behind JPEG2000, which is expected given JPEG2000's sophisticated entropy coding. For instance, under 0.2 bits/pixel, FINOLA achieves 25.6 PSNR on Kodak dataset, outperforming JPEG (24.0 PSNR) but having lower performance than JPEG2000 (28.0 PSNR).

In future work, we plan to investigate vector quantization and entropy coding schemes for FINOLA. This will enable a fair and comprehensive comparison with other entropy-based autoencoders like JPEG2000 and provide a more complete picture of FINOLA's performance and potential.

$\color{blue}{**[Weakness 3 + Question 3]:**}$

What is the meaning of the matrices A and B?. Can you visualize and interpret them? (this interpretation depends on the previous question). When you diagonalize, and you find the transformed coefficients, what (qualitative) information is transferred in the horizontal and vertical directions?

What is the meaning of the matrices A and B? (see above)

Thank you for this question! Understanding the meaning of matrices $\mathbf{A}$ and $\mathbf{B}$ is key to grasping FINOLA's core mechanism.

What do $\mathbf{A}$ and $\mathbf{B}$ represent?

Matrices $\mathbf{A}$ and $\mathbf{B}$ model the local structure within the latent vector space ($\mathbf{z}(x,y)$). They essentially describe how the feature map changes as you move horizontally or vertically across the image.

We've analyzed these matrices and confirmed they are full rank and diagonalizable. This allows us to express FINOLA as:

• $\Delta_x\mathbf{z}(x,y)=\mathbf{A}\widehat{\mathbf{z}}(x,y)=\mathbf{P}_A\mathbf{\Lambda}_A\mathbf{P}^{-1}_A\widehat{\mathbf{z}}(x,y)$

• $\Delta_y\mathbf{z}(x,y)=\mathbf{B}\widehat{\mathbf{z}}(x,y)=\mathbf{P}_B\mathbf{\Lambda}_B\mathbf{P}^{-1}_B\widehat{\mathbf{z}}(x,y)$

where $\mathbf{P}_A$ and $\mathbf{P}_A$ are the eigenvector matrices, and $\mathbf{\Lambda}_A$ and $\mathbf{\Lambda}_B$ are the diagonal eigenvalue matrices of $\mathbf{A}$ and $\mathbf{B}$, respectively.

Interpreting $\mathbf{A}$ and $\mathbf{B}$

Imagine each point in the latent space as having a set of "springs" attached to it, pulling it horizontally and vertically. Matrices $\mathbf{A}$ and $\mathbf{B}$ define the stiffness of these springs, while the eigenvectors ($\mathbf{P}_A$ and $\mathbf{P}_B$) represent the directions of these springs.

• Diagonalization: Diagonalizing $\mathbf{A}$ and $\mathbf{B}$ helps us understand these "springs" in a simpler way. It's like projecting the latent space so that each dimension (eigenvector) corresponds to an independent spring with a specific stiffness (eigenvalue).

• Horizontal and Vertical Changes: After this projection ($\mathbf{P}^{-1}_A$ and $\mathbf{P}^{-1}_B$), the horizontal change ($\Delta_x\mathbf{z}$) is simply a scaling of the current position along each independent dimension by the corresponding eigenvalue in $\mathbf{\Lambda}_A$. Similarly, the vertical change ($\Delta_y\mathbf{z}$) scales the position by the eigenvalues in $\mathbf{\Lambda}_B$.

In essence, $\mathbf{A}$ and $\mathbf{B}$ encode the directional "stretching" or "compressing" factors that govern how the feature map changes locally. The eigenvalues represent the strength of these factors along specific directions in the latent space.

Thanks for the followup, let us further explain the meaning of encoder and matrices $\mathbf{A}$ and $\mathbf{B}$.

$\color{blue}{**[Weakness 2 + Question 2]:**}$ What is the qualitative meaning of the encoder? (and hence of the encoding vector). Incidentally, this is not very much clear in the original paper on Mobile-Former [Chen et al. CVPR 22]. Can you show receptive fields of the coeficients of the encoding vector?. Can you compare vectors corresponding to different spatial locations?. Do they contain different seeds that are related using the differential equation?

What is the qualitative meaning of the encoder? (see above)

Thank you for this insightful question! We agree that clarifying the qualitative meaning of the encoder and the encoding vector is crucial. Here's a breakdown of the encoder's role and the nature of the encoding:

What does the encoder encode?

The encoder aims to compress an image into a single vector that captures the essential information needed for reconstruction using only local relationships (via FINOLA).

To achieve this, we add an attentional pooling layer on top of the Mobile-Former encoder. This layer:

• Starts with a learnable summary token.
• Aggregates information from the entire feature map (output of Mobile-Former) through cross-attention between the summary token and different spatial locations in the feature map.

This process results in a single output vector with a global receptive field, but without the explicit spatial information that a grid of feature vectors would provide.

As you suggested, we analyzed the feature maps before pooling and confirmed significant differences between spatial locations. We also observed clear variations in the attention maps across different images. This indicates that the attentional pooling learns to selectively attend to different image regions based on their relevance to the summary token.

What is the purpose of the encoding?

The purpose of this encoding is to transform an image into a new vector space where all images share a common local structure, regardless of their specific content. This local structure is defined by FINOLA (Equation 1):

• $\Delta_x\mathbf{z}(x,y)=\mathbf{A}\widehat{\mathbf{z}}(x,y)$

• $\Delta_y\mathbf{z}(x,y)=\mathbf{B}\widehat{\mathbf{z}}(x,y)$

These equations show that the horizontal ($\Delta_x\mathbf{z}$) and vertical ($\Delta_y\mathbf{z}$) changes in the feature map are determined solely by the current position ($\mathbf{z}(x,y)$) through a simple linear relationship (after normalization).

Why does such encoding matter?

This encoding enables a remarkably simple autoregressive process in the image domain – first-order and linear after normalization. By transforming the image into this compressed vector space, we can reconstruct the image based purely on these simple local relationships.

In essence, the encoder learns a compressed representation that discards explicit spatial information but preserves the essential information about local relationships within the image. This allows FINOLA to effectively reconstruct the image by exploiting these consistent local structures.