Reset Method based on the Theory of Manifold Optimization on Real Manifolds

Presentation: Firstly, several serious representation issues make the paper incoherent and difficult to follow, obscuring the key messages and contributions. Key issues include:

• Related Works and Preliminary Sections: The related works and preliminary sections are placed in the appendix. To improve clarity, many sections could be condensed or moved to the appendix, creating space for these key areas—for example, explanations of SGD, Adam, AdamW, and most experimental results in the tables.

• Main Algorithm Presentation: The algorithm presentation has gaps in clarity and completeness. For instance, variables such as $\zeta_{i_1}$ and $\zeta_{i_2}$ appear in the pseudocode but are unused in subsequent steps. In Eq. (5), the process of obtaining $\alpha_{i+1}$ is unclear. Additionally, the line "Set $R_{x_{i+1}}(-\alpha_{i+1} \nabla f(x_{i+1})) \leftarrow B_{x_i}(x_i)$" is confusing, as we cannot "set the retraction" to a specific value.

• Function $B_{x_i}(x_i)$: The definition and role of $B_{x_i}(x_i)$ are unclear and potentially redundant. What are its inputs and outputs? Although it’s described as a "step size correction function," suggesting it outputs the step size, in Eq. (5), it produces $x_{i+1}$, implying it represents the next model. Furthermore, it is stated that $B_{x_i}$ is selected from SGD, Adam, or AdamW—all Euclidean optimizers—suggesting $x_{i+1}$ might not lie on the manifold, making it impossible to compute $x_{i+2}$ directly from $x_{i+1}$. Additionally, the use of $B_{x_i}(x_i)$ as notation is confusing since it appears to take only a single input $x_i$. The reasoning for adding a correction step with operator $B$, which is supposed to be a primary contribution, is also not discussed.

• Experimental Results: The experimental results were provided on many real manifolds. However, it is unclear if the improvements are attributed to the Reset method or the manifolds. In particular, it would be more fair to compare the proposed method with other Riemannian optimizers, such as RSGD.

• Minor Formatting Issues: Multiple minor issues are present, such as inconsistent font sizes in table captions, unusual word choices (e.g., “contenting” in the pseudocode), inconsistent font styles (e.g., $x_i$ in the text but not italicized in Eq. (5)), table formatting inconsistencies, and citation format errors (e.g., \cite{} vs. \citep{}).

Soundness: In both theory and experiments, it is critical to compare the Reset method with other Riemannian optimizers, such as RSGD [2] or Riemannian-SAM [3], in addition to Euclidean baselines like SGD, Adam, and AdamW.

The theoretical contributions are relatively limited. Theorem 4.2 only demonstrates the gradient deviation of the Reset method when compared with SGD, Adam, and AdamW, and it includes an unavoidable term $\epsilon_0^2$, indicating a potentially unfavorable trait of the Reset method. Additionally, comparing the Reset method, a Riemannian optimizer, to Euclidean methods may be unfair from a theoretical perspective.

The experimental results are also not particularly insightful. Specifically, all results on the CIFAR-10, CIFAR-100, STL-10, and SVHN datasets achieve accuracies of at least 98%, making it challenging to discern performance differences. On the Market-1501 and DukeMTMC-reID datasets, variations of the Reset method applied to the same base optimizer yield considerable performance differences, suggesting that the method may be sensitive to variant choices, making tuning more challenging.

Contribution: To my understanding, the restart method was proposed by O'donoghue & Candes [1]. This work seems to be a trivial extension of this work to the Riemannian setting, which makes its contributions limited. Moreover, the motivation and intuition of the proposed algorithm are unclear, mostly due to the incohesive presentation.

It is not clear about the insight/motivation of each mentioned/proposed method presented through out the paper. The first one are derivations in inequalities (6) and (7). There is not picture or figure to illustrate clearly the advantage of method. I could only imagine that the first order of Taylor approximation is not good enough, thus we could obtain a better one via Armijo search, when some certain conditions satisfied. That goes through an interpolation between some bounds and expect that the interpolation will help to achieve a better bound. The work also mentioned the Barzilai-Borwein method but there is no explanation about reason for computing the correlation between two vectors.

Experiment results: For Image generation task, in Table 2, when adding your method with different type of manifold (in fact it is not clear for me why we have different type of manifold here), the proposed method is at best with spv, but does not work better with "e", "fr", "o", etc. In Table 3, there are some mixed performances between those method when comparing with each other except the "spv". Is there any explanation for the performance of each method?

For image generation task, there is no picture shown, rather than the number. Since average precision is already high, is the improvement noticeable in the pictures?

See summary