Learning Unorthogonalized Matrices for Rotation Estimation

Thanks for providing feedback on our submission. We address individual concerns below.

Q1: Contrain orthonormality with $I-M^{\intercal}M$?

A1: Thank you for your suggestion. We compare our method with adding the rotation constraint loss, which is implemented by ProHMR [1]. The PA-MPJPE is 55.3 vs. 54.8 (PRoM). The experimental setting is the same as the ablation study. Adding this loss makes the learning in between PRoM and previous orthogonalization incorporated methods. The results are also in between.

[1] Kolotouros, Nikos, et al. Probabilistic modeling for human mesh recovery. CVPR'21

 

Q2: Quaternions are also used in other works; Continuity lacks solid theorems

A2: In our work, we focus on whether to use orthogonalizations during training with the matrix representation. 6D is one of the representative works that incorporate orthogonalizations. 6D is also frequently used in pose estimation tasks. Our work mainly improves upon 6D representation, which will have an impact on the understanding of learning rotation matrices and the further learning paradigm of pose estimation. We acknowledge that other rotation representations, e.g., quaternions, are used in other works, comparing 6D with quaternion with solid theorems is out of the scope of our paper.

However, we still provide a more detailed empirical comparison among all rotation representations. The experimental setting is the same as the ablation studies, e.g., Table 3 (a). The results are shown below.

Q3: include the discussed methods

A3: Thank you for your suggestions. We will include all the experiments above in our manuscript.

Thanks for providing feedback on our submission. We address individual concerns below.

Q1: Lack of comparison with recent methods

A1: Thank you for pointing this out. Considering [B] didn’t release their code, we reproduce the results of [A] with our ablation study setting. With the only change of rotation representation from 6D to RPMG-6D, [A] achieves the PA-MPJPE of 55.8, whereas PRoM has a lower error of 54.8. We want to highlight that one of the main contributions of our paper is that we achieve much better results on the challenging real-world human pose datasets, e.g., 3DPW and Human3.6M.

Q2: PRoM is applicable to other baselines in pose estimation?

A2: Yes, PRoM will improve other baselines. We have already included these experiments in the last paragraph of Sec. 5.3. The detailed numbers are presented in Table 4 in the Appendix due to the space limit.

Q3: it has been shown that predicting pseudo rotation matrix and quaternion without orthogonalization is also possible

A3: Yes, also as reviewer h3do commented, direct regression of rotation matrices is intuitive and has been used in the past literature. However, our contribution is mainly the gradient and optimization analysis, which shows that incorporating orthogonalization in the training is problematic. Our work gives theoretical support for why using unorthogonalized matrices is more beneficial.

Thanks for providing feedback on our submission. We address individual concerns below.

Q1: The proposed method is somewhat basic and is frequently used in the literature.

A1: PRoM is just a simple yet effective solution according to our optimization analysis, which is the main contribution of our paper. Most works still incorporate orthogonalizations, e.g., in the fields of point cloud pose estimation, human pose and shape estimation tasks. People tend to apply orthogonalizations on rotation matrices to ensure it’s valid to rotate the downstream inputs. It’s a tradeoff between the validity of rotation matrices and optimization. Our results show that the optimization matters more.

Q2: Eq. (5) is vague and need to explicitly derive the two terms will not die out? Q4: Fig. 3 is vague? Incorporating orthogonalization is obviously more complex.

A2 & A4: We answer the concerns together since they are related. For the ambiguous gradient, we only give some intuitive derivation (Eq. (5)) and Fig. 2 (a) is the exact empirical verification. Fig. 3 serves as a supplement to Fig. 2 (a). We leave explicit theoretical support for this in Sec. 4.2. We answer the questions separately below in detail.

A2: Eq. (5) is an observation of the gradients from the orthogonalizations and serves as the motivation of PRoM. The explicit influence of these conflicting terms on the optimization is shown in Sec. 4.2. We leave strong theoretical proof for the optimization analysis instead of gradient because the optimization analysis is more established in the convergence and optimization papers (see references below) Therefore, we think it’s more convincing to demonstrate the negative impact of the orthogonalizations in the optimization analysis. Also, the two terms will not die out as shown in Fig. 2 (a).

A4: Fig. 3 is an empirical study with different $\sigma$s for comprehensive understanding. The focus is that the larger $\sigma$ is, the more diversity of gradients will be. It serves as a supplement of Fig. 2 (a). As stated in the review, it is indeed obvious that the gradient is more complex for orthogonalization. We are actually targeting the complex gradient and demonstrating both the theoretical and empirical benefits of removing this complex gradient during the training.

Q3: better upper bound does not guarantee anything?

A3: The upper bound analysis of loss is widely adopted both in machine learning theory and its application in vision [1,2,3,4,5,6]. It both considers the convergence rate and irreducible error. Our work follows the convention of analyzing the different upper bounds for orthogonalization incorporated methods and PRoM. For the whole term, the only difference is caused by the orthogonalization function, which gives non-negligible $\psi(B_i)$ and further leads to a slower convergence rate and worse upper bound. Additionally, we show by experiments that PRoM converges much faster than orthogonalization incorporated methods (Fig. 3 (a) and (b)).

[1] Cohen, Jeremy M., et al. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR’21

[2] Ahn, Kwangjun, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. ICML’22

[3] Fehrman, Benjamin, Benjamin Gess, and Arnulf Jentzen. Convergence rates for the stochastic gradient descent method for non-convex objective functions. JMLR’20

[4] Mertikopoulos, Panayotis, et al. On the almost sure convergence of stochastic gradient descent in non-convex problems. Neurips’20

[5] Lei, Yunwen, et al. Stochastic gradient descent for nonconvex learning without bounded gradient assumptions. TNNLS’22

[6] Chen, Joya, et al. DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training. ICLR’23

Q5: What if we do not have ground truth rotations?

A5: Our analysis still applies when ground truth rotations are not available. In Theorem 1, we do not need the presence of the ground truth $R$, but only the predicted rotation matrices $\hat{R}$. Also, empirically, in sec. 5.2.3, we show that when the whole task only has the downstream supervision without ground truth rotation, it can still perform better than orthogonalization incorporated methods.

The authors have not answered my concerns regarding whether the two of the three terms cancel each other. This is the core argument in the paper, and I have already demonstrated the case where they cancel each other in the previous round.

If there is no ground truth rotation, then there will be no term that makes the (pseudo-) rotation estimate close to a rotation matrix. Of course, this might be easier for optimization, but the estimate can be far from a rotation matrix, which can cause the quality of the solution to be poor.

I'm not at all convinced by the upper-bound argument. The papers the authors mentioned show upper bounds to guarantee the least bound for convergence for some specific scenarios. However, comparing upper bounds cannot prove that orthogonalization is worse than no-orthogonalization.