[Question 1]

The title of the paper is not appropriate in my opinion. The title emphasizes a face alignment contribution, while the content focuses more on face landmarks detection.

[Answer]

Our main task is 3D alignment, while landmark detection is only an auxiliary task. The major goal is to predict the 3DMM parameters to reconstruct 3D head shape. As shown in Figure 1, our model mainly consists of a 3DMM parameter regression branch and an auxiliary facial landmark detection branch. The 3D alignment relies on 3DMM parameters regression branch which can regress a set of 3DMM parameters and reconstruct the 3D head shape by a differentiable FLAME decoder. In the paper, Figure 1 shows the overall 3D alignment pipeline, Sec. 3.1 introduces the proposed 3D alignment framework, and Sec. 3.4 describes the overall losses for both landmark detection, as well as 3D shape reconstruction and reprojection. Besides, the Euler angle loss is proposed in Sec. 3.4.1 for improving the 3D alignment via 3D pose estimation. Thank you for your comment. We will further improve the presentation of Sec. 3.1 for the whole 3D alignment framework.

[Question 2]

Table 2 indicates results that are comparable to the state-of-the-art but for some cases do not improve on it.

[Answer]

Thank you for the comment. Please refer to the Common Questions and Responses 1 for the comparison to SynergyNet.

[Question 3]

Parameters of a 3DMM are regressed in this work. However, it is not clear how much the choice of the 3DMM impacts on the results. Did the authors try with different 3DMMs? In the ablation study it is not clear the impact of the 3DMM on the results. The contribution of the used 3DMM on hte final results is not clear. Did the authors try with different 3DMMs? Can they show results using different 3DMMs? In the ablation study it is not clear the impact of the 3DMM on the results.

[Answer]

Thank you for your valuable suggestion. The 3DMM parameters used in DAD-3DNet and the proposed Trans3DHead are adapted to FLAME, which are extensions of the standard 3DMM parameters. The number of the extended 3DMM parameters used in the final Trans3DHead is 413, including 300 shape parameters, 100 expression parameters, 3 jaw parameters, 6 rotation parameters, 3 translation parameters, and 1 scale parameter to control FLAME, which allows to use the first few parameters of the 300 shape parameters, 100 expression parameters, and 3 jaw parameters to express the shape and expression.

We have conducted experiments with two additional 3DMM parameter settings. The first one is with the same number of 3DMM parameters as in SynergyNet (40 shape parameters, 10 expression parameters without jaw parameter), while the second one is used in RingNet (100 shape parameters, 50 expression parameters without jaw parameter). Keeping the original setting of 6 rotation parameters, 3 translation parameters, and 1 scale parameter in FLAME, we use two parameter settings of 60 parameters and 160 parameters to train our Trans3DHead and DAD-3DNet on the DAD-3DHeads dataset, respectively.

As shown in Table 8 in the appendix, regressing more parameters is helpful for better reconstructing 3D head shape for both DAD-3DNet and the proposed Trans3DHead. Besides, the proposed Trans3DHead is superior to DAD-3DNet in all three parameter settings. When only regressing 60 parameters, our model shows more significant advantages over DAD-3DNet, which also indicates that the cross-attention mechanism in Trans3DHead is effective to achieve the information communication among different elements of the 3DMM parameters.

[Question 4]

An analysis of the computational cost of the method in comparison with other solutions is missing. For landmarks detection the capacity of the approach to work in real time is important. Authors should clarify this point. Authors should discuss and illustrate the computational cost of the method in comparison with other solutions is missing.

[Answer]

Thank you for the comment. Please refer to the Common Questions and Responses 2.

[Question 1]

Less qualitative results: only 1 dataset is used for qualitative results. More is required.

[Answer]

Thank you for your comment. We have added qualitative results on AFLW2000-3D dataset to visually display the facial landmark detection results predicted by SynergyNet, DAD-3DNet, and the proposed Trans3DHead. As shown in Figure 12 in the appendix, our Trans3DHead performs better in the mouth, face contour regions, and atypical head poses.

[Question 2]

Even though the authors insist that the proposed method is efficient; while there is no report for the time complexity in their results.

[Answer]

Thank you for the comment. Please refer to the Common Questions and Responses 2.

[Question 3]

In ablation study, when comparing Euler+QA and MARR+QA, the accuracy improvement is quite limited. It is unclear the accuracy improvement was due to the components, or not. Why in ablation study, the three combination only can improve the final accuracy, even though partial combination is not that effective?

[Answer]

Thank you for your comment. Since each component is proposed to address a specific problem, the accuracy improvement is limited when each component is used independently. However, when considering the comprehensive problems, the combination of all the components will achieve better results. Specifically, the Euler angle loss is only proposed for head pose estimation, so comparing Euler+QA to QA, it only improves the pose error and Z5 accuracy of the entire 3D head, but it does not improve the landmark detection and chamfer distance for face shape, as 3D head is quite related to head pose, but face shape is relatively not. As for MARR, as commented by Reviewer QjYA, the average face may have some limitation in handling large head pose variations, therefore, its effectiveness is more observed when the Euler angle loss is applied to improve the performance under large pose variations, that is, from Euler+QA to Trans3DHead (Euler+QA+MARR). We will make this point clearer in the paper.

[Question 4]

Explanations for some equations are rather blurry (for eqs 2 and 3). There is less explanation for how to proceed the equations from 2 to 3.

[Answer]

Thank you for pointing out this. We have updated the Eq. (2) and Eq. (3) to make it easier to understand. The Eq. (2) is the original calculation form of the QAMem module, which represents that each query corresponds to a separate memory rather than the shared memory. In order to reduce computations in practice, according to the associative law of multiplication, the operation order of multiplication can be changed to the form of Eq. (3), with the advantage that A $\times$ M can be precomputed only once, resulting in only one matrix multiplication for each query i, in contrast to two matrix multiplications for each query i in Eq. (2). This significantly reduces redundant computations and memories.

Specifically, with Eq. (2) the complexity in terms of multiplications is O(NSd + NSdd)), while that of Eq. (3) is O(NSd + Ndd). Therefore, Eq. (3) reduces the computation of O(Ndd(S-1)), which is significant especially when N=68 and S=8 $\times$ 8 in the Trans3DHead model.

[Question 1]

The contributions seem to be a combination of deep learning tricks like multi-task structure, QAMem module, MARR and Euler Angles Loss, the improvement of each module seems to be incremental and the combination of these incremental contributions, do not become a significant contribution.

[Answer]

Thank you for your review but we do not agree. This work is well motivated to solve respective problems instead of using deep learning tricks to improve the performance. First, this is the first work in regressing 3DMM parameters through Transformers, which can help information communication among 3DMM parameters via attention mechanism in joint estimation. With this architecture, the proposed multi-task framework for 3D head alignment simultaneously predicts 3D head shape and 2D landmarks. This not only facilitates applications by directly providing multiple available outputs, but also helps mutual improvements between the two branches. Our experimental findings show that with this design the proposed method is more robust with reducing 3DMM parameters (see A.1.6 and Table 8 in appendix), thanks to the unified Transformer architecture for two branched tasks.

Second, the proposed lightweight QAMem module is an interesting and valuable idea (supported by Reviewers QjYA and PGZe) for dense prediction tasks requiring high-resolution feature maps. It ingeniously uses separate memories with low-resolution feature maps and designs an efficient implementation to get rid of the high-resolution feature map requirement. The QAMem design enables improved accuracy with minimal additional computational burden.

These are the two major innovations in this paper. We do not agree that they can be simply categorized into deep learning tricks or incremental combinations.

[Question 2]

The pose estimation results on AFLW2000-3D dataset, shown in Table 2, don't support that the proposed approach achieve better performance than SOTA. Th MAE, pitch and roll of the proposed approach are not as good as SynergyNet published in 2021.

[Answer]

We may did not make it clear. Thank you for the comment. Please refer to the Common Questions and Responses 1 for the comparison to SynergyNet.

[Question 3]

The ablation study in Table 3 don't support the effeictiveness of proposed modules as well. For example, the performance of Cham Dist for Trans3DHead is not as good as the baseline.

[Answer]

Thank you for the comment. The measurements and comparisons in Table 3 are comprehensive, including evaluations of not only 3D face shape prediction (Chamfer distance), but also face landmark detection (NME), head pose estimation (pose error), and 3D head shape estimation (Z5). In DAD-3DHeads benchmark, Chamfer distance is used for measuring the 3D shape of human face, and the proposed method achieves quite comparable results with the baseline in terms of Chamfer distance (2.796 vs. 2.791). Except this the proposed method outperforms the baseline in all other metrics. Especially, the proposed model performs better than the baseline with Z5 (0.9585 vs. 0.9578). As Z5 is used for measuring 3D head shape, it is more important than Chamfer distance for 3D face shape only.

[Question 1]

For the qualitative comparisons, only DAD-3DNet is compared with. From table 5, DAD-3DNet is already worse than the proposed method. While SynergyNet, which performs better than the proposed method, is not compared with in the visual demos.

[Answer]

Thank you for the comment. Please refer to the Common Questions and Responses 1 for the comparison to SynergyNet.

In the 3D head alignment task, the DAD-3DNet is the most relevant work and the closed competitor to the proposed Trans3DHead, so the most important experimental comparisons of this work are shown in Table 1. Thank you for the encouraging statement that “DAD-3DNet is already worse than the proposed method”.

Anyway, including SynergyNet for visual comparison is indeed a good idea. Thank you for your suggestion. Besides the new Figure 10, to further illustrate the difference between the SynergyNet and the proposed Trans3DHead, we have additionally visualized some examples of head pose estimation predicted by the SynergyNet and the proposed Trans3DHead on the AFLW2000-3D dataset, as shown in Figure 11 in the appendix. It can be observed that the SynergyNet performs slightly better in predicting pitch and roll, while the proposed model has more advantages in handling side faces.

[Question 2]

section 3.2, the description about QA memory is not super clear. Does the proposed method use more features maps to trade for a lower resolution? What is the benefit in doing this? Can the authors give more explanation and proof about the QA module and its effectiveness? Does the module use more feature maps to trade for lower resolution?

[Answer]

Thank you for your comment. Using low-resolution feature maps often leads to a decrease in accuracy. Based on an observation, we find that in Transformers the queries will act differently if the memory is of higher resolutions. Therefore, QAMem is proposed to simulate the working mechanism of high-resolution feature maps in the actual low-resolution feature maps we used to improve the accuracy. In practice, the QAMem layer can be simply implemented as a convolutional layer with 1 $\times$ 1 kernel, 1 stride, and N groups. Note that we do not use more feature maps to trade for a lower resolution. In our implementation of QAMem in Eq. (3), the feature map channels are the same $d$ channels, but N groups of 1 $\times$ 1 convolutions are applied to transform the $d$ channels per query. As a result, QAMem provides discriminative values for different queries with marginal additional computation, and enhances discriminative abilities of the model on using low-resolution feature maps.

Note also that the implementation of applying high-resolution feature maps usually uses additional network for up-sampling, such as the commonly used FPN structure where the parameters, computation, and memory requirement are significantly larger.

[Question 3]

Due to large variation of head poses and face shape, it might not be a good idea to compute the average face and predict the residuals.

[Answer]

The average face provides a general initial topology of the landmark detection with which the predicting task will be easier with small or not too large head poses. When encountering large head poses, the decoders with sufficient training are also able to output larger residual to correct the initial landmarks. Thank you for pointing out this. To address large variation of head poses and face shape better, a better way could be considering multiple average faces, which will be our future study.

[Question 4]

The paper proposes a Euler Angles loss, but from figure 4, it requires an estimation of the Euler angles from predicted 3DMM parameters. Therefore another module needs to be introduced do the estimation. This module introduces additional errors and might not benefit the supervision for 3DMM parameters. section 3.4.1, how are Euler angles estimated from 3DMM parameters? Is a neural network used here?

[Answer]

The estimation of the Euler angles from the predicted 3DMM parameters is a deterministic matrix calculation, with no additional network or learnable parameters in introducing errors. Specifically, the 3DMM parameters used in DAD-3DNet and the proposed Trans3DHead are adapted to FLAME, which are extensions of the standard 3DMM parameters. The number of the extended 3DMM parameters regressed in the final Trans3DHead is 413, including 403 3DMM shape parameters, 6 rotation parameters, 3 translation parameters, and 1 scale parameter. Then, the Euler angles are estimated by a deterministic matrix calculation from the predicted 6 rotation parameters, because any orientation can be expressed as a composition of rotations. In our implementation, we use SciPy, a scientific computation Python library, through scipy.spatial.transform.Rotation.as_euler().