We sincerely appreciate the reviewer’s constructive comments and recognition of the key contributions of our paper. We address the reviewer’s concerns and questions point by point.

1. Though the model delivers better temporal consistency, it does not explicitly consider the temporal dimension.

Thanks for pointing that out. We will explore the explicit temporary modeling during animation in our future work.

2. Properties of the global encoder:

The primary role of the global feature is to encode the relative positions between disconnected components. This information is crucial for preventing penetrations during deformation, as the Diffusion Blocks alone cannot propagate information across disconnected mesh components.

To quantify this, we evaluate the penetration between inner components (e.g., mouthbag) and the outer face surface by reporting the percentage of penetrating vertices with and without our global feature. Specifically, we compute the signed distance function (SDF) of each vertex on the inner component with respect to the outer face surface in both the neutral and predicted meshes. Vertices are considered to be penetrating when their SDF values change from negative in the neutral mesh to positive in the deformed/predicted mesh. As shown in the table below, involving the global feature significantly reduces the number of such penetrating vertices. Please also refer to Fig. 5A (first column) in the main paper for the qualitative comparison.

Additionally, we will include two visualizations in our revision to illustrate the role of the global encoder. Specifically, we will show T-SNE plots of the global features compared against those from unmodified meshes in our dataset: (1) when the positions of connected components are randomly perturbed, and (2) when different numbers or types of connected components are removed. In both cases, the resulting features form separate clusters from the original ones, demonstrating that the global encoder captures structural changes in the mesh.

3. Discussion of limitations:

Thanks for pointing that out. In our revision, we will include qualitative examples of our failure cases particularly where our method struggles to deform shell-like structures or meshes with poor discretization. These failure cases are primarily caused by the following factors:

• Shell-like meshes (e.g., deformed spheres) often depict facial features such as eyelids and lips using texture rather than geometry, making meaningful deformation ill-posed or ambiguous.
• Poorly discretized meshes, where the main facial surface breaks into disconnected parts instead of a complete one, hinder feature propagation through the DiffusionNet backbone and make it difficult to establish consistent deformation bases.

4. Will the code and the dataset be released?

We will release the code, trained model weights for reproducibility. We are also working on releasing the full dataset, pending internal approval and legal clearance upon acceptance.

5. Ablation on Neural Jacobian Fields

We initially explored Poisson-based approaches such as Neural Jacobian Fields (NJF), but found several issues. In our early experiments, NJF required double-precision for stable training, which increased memory usage and slowed down the training. Moreover, passing gradients through the Poisson solver introduces additional computational overhead without a prominent benefit in terms of accuracy.

In contrast, directly predicting vertex displacements proved to be much simpler, more efficient, and yielded strong empirical results. Given these considerations, we decided to focus on the direct offset prediction and discontinued the NJF direction early in development.

6. Would the model be able to generalize to meshes with unseen components?

Yes. During dataset curation, we ensured coverage of common deformable facial components needed for rigging, such as mouth bags and eyeballs. Unseen components typically refer to non-deforming accessories (e.g., earrings, hats, or hair), which do not require animation.

For example, in the sample from Objaverse (Please refer to Fig. 1, third column; Fig. 7, last column), earrings are preserved without deformation. Similarly, in our video demo (Please refer to Section 3: Animating Generated Facial Mesh from Text-to-3D Model), components like hair and hats remain unaffected during animation. Even though these parts were unseen during training, the model naturally preserves them on the mesh.

We sincerely appreciate the reviewer’s valuable and constructive feedback and recognition of our work’s scope and complete comparison. For presentation-related issues, we will further polish the writing, correct grammatical errors, clarify technical details, and add the missing references as suggested. We address the reviewer’s technical concerns below.

1. Why is supervision applied in 2D, as the ground truth mesh is assumed? Why not simply supervise in 3D directly?

Thank you for the question, this gives us a chance to clarify an important part and contribution of our approach. Not all meshes in the training set come with artist-labeled 3D blendshape (deformed mesh) ground truth. Due to the high cost of manual annotation, only a subset of samples have full 3D labels. The remaining unlabeled data consists of only a neutral mesh without any corresponding 3D ground truth deformations.

Therefore, our use of 2D supervision is not intended to replace 3D supervision when it is available, but to enable learning from the unlabeled data. Specifically, we employ a two-stage training pipeline. During the first stage of training, we use 2D losses to supervise both labeled and unlabeled samples. For the unlabeled ones, we generate pseudo-2D ground truth using a 2D face animation model (based on MegActor) and a RAFT-based optical flow estimator.

Then, in the second stage, we fine-tune the model using the subset of data with accurate 3D ground-truth deformations with 3D loss. This two-stage strategy allows us to scale learning while still benefiting from the precision of 3D supervision when available.

2. The global encoder and FACS conditioning can be considered minor (task-specific) adaptations.

While the global encoder and FACS conditioning are indeed tailored to the facial rigging, we believe this task-specificity reflects a thoughtful and necessary adaptation to challenges that have not been addressed in prior work. The global encoder enables information flow across disconnected components, and FACS conditioning allows expression-aware deformation. These designs are essential components of our auto-rigging system.

Importantly, these architectural components are just one part of our contributions. Another key contribution of our work lies in the 2D supervision design that enables scaling-up training on unlabeled neutral facial meshes without rig annotations. Unlike prior methods that rely exclusively on costly 3D-labeled data, we introduce a tailored 2D animation and optical flow pipeline to generate pseudo-2D ground truth supervision for unlabeled data. To the best of our knowledge, RAF is the first scalable approach to make effective use of unlabeled data for neural auto-rigging.

Taken together, the architectural innovations and our scalable training framework form a unified and practical solution for real-world auto-rigging across diverse mesh topologies with multiple disconnected components, which we believe constitutes a meaningful step beyond prior work.

3. Global encoder compressing all vertices' features in one single vector destroys locality information.

The primary role of the global feature is not to capture fine-grained geometric details, but rather to encode the relative positions between disconnected components. This information is crucial for preventing penetrations during deformation, as the Diffusion Blocks alone cannot propagate information across disconnected mesh components.

To quantify this, we evaluate the penetration between inner components (e.g., mouthbag) and the outer face surface by reporting the percentage of penetrating vertices with and without our global feature. Specifically, we compute the signed distance function (SDF) of each vertex on the inner component with respect to the outer face surface in both the neutral and predicted meshes. Vertices are considered to be penetrating when their SDF values change from negative in the neutral mesh to positive in the deformed/predicted mesh. As shown in the table below, involving the global feature significantly reduces the number of such penetrating vertices. Please also refer to Fig. 5A (first column) in the main paper for the qualitative comparison.

Regarding the concern about capturing geometric details, the final deformation predictions are contributed by both the global feature and pre-vertex features from the Diffusion Blocks, enabling accurate deformation even for complex meshes. For instance, while the training set meshes have 1,704 vertices and 3,267 faces on average, our model generalizes well to in-the-wild meshes with significantly higher complexity, averaging 9,497 vertices and 17,958 faces, as shown in Fig. 7 of the main paper.

4. Ablation study on the global encoder.

We conducted an additional ablation as the reviewer suggested, removing the global encoder only while keeping all other components unchanged. As shown in the second row of the above table, the model’s performance drops noticeably compared to the full model (the last row) and becomes similar to the variant where the global encoder is removed together with other key components (the first row). This demonstrates that the global encoder plays a critical role, and its removal alone creates a significant bottleneck in the overall performance, even when the rest of the architecture remains intact.

5. How does the interpolation strategy work? Does it assume correspondence, and how can it be guaranteed?

Yes, our interpolation strategy assumes correspondence between meshes of different topologies. Specifically, we standardize the UV layout across all head meshes so that key facial features (e.g., eyes, mouth) are consistently mapped to the same regions in UV space. This shared UV space enables us to identify correspondences between vertices across meshes, even when their topologies differ. Leveraging these correspondences, we perform linear blending in geometry space to smoothly interpolate between different head shapes.

6. Is NFR retrained on the same data?

NFR’s released pretrained model didn’t provide training scripts, and it does not support meshes with multiple connected components by design. Therefore, we could not re-train the NFR model. Given these constraints, the fairest comparison we can make is to evaluate both methods on unseen artist-crafted and in-the-wild head meshes, strictly preprocessed to meet NFR’s input requirements. Specifically, for the data used to compare our model with NFR, we retain only the largest connected component of the neutral mesh, removing auxiliary structures such as eyeballs and mouthbags, and trimming inner surfaces of the lips and eyelids (please refer to Figs 3 and 4 in the supplementary appendix).

Besides, we also evaluated our model on NFR’s training dataset, ICT FaceKit (please refer to the first sample in our Fig.7 ), while our method did not train on this dataset, we still got comparable performance while handling multiple connected components.

We sincerely thank the reviewer for the valuable feedback and recognition of the key contributions of our work, including the tailored architectural design for handling disconnected mesh components and the scalable training pipeline. We address the reviewer’s concerns as below:

1. Limited Analysis of Failure Cases:

Thanks for pointing that out. In our revision, we will include qualitative examples of our failure cases particularly where our method struggles to deform shell-like structures or meshes with poor discretization. These challenging cases are mainly in-the-wild samples sourced from Objaverse, which is a large-scale dataset of general-purpose 3D assets but not curated for facial modeling. Therefore, quantitative analysis of failure cases is unfortunately not feasible as the lack of ground-truth rig annotations. These failure cases are primarily caused by the following factors:

• Shell-like meshes (e.g., deformed spheres) often depict facial features such as eyelids and lips using texture rather than geometry, making meaningful deformation ill-posed or ambiguous.
• Poorly discretized meshes, where the main facial surface breaks into disconnected parts instead of a complete one, hinder feature propagation through the DiffusionNet backbone and make it difficult to establish consistent deformation bases.

2. Incremental Architecture Changes:

Our RAF, built on top of DiffusionNet, is specifically aimed at addressing facial rigging challenges that previous neural methods are not capable of handling. The global feature inherits the information of global arrangement between disconnected components, and our FACS conditioning allows for expression-aware deformation. These designs are essential components of our auto-rigging system.

3. 3D mesh learning with 2D losses already has many prior works. The originality of the loss functions is limited:

While prior works [1, 2, 3] have explored applying 2D losses for 3D face reconstruction or generation using direct 2D supervision from input images or videos, our approach first generates pseudo-2D ground truth tailored to each unrigged mesh via a 2D face animation model and an optical flow estimator. We then train on large-scale unrigged meshes with 2D supervision generated above, eliminating the need for artist-provided annotations while enabling scalable training.

In addition, we empirically found that relying solely on 2D photometric loss is insufficient for accurate deformation. It often fails to capture subtle expressions as limited changes in pixel RGB values provide only marginal gradients. To address this, we introduce a 2D displacement loss based on pixel motion flow, which provides a stronger signal as an additional supervision for capturing fine-grained facial motions.

[1] Feng, Yao, et al. Learning an Animatable Detailed 3D Face Model from In-The-Wild Images. (TOG2021)

[2] Chan, Eric R., et al. Efficient geometry-aware 3d generative adversarial networks. (CVPR2022)

[3] Trevithick, Alex, et al. Real-time radiance fields for single-image portrait view synthesis. (TOG2023)

4. Evaluation Setup: only compared with NFR in the evaluation, lacking analysis with other SOTA methods.

Indeed, Neural Face Rigging (NFR) is our primary baseline as it is the most recent and relevant method for our task, to the best of our knowledge. As also mentioned by Reviewer cQPn, NFR is the latest SOTA method that we can compare with.

5. How is the model's ability to handle extreme expressions of driving images?

Please refer to Section 2 of our supplementary video: Retargeting Human Expression from videos (01:09–03:17), which showcases retargeting of real-world videos with a variety of extreme expressions (including one example from the Facial Extreme Emotions Dataset [4]) to unrigged 3D meshes. These results demonstrate our model’s ability to generalize to intense and diverse facial expressions.

[4] Drobyshev, Nikita, et al. Emoportraits: Emotion-enhanced multimodal one-shot head avatars. (CVPR2024)

We sincerely appreciate the reviewer’s constructive feedback and recognition of the key strengths of our work, including its support for meshes with multiple disconnected components and the scalable training enabled by unlabeled data. Below, we address the reviewer’s questions and concerns point by point.

1. Global features are compressed into a single vector, potentially losing details.

The primary role of the global feature is not to capture fine-grained geometric details, but rather to encode the relative positions between disconnected components. This information is crucial for preventing penetrations during deformation, as the Diffusion Blocks alone cannot propagate information across disconnected mesh components.

To quantify this, we evaluate the penetration between inner components (e.g., mouthbag) and the outer face surface by reporting the percentage of penetrating vertices with and without our global feature. Specifically, we compute the signed distance function (SDF) of each vertex on the inner component with respect to the outer face surface in both the neutral and predicted meshes. Vertices are considered to be penetrating when their SDF values change from negative in the neutral mesh to positive in the deformed/predicted mesh. As shown in the table below, involving the global feature significantly reduces the number of such penetrating vertices. Please also refer to Fig. 5A (first column) in the main paper for the qualitative comparison.

Regarding the concern about capturing geometric details, the final deformation predictions are contributed by both the global feature and pre-vertex features from the Diffusion Blocks, enabling accurate deformation even for complex meshes. For instance, while the training set meshes have 1,704 vertices and 3,267 faces on average, our model generalizes well to in-the-wild meshes with significantly higher complexity, averaging 9,497 vertices and 17,958 faces, as shown in Fig. 7 of the main paper. Given the effectiveness of this setup, we found that a more complex global encoder (e.g., a hierarchical encoder) might not be necessary.

2. Domain gap of RAFT and MegActor’s pretrained models on stylized/non-humanoid meshes. Provide error analysis.

Yes, there is indeed a domain gap when directly applying the pretrained RAFT and MegActor models to stylized head mesh renderings. To address this, we fine-tuned both models using rendered ground-truth optical flow and expression images from our artist-annotated training set, as described in L222–224 of our paper.

For error analysis, we used similarly rendered ground-truth data from the artist-annotated test set, with all images rendered at 512×512 resolution. The fine-tuned RAFT achieves an end-point error (EPE, lower is better) of 0.99, i.e., the average per-pixel Euclidean distance between predicted and ground-truth optical flow vectors is equal to 0.99. The fine-tuned MegActor yields a mean absolute error (MAE) of 0.00346 between generated and ground-truth images, computed over RGB values normalized to [0, 1]. Please also refer to Figure 1 in our supplementary material for qualitative comparisons of the generated 2D supervisions.

These results indicate that, after fine-tuning, both RAFT and MegActor handle stylized or non-humanoid meshes with high accuracy and show strong robustness against domain gaps between realistic and stylized inputs.

3. Degraded performance on shell-like meshes and meshes with poor discretization. How does RAF handle them?

We will include qualitative examples in the revision to show that our model typically leaves such meshes undeformed. Even though we would also like to clarify that such meshes are not typical inputs for facial rigging pipelines and are generally not considered by prior methods. Shell-like meshes (e.g., deformed spheres) often depict facial features such as eyeballs and mouths using textures instead of explicit geometry, making them ill-suited for animating expressions like eye closure or mouth opening. Meshes with poor discretization, where the main facial surface can break into multiple disconnected pieces, are very challenging to rig and annotate, even for professional artists. We include them in our evaluation to explore the robustness and limitations of our approach beyond its intended scope.

4. Are there any preprocessing recommendations for failure cases?

For poorly discretized meshes, we experimented with preprocessing such as remeshing to improve mesh quality. However, we found that remeshing often mistakenly connects disconnected components (e.g., eyeballs and mouth bags) to the main facial mesh, which deteriorates the pose prediction. For instance, animation of eye gazing requires the eyeballs to deform independently within the eyelid region. If these parts are merged, the model would struggle to predict correct deformations. As a future direction, incorporating a diffusion operator defined on a high-quality background triangulation [1] may help with these failure cases.

[1] Nicholas Sharp, Yousuf Soliman, and Keenan Crane. 2019. Navigating intrinsic triangulations. ACM Trans. Graph. 38, 4 (2019)

5. Suboptimal inference speed could hinder real-time deployment.

As discussed in L242–L244 of our paper, our model is lightweight, with only 5.4M parameters. It performs a single forward pass to generate all the FACS poses to form a blendshape rig for each input mesh, a process that takes on average 8.72s on an Apple M2 Max CPU and 3.1s on an Nvidia T4 GPU (test set average: 1,750 vertices and 3,362 faces).

Importantly, this process is conducted offline. Once the blendshape rig is generated, animations can be executed in real-time at deployment using standard linear blendshape interpolation, achieving over 1000 FPS on an Apple M2 Max CPU in our testing.

6. Moderate originality as core innovations build incrementally upon DiffusionNet.

Our architectural design is specifically tailored to address facial rigging challenges that previous neural methods have not effectively handled. The global feature enables information flow across disconnected components, and FACS conditioning allows expression-aware deformation. These designs are essential components of our auto-rigging framework. More importantly, our design of learning 3D deformation with 2D supervision enables scalable training on unlabeled data.

Taken together, the architectural innovations and our scalable training framework form a unified and practical solution for real-world auto-rigging across diverse mesh topologies with multiple disconnected components, which we believe constitutes a meaningful step beyond prior work.

We sincerely thank the Area Chair for facilitating the discussion and supporting a thorough review process.

We also appreciate Reviewer U3KH for the follow-up and for reconsidering the evaluation. We appreciate the constructive feedback and will incorporate the suggested discussions and experiments into the revised manuscript.