EgoLM: Multi-Modal Language Model of Egocentric Motions

• The method contribution: most of the method designs in this work are common practice from the other VLM or MLLM works, like instruction tuning tricks, VQ-VAE encoding, and training, joint space training, multi-modal tokenization, etc. Please add a comparison between previous works, or which part is your unique contribution.

• Though this work focuses on the sparse headset pose data, many existing 3D pose datasets on videos (like BABEL) can also be formalized as similar benchmarks. At least, these data can be considered as the data supplementary in the pre-training. Moreover, there are also many baselines from the other tracks that can be adopted here, like different recovering methods from incomplete inputs.

• Even the motion captioning part, many datasets like BABEL can also be seen as a testbed to verify the proposed method.

Overall, both the method and experiment parts have a lot of room for improvement.

1. How does the method deal with scenarios when both the video and sensor data become ambiguous? For example, if the person is using the treadmill and faces the white wall, can the sensor data confuse between walking and running?
2. How is the correspondence/association established between the two modalities - clip embedding and pose data?

While I believe that using an egocentric body model with a language model is a promising research direction, I have concerns about its practicality in real-world applications due to potentially slower inference speeds. If inference time is slow, it may hinder real-time motion tracking capabilities, even with an online setup.

Regarding motion tracking, state-of-the-art methods such as [1], [2], and [3] were available at the time of submission to ICLR. Comparing EgoLM's results to these more recent methods could provide a more comprehensive evaluation. Is there a specific reason for selecting BoDiffusion and AvatarPoser for comparison instead?

Given EgoLM's significant performance boost when using image features, I also suggest comparing inference times with and without image features to see if the input type affects performance. Does inference time remain consistent regardless of input type?

[1] Du, Y., Kips, R., Pumarola, A., Starke, S., Thabet, A., & Sanakoyeu, A. (2023). Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 481-490).

[2] Feng, H., Ma, W., Gao, Q., Zheng, X., Xue, N., & Xu, H. (2024). Stratified Avatar Generation from Sparse Observations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 153-163).

[3] Zheng, X., Su, Z., Wen, C., Xue, Z., & Jin, X. (2023). Realistic full-body tracking from sparse observations via joint-level modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 14678-14688).

• Limited technical novelty: Sections 3.2 and 3.3 are almost identical to T2M-GPT, as they discuss training motion VQ-VAE and training an autoregressive LM on motion tokens. This makes it challenging to consider these sections as novel contributions. Section 3.4, which introduces multi-modal multi-task training, is the only unique technical contribution; however, it merely combines different modalities to instruction-tune the LM without introducing new technices.
• The proposed framework does not include design for egocentric video. It lacks a specific algorithm tailored to egocentric views and instead emphasizes combining different modalities, which does not need to be egocentric applications.
• The model is only compared to baselines that do not include video. From the experiments, it seems that the performance improvement may be attributed to the addition of a video rather than the proposed framework itself, as the model’s performance is almost similar to the baselines without video. Experiments under identical conditions are needed to clearly establish the framework’s contribution.