ChatCam: Empowering Camera Control through Conversational AI

Thank you for your time and valuable feedback. While we are encouraged by your appreciation for our ideas and results, the issues you point out are crucial.

Collision. Collisions can indeed occur, especially when the scene is complex or the user does not provide enough anchor points. To keep the pipeline simple yet effective, we do not handle collisions specifically. In particularly complex scenes, such as indoor scenes with multiple rooms, users can provide more detailed text descriptions to guide ChatCam in finding anchor points to avoid collisions. Moreover, if ChatCam generates trajectories that include collisions, users can guide it to make corrections through further dialogue, as discussed below and demonstrated in the example we provide in the general response, where ChatCam first generates a trajectory that collides with the wall, so the user avoids it by providing more anchor points.

User Interaction & Modification. The user and ChatCam can engage in multiple rounds of dialogue to iterate and improve the generated trajectories, similar to interactions between a director and a photographer. We verify this advantage in the example provided in the general response. In this example, ChatCam first generates a trajectory that collides with the wall, so the user avoids the collision by providing more anchor points. Furthermore, the user proposes to extend the trajectory with a dolly zoom and complete this improvement simply through dialogue.

Is Natural Language Really the Best Way for Users to Interact? As discussed in the general response, ChatCam allows lay users without professional knowledge such as 3D rotation, translation, camera intrinsics, and keypoints to produce production-quality cinematography. ChatCam also benefits from a pre-trained LLM with multi-lingual understanding, enabling users to interact with ChatCam in languages other than English. Compared with professional software and plug-ins with UI interfaces, this greatly lowers the entry threshold for lay users.

Through ChatCam, users can engage in further conversation with ChatCam to interactively and iteratively refine the target camera trajectory, much like how a human director gives natural language suggestions to a human cinematographer in a movie post-production process.

Putting ChatCam into perspective, its contributions are not limited to current camera applications. We believe our explorations in conversational AI’s capability for multimodal reasoning and planning have important implications for building embodied AI agents.

More Complicated Trajectories. In in the general response, we present an additional example of a complex camera trajectory. This trajectory involves panning downstairs, passing through a tunnel, performing a U-turn, and executing a dolly zoom within a complex room. We believe such intricate operations are not easily achievable by lay users through professional software and plug-ins with UI interfaces, and demonstrate "a wide range of trajectories on complex scenes with several interesting elements," as attested by reviewer dvT5.

Thank you for your positive feedback as well as thoughtful suggestions and questions. Below, we address your points individually.

Limited Scenes & Overfitting & Trajectory Length. The cases we test cover "a wide range of trajectories on complex scenes (indoor/outdoor, object/human-centric) with several interesting elements", as attested by reviewer dvT5. We do not observe significant overfitting, due to the comprehensive dataset and the leverage of CLIP. Furthermore, ChatCam allows users to correct suboptimal results through conversations, further mitigating possible errors.

The length of each trajectory in our dataset is about 60 frames, matching the length of a single inference of CineGPT. ChatCam has no limit on the length of the trajectory, and users can continue to add requirements to infer CineGPT multiple times and extend trajectories.

Please also refer to the new example we provide in the general response, where the trajectory involves panning downstairs, passing through a tunnel, performing a U-turn, and executing a dolly zoom within a large complex room.

Prompts about Speed. Prompts such as "smooth panning speed" affect the speed of the trajectory generated by CineGPT. Specifically, for the same duration (frame number), trajectories can cover different distances. The table below illustrates the effect of such prompts.

Validating Anchor Determinator. In Figure D of the attached PDF, we qualitatively show how the same text prompt results in different trajectories in different scenes due to Anchor Determination. Also, in the original submission, we conduct an ablation study on the Anchor Determinator, with quantitative results in Table 1.

Anchor Determinator Inference Time. On a single NVIDIA RTX 4090 GPU with 24 GB RAM, a single inference of the Anchor Determinator takes approximately 10-15 seconds. The majority of this time is spent on anchor refinement rather than CLIP-based initial anchor selection.

Trajectory Dataset. We will introduce the dataset in detail in Section 4.2 as suggested. We enumerate the textual descriptions of camera trajectories and then build a camera trajectory in Blender for each description. The camera movements included are basic translations, rotations, focal length changes, and combinations of these (simultaneous or sequential). We also include some camera trajectories mentioned in professional cinematography literature and build them in Blender. All camera trajectories cover similar volumes, with aligned translations and rotations in the first frame.

Dynamic Scenes. CineGPT can be directly applied to dynamic scenes due to its independence from specific scenes. Extending our approach to dynamic scenes would be straightforward by introducing a timestamp in the Anchor Determinator so that the "anchor" contains temporal coordinates in addition to spatial coordinates. We will show results on dynamic scenes in an updated version.

Inaccurate References. Thank you for pointing this out; we will correct them.

Thank you for your time and valuable feedback. While we are encouraged by your appreciation of our efforts, the issues you point out are constructive. Below, we address them:

Conversation Example & Planning Output. Figure 2 shows an actual conversation with the LLM (with visualizations and translation from JSON to human language by the authors). As suggested, in the new example in the general response, we show dialogue where users and ChatCam gradually correct and improve the output trajectory. At the same time, we also post our designed LLM prompt in the general response to help better understand such conversations.

Reproducibility. To increase the reproducibility of trajectory generation through user-friendly interaction in Section 3.3, we post our designed LLM prompt in the general response and will make it public upon acceptance.

L142-L146 describes the training of CineGPT. To provide more details: i) In the first stage, we perform unsupervised pre-training in an autoregressive manner, where it conditions on previous tokens in a sequence to predict the next token [1, 2]. This process involves maximizing the likelihood of the token sequences (Equation 3), allowing the model to capture relationships between tokens and generate coherent outputs. ii) In the second stage, we use the paired text-trajectory dataset to supervise CineGPT for text-to-trajectory and trajectory-to-text translation, to obtain the model weights we finally use.

We are willing to answer any further technical questions to increase the reproducibility of our work.

Multiple Outputs/Ground Truths. To strengthen our evaluation by considering multiple possible ground truth trajectories, we added a set of experiments where we built multiple GT trajectories. We calculated the distance between the predicted trajectory and the nearest GT trajectory using a new metric $\min_i \|\| x_{\text{pred}} - x_{\text{gt}_i}\|\|_2$ that we call Minimum Distance to Ground Truth (MDGT), and compare it with baselines. From the results in the table, it is evident that considering multiple possible GTs, our method still achieves the best quantitative results.

Our method can generate diverse results. Therefore, we report the Frechet Inception Distance (FID), which measures the distribution discrepancy between the ground truth and generated trajectories, and Diversity (DIV), which evaluates the diversity of generated motion by calculating the variance from the extracted features from the trajectory encoder. We do not report FID and DIV for our baselines because they can only generate deterministic trajectories using simple interpolation.

Extent of Camera Movement. The extent of paths like S-shaped ones depends heavily on CineGPT’s training data. Users can control the scale of the trajectory by using textual cues describing the scale. Additionally, users can provide explicit cues to add more anchors to adjust the scale, as demonstrated in the conversation example provided in our general response.

Baseline Result Video. We have highlighted the artifacts and inaccuracies caused by the baseline output in the form of images. Due to the limitations of external links, we are unable to include new videos at this stage but will update the baseline result video in our future version.

Limitations and Failure Cases. We have discussed the limitations of our method in the appendix, including its efficiency depending on LLMs and the lack of exploration of dynamic scenes. We will incorporate the limitations pointed out by other reviewers in the updated version. The most common failure case of our method occurs when the generated trajectory bumps into or goes through objects like walls or doors. Additionally, if the text prompt input to CineGPT is too complex, especially containing rare descriptions of shapes, it may not generate the correct trajectory. However, we kindly point out that these failure cases can be corrected by chatting with the LLM agent.

References:

1. Radford et al., "Improving Language Understanding by Generative Pre-Training".
2. Radford et al., "Language Models are Unsupervised Multitask Learners".

The algorithm for trajectory composition works as follows:

1. It takes as input a list of trajectories and anchor points. If adjacent anchor points are encountered in the input, the algorithm will report this as illegal input.
2. Merge adjacent trajectories until there are no adjacent trajectories left in the list. During the merging process, calculate a Euclidean transformation (6 DoF) and apply it to the latter trajectory, ensuring its starting point coincides with the position and rotation of the endpoint of the previous trajectory.
3. For all trajectories, if a trajectory has two adjacent anchor points, find a similarity transformation (7 DoF) to make the position of its starting point and endpoint coincide with the two anchor points. If it has one adjacent anchor point, calculate a Euclidean transformation (6 DoF) to make the position and rotation of its starting or ending point coincide with this anchor point.

We will include pseudo-code for this algorithm in future versions to enhance clarity.

Thank you for your valuable feedback and insights. We appreciate your comments and would like to address your concerns in the following responses.

Baselines & Comparisons

• Baseline Interpolation Algorithm. We use cubic spline interpolation for translations and spherical linear interpolation (SLERP) for rotations, fixing camera intrinsics since baselines don’t account for them.

• Baseline Failures. Baselines fail due to their limited understanding of text prompts, missing concepts like orientation ("turn left") or shape ("S-shaped"). Simple keypoint interpolation often produces incorrect results.

• Path Prediction Network. We surveyed path prediction networks but found no related work. Specific references from the reviewer would be appreciated for comparison.

• Ordered Keypoint Prediction Network + Spline Interpolation closely aligns with the approach used by ChatCam and our selected baselines, predicting keypoints through the Anchor Determinator or 3D latent embeddings, followed by cubic spline interpolation.

• Rule-based System We built a rule-based system based on the suggestion. generating sub-trajectories from text via LLM, using set rules (1-to-1 mappings from text to trajectory) instead of ChatGPT. It predicts key locations using LERF. Quantitative results are in the table (as suggested by reviewer dvT5, reporting metrics computed against multiple GTs for better fairness). Qualitative results are in Figure C of the attached PDF. The rule-based system performs worse than our baselines, as it cannot handle complex text descriptions like “go through the tunnel.” | Method | Translation MDGT (↓) | Rotation MDGT (↓) | |-----------------|-----------------------|--------------------| | Rule-based System | 23.2 | 5.5 | | Ours | 4.7 | 2.1 |

• Weak Comparisons. We argue that “as the first method to enable human language-guided camera operation, there is no established direct baseline for comparison.” To our knowledge:

  1. No method achieves text-to-trajectory translation like CineGPT.
  2. No multimodal-LLM system allows user-guided trajectory generation and optimization through conversation.

  Therefore, we respectfully point out that stating “the comparisons were extremely weak” is somewhat unfair, as such interactions are not achievable by any baselines.

Extent of Camera Movement. To keep the pipeline simple, we do not process scale specially. The trajectory scale largely depends on CineGPT's training data when the user doesn’t provide end anchor clues. This explains pan length decisions. Without clear visual modality or start-end points, bumping into walls is possible. Users can guide ChatCam to adjust trajectory scale by providing more anchor points through dialogue, as shown in the conversation example provided in our general response.

Dataset. Our text-trajectory dataset contains trajectories and text descriptions, not attached to specific objects or scenes. Anchor determination bridges trajectories and specific scenes by placing trajectories in scenes through anchor points.

• “Why would the method generalize to out-of-domain objects/places?” CineGPT and its training data are not tied to objects or places (no "domain" concept). Scenes like the opera house are recognized by the Anchor Determinator and its underlying CLIP model, allowing our method to work on them.

• Collection Details. Data was collected by the authors. We enumerated textual descriptions of camera trajectories and built them in Blender. Movements included basic translations, rotations, focal length changes, and combinations (simultaneous or sequential). We included trajectories from professional cinematography literature. All the trajectories cover similar volumes with aligned translations and rotations.

• Blender Movies. Since our dataset is not attached to any objects or places, suggested movies cannot be directly used in the dataset construction. However, they might be a meaningful application scenario for our future study with dynamic scenes.

CLIP-based Anchor Determination. We use CLIP to select the image that best matches the text description of the anchor point, as it effectively aligns visual and textual information. CLIP can be replaced by other multi-modal models like BLIP. Our anchor refinement ensures key location accuracy. We verify CLIP-based anchor determination accuracy in the table below. We manually specified the optimal anchor points as ground truth in 10 cases and calculated the MSE between the Anchor Determination results and the ground truth.

These results show that the choice of multi-modal model has minimal impact on accuracy, with both models performing well. However, anchor refinement greatly enhances anchor determination accuracy.

LLM Prompt. We post our designed prompt in the general official comment, containing detailed instructions, response template, and examples for LLM. We encourage reviewers to test it with LLMs to better understand our approach. We will also make it public upon acceptance.

Static Trajectories. Our dataset does not contain static trajectories, and thus CineGPT fails to produce correct results when prompted with "hold still" or something close. This issue can be addressed by adding static trajectories to the dataset. We will discuss works on camera trajectories in the related works as suggested.

Trajectory Composition. We elaborate on this step in the official comment below.

Limitations. We have discussed the limitations of our method in the appendix and will add a discussion of supporting static trajectories as suggested.

• As you created the trajectories in the dataset, in similar way you could create a dictionary of camera movements and give them a name (S-shaped, U-turn). Add few parameters to control and then use them across anchor points. Seems doable to me. Optionally, apply some optimization to post-process (example: L1 norm optimization or smoothing)

• There are too many stochastic components and the reliability of the final result is doubtful

I am still not convinced and I am staying with my original rating.

Thank you for your feedback!

Your suggestion of "creating a dictionary of camera movements and giving them names (S-shaped, U-turn)" aligns with our understanding and implementation of the rule-based baseline during the rebuttal. However, as shown in the attached PDF and discussed with Reviewer Shd5, the results from this approach do not perform as well as our proposed method.

This rule-based approach relies on a predefined dictionary of camera movements, which is inherently finite, whereas our GPT-based text-to-trajectory translation model can handle theoretically infinite input prompts.

Additionally, we want to remind the reviewer that the rule-based approach cannot determine anchor points independently. The accurate determination of anchor points is a key part of our technical contribution. Moreover, the rule-based method lacks the ability to interact with users to modify or improve results, which is another significant advantage of our proposed approach.

Regarding your concerns about the reliability of our final results, we respectfully disagree with the assertion that there are "too many stochastic components" in our approach. We welcome the reviewers to point out specific stochastic elements in our method so we can engage in further discussion.

Before receiving your feedback shortly before the discussion period ends, we had already provided the LLM prompt as per your suggestion, allowing the conversational process in our method to be fully reproducible. We encourage reviewers to try it out. Additionally, any aspects of the final results that the user finds unsatisfactory can be adjusted through further interaction. Therefore, we believe that our results are reliable, especially when compared to rule-based approaches.

In my opinion, you replacing CineGPT with a rule-based algorithm in the Figure C. So the anchor points should be same for these two methods. Figure C does not show the anchor 'entrance of the green tunnel'. CineGPT generate camera trajectories only based on text description without spatial information. How can you avoid collision when calling CineGPT with 'move forward through the tunnel'? I think if the anchor point is accurate, the collision will not occur with a high probability. In the same time, this will also be easy for rule-based method too.