Active Procedure Planning with Uncertainty-awareness in Instructional Videos

Thank you for the insightful comments!

Weakness: Insufficient discussion of limitations

The current work is possibly constrained by the quality of observations and the relatively rigid formulation of the procedure planning task. While the latter is inevitable for the purpose of rigorous benchmarking, it does affect the practical application (as is also pointed out by reviewer 9LnP). Please refer to our detailed explanation below. We will discuss this together with other limitations in the revised manuscript.

Q1: Noisy & incomplete visual observation

If the visual observations (VOs) are on the start and goal states, they may lead to high uncertainty score, and hence additional observations are needed. Our method was specifically designed to handle this situation. If the VOs are on the additional (intermediate) observations, they may lead to a situation where the new observations do not enhance the prediction. There is no specific mechanism designed to deal with this situation in the current solution. We postulate that based on probability, at least a proportion of instances will have good VOs. Therefore, it is statistically probable (and validated in our experiments) that the additional VOs would improve the collective performance.

One possible solution to deal with noisy VO is to design an iterative process of visual observations. That is the algorithm allows multiple rounds of uncertainty-based sampling and decision making, until the confidence of prediction reaches a pre-set threshold. An alternative solution is to evaluate the information content of new observations, i.e., if it adds adequate new information compared to existing observations. We will put these as future work to account for the quality of VOs in the revised manuscript.

Q2: Trusted & explainable AI

A simple scenario can be conceived in a human-AI interactive environment: the AI agent may report its lack of confidence in a judgment/decision (based on high uncertainty score), and requests for more observations or even directly asks a human user to intervene. Provided that the uncertainty estimation is well-calibrated, such an ability will help the human user understand the situation and agent's decision process (explainable) and accordingly boost his/her trust to the agent.

Q3: Technical details

Details of uncertainty estimation is given in Section 4.1. The process to select visual observations is explained in 4.2. If there is any specific information needed by the reviewer, please kindly clarify and we’d be happy to elaborate further.

Thank you for the insightful feedback!

Weakness: Quality & Significance

In this study, we focus on developing a new methodology of active procedure planning that systematically incorporates uncertainty estimation as a feedback signal to enhance planning efficacy, which represents a new paradigm of the procedure planning task. To do so, we first investigate how to capture uncertainty in a principled manner and then develop mechanisms to perform active sampling based on uncertainty scores. We formulate the uncertainty metrics by combining (1) prediction confidence and (2) task sequence variation, which are explained next.

Q1: Temperature scaling

We apply standard temperature scaling (TS) to set T, which is optimized with respect to the Negative Log-Likelihood (NLL) loss on the validation set (Guo et al. 2017). Parameter T does not change the maximum of the softmax function, so that the class prediction remains unchanged. In other words, TS only affects the distribution of uncertainty values, but not the model’s accuracy. In our experiment, it affects the proportion of samples that require additional observations at specific thresholds. We presented the effect of TS in Appendix 7 (Fig.11). We show that TS facilitates more balanced and progressive increase in sample selection, and in turn steady performance change. Hence, it reduces the model’s sensitivity to the threshold, making it more robust against testing conditions. We will add reliability diagrams to further demonstrate the effect of TS-based calibration in the revised manuscript (Note: we do not show it here since image insertion is disabled in rebuttal).

Q2: Modeling task sequence variation

We used Shannon entropy to estimate task sequence variation (TSV). While it is a simple formalism, it effectively captures two key aspects of TSV, namely the number of alternative trajectories and the data distribution among them, as explained after Eq.1. The elegance of this formalism is that it models a complex phenomenon (task variation) and obtain plausible outcome, which is often a desirable feature of technical solutions. To demonstrate its effectiveness, we provided statistics in A6 (Fig.10). The distribution of uncertainty score is consistent with observations in literature, e.g., higher variation on CrossTask than on COIN dataset, and increased variations in longer planning horizons, whereas our method gives more refined quantitative metrics compared to existing works.

For the 2nd issue, i.e. (a_1, a_T) tuple may be unseen, we acknowledge that this indeed happens in a small portion of instances. In our implementation, if an observation tuple is unseen, we first compute the similarity of (a_1, a_T) with other “seen” instances based on their visual embeddings. Then we use the uncertainty score of the nearest instances as an approximation of (a_1, a_T).

Q3: Multiple observations

As stated in Sec 4.2, we restrict the number of additional observations to 1, so as to control the cost of data acquisition. Thanks to the reviewer’s constructive suggestion, we conducted experiments to evaluate the effect of multiple observations. The results of success rate on CrossTask with planning horizons T=4,5,6 are shown below. Pls refer to Tabs.1&2 in the paper for notation details. As expected, two-observations results in higher performance than 1-observation. Meanwhile, it incurs higher computational cost, thus leading to the issue of efficiency-performance tradeoff. Please refer to our explanations to reviewer 9LnP on this. We will add results of other metrics/dataset in the revised manuscript.

Q4: Baseline models

Random sampling refers to randomly selecting instances that are augmented by additional observations. In practice, given a batch of instances and an observation budget (which controls the proportion of instances to be augmented), a set of instances will be randomly selected from the batch that is augmented by additional observations.

We use random sampling as a baseline because this is a new problem formulation and there is no alternative solutions yet. Meanwhile, we included two additional benchmarking solutions based on the modelling of uncertainty, i.e. task variation score and predicted confidence score. The results are shown in Sec. 5.2.1 (Fig. 5), where the relative weights of uncertainty metrics are set as (w_1=1, w_2=0) and (w_1=0, w_2=1), respectively. We show that using only one uncertainty metrics is inferior to combining two, which is the proposed solution. More results are provided in A2 & Fig.9.

Thank you for the insightful comments!

Q1: Implementation of diffusion model and alternative models

In our implementation, we trained two diffusion models, one without additional observations (VOs)(model A) and one with VOs (model B). In the diffusion model, the tensor that holds the VOs is fixed as “0” in model A; and it is set as the embedding of the VO in model B. During inference time, when an instance is selected to include additional observations, we use model B to perform the inference; otherwise, we use model A.

So far, we implemented our method based on a diffusion model but the concept is appliable to other model structures. There are two prevalent models for weakly-supervised procedure planning. Other than PDPP (diffusion-based), the other is P3IV [1], which combines transformer and GAN. However, P3IV codes are not published and the paper lacks adequate technical details to be re-implemented. Other legacy procedure planning methods adopted supervised learning, which assumed access to all intermediate observations. This is incompatible with the assumptions of active procedure planning, making them not comparable. Therefore, they cannot be tested in the current problem context.

[1] Henghui Zhao, Isma Hadji, Nikita Dvornik, Konstantinos G. Derpanis, Richard P. Wildes, and Al- lan D. Jepson. P3IV: Probabilistic procedure planning from instructional videos with weak super- vision. In CVPR, 2928–2938, 2022

Q2: Efficiency-performance tradeoff

As mentioned earlier, we trained two models: Model A that uses only start and goal observations, and Model B that incorporates additional observations. During inference, we initially employ Model A to make predictions. If the uncertainty score of the action prediction exceeds a predefined threshold, we switch to Model B to predict actions. Based on the above design, active procedure planning involves two computational overheads as explained below.

• Training two models (A & B) rather than one (i.e. only Model A is needed in passive procedure planning), which doubles the training time. In our implementation, a model with PDPP framework can be trained in about 3 hours on CrossTask dataset and 5.5 hours on COIN based on the current hardware configuration (4 NVIDIA RTX A5000 GPUs; Please refer to Implementation Details for more information on training configurations). Training time on MDPP framework is about 8 hours on CrossTask and 12 hours on COIN. Note that the time and memory needed for training both models A and B are identical under the same conditions. Since training happens only once, it is affordable in the current problem.

• During inference phase, a subset of instances will be selected to perform additional observations (due to high uncertainty score). They will go through both Models A and B, instead of only Model A in passive procedure planning. The inference time on individual models is identical (so is the memory usage). Therefore, the computational time for those selected instances is doubled. The total inference time overhead is dependent on the proportion of instances that require additional observation, which is in turn contingent on the uncertainty threshold. For example, if at a given threshold, 30% of instances require additional observation, the inference time of active procedure planning will be 30% higher than passive planning.

We will add the above explanations in the revised manuscript.

Q3: Application scenarios

In this study, we focus on an initial investigation of methodological issues in active procedure planning and evaluate it on an established protocol and datasets, i.e. procedure planning in instructional videos. In the example where a robot is tasked to make a plan (thanks for the reviewer’s good suggestion), a robot may need to observe the current scene and come up with a plan. When the robot has low confidence in the prediction, it would be costly or even dangerous to execute it blindly. The major difference between the above robotics scenario and procedure planning in instructional videos is that for the latter case, the intermedia states are readily available, while in the former case, the robot needs to take certain actions (e.g., navigate to an alternative perspective, move an object to reduce occlusion, ask human for intervention, etc.) to actuate new observations. Although these actions may pose extra challenges beyond procedure planning on instructional videos, we believe that the basic concept of leveraging uncertainty estimation and incorporating new observation information in the decision loop is generic and re-usable in other situations.

In summary, the scope of the current paper is to study the effect of active planning in the context of instructional videos. We put it as future work to implement alternative practical scenarios (e.g., a robot makes a plan, executes it, and observes the effects), which may lead to many more impactful outcomes.