Customized Procedure Planning in Instructional Videos

Response to point 1:

Thank you for your feedback. We appreciate your suggestions and will include more example outputs in the camera-ready version to better demonstrate the effectiveness of the generated plans.

Regarding your point on customization keywords, we believe one of the strengths of our work lies in how we collect these keywords. Rather than predefining keywords or constraints, we directly extract customization keywords from a large-scale set of real-life YouTube instructional videos. Some videos are more generic, while others are highly specific and customized, reflecting the diversity of real-world content. This approach avoids introducing customization bias, as it reflects the natural variability in how users express their needs, rather than simulating customized keywords.

By processing and extracting keywords directly from the dynamic content of real-world videos, our model generates more authentic and user-aligned plans. This method is less imposed and more reflective of actual user behavior, where customization evolves from the content itself, ensuring that the generated plans are truly tailored to the task at hand.

Response to first comment

Thank you for your feedback. We respectfully disagree and believe the manuscript adequately addresses the gravity and scientific significance of the problem.

1. Introduction (Lines 046–079):
   We highlight the semantic gap in generating detailed and customized plans—a key limitation in prior literature—underscoring the need for user-specific, tailored plans rather than generic task objectives.

2. Figure 1 Illustration:
   Figure 1 contrasts our approach with prior methods, which generate generic action labels. Our framework produces detailed, customized plans that incorporate user-specific needs and task conditions.

3. Lines 086–093:
   We explain the need for models to integrate both the current visual state and user-specific textual requirements, enabling tailored, user-aligned plans. Figure 1b further emphasizes the practical value of this approach in comparison to traditional models.

4. Lines 094–099:
   We discuss how closed-vocabulary settings restrict predictions to predefined labels, leaving the challenge of generating detailed, user-specific plans unresolved.

5. Lines 099–107:
   We address the significant gap in datasets for training customized procedure planners. Existing datasets, such as CrossTask and COIN, lack detailed, time-stamped plans tailored to specific videos and user requirements. This limitation is further explored in Section 3.3.

6. Lines 123–131:
   We introduce a novel evaluation framework, addressing the inadequacy of traditional metrics for open-vocabulary, customized plans. We propose LLM-based metrics and validate our model’s performance on CrossTask and COIN datasets, establishing a strong baseline that outperforms state-of-the-art approaches.

These points collectively advocate the significance of the problem and highlight our contributions in addressing these challenges. However, we appreciate your feedback and would welcome specific suggestions on areas you feel were overlooked so we can further improve the manuscript.

Thank you for your thoughtful feedbacks. We provide the following responses to points made:

1

We agree that automatic metrics, including those based on LLMs, inherently carry some biases. However, we have taken several steps to minimize these biases and maximize accuracy:

1. Coupling with Objective Metrics:
   Our LLM-based metrics (aSR, aAcc, aIoU, and aRelevance) are designed to complement each other and objective metrics like aBERT-Score. This balanced framework captures nuanced aspects of customization through LLMs while maintaining interpretability via objective, quantitative measures.

2. Systematic Design with Rubrics and Examples:
   To standardize LLM evaluations, we provide a detailed rubric and use few-shot in-context learning examples (4 examples per rubric rule) to anchor the model's outputs. This ensures that evaluations follow a consistent and systematic framework, reducing variability and improving reliability.

3. Transparent Evaluation Process:
   To enhance transparency, for each evaluation instance, the LLM is asked to provide reasoning for order mapping, alignment to generic plans, and relevance before scoring based on the rubric. This systematic reasoning ensures logical consistency and provides traceability for the model’s ratings.

4. Human Validation and Performance Consistency:
   Validation tests demonstrated high inter-rater agreement (>85%), confirming the reliability and consistency of our evaluation approach. By combining LLM-based evaluations with human validation and objective metrics, we minimize potential biases while maximizing the robustness of our framework.

We believe this combined approach provides a reliable and interpretable evaluation methodology while addressing the inherent challenges of automatic metrics.

2

Thank you for your insightful observation. We agree and believe that this distinction is indeed a strength of our framework. As correctly identified, defining the goal using an Objective along with specific Conditions enables a more nuanced and customizable approach compared to traditional goal-oriented methods.

We will revise the introduction to bring greater attention to this key contribution.

3 & 4

We agree that potential bias is an inherent Challenge of using models like GPT-4o and GEMINI for generating pseudo-labels and weak-supervision frameworks. However, we have implemented several measures to minimize these biases and ensure the reliability of our pseudo-labels:

1. Limited Reliance on Models: While GPT-4o and GEMINI generate customization details, critical elements like plan actions and timestamps are manually annotated, reducing model bias.

2. Rigorous Quality Assessment: We manually evaluated the pseudo-labels on subsets of the COIN and CrossTask datasets (Section A), with high scores (e.g., 4.18/5 for customization on CrossTask), confirming label reliability.

3. Independent Models: GEMINI extracts customization keywords, and GPT-4o evaluates them, ensuring independence. Both models, trained on diverse datasets, reduce the risk of overlapping biases. We also use objective metrics like aBERT-Score (Table C) for unbiased model evaluation.

4. Error Correction During Training: GPT-4o is incorporated into the objective function to iteratively refine predictions, reducing the impact of pseudo-label inaccuracies.

5. Acknowledgment of Limitations: We recognize that biases are an inherent challenge in weak supervision. While pseudo-labels enhance model robustness, we will include a more detailed discussion of these limitations in the paper for transparency.

Response to Reviewer Question:

1. Mismatch Between User Directions and Images:
   In our dataset, there is minimal mismatch between user-specified conditions and images, as both were sourced from aligned YouTube instructional videos. However, pseudo-labels generated by GPT-4o and GEMINI may still be biased toward task objectives, especially when the model prioritizes common patterns or assumptions from the training data.

2. Task-Irrelevant Keyword Leakage:
   Despite careful dataset selection, pseudo-labels can be influenced by irrelevant information, such as frequently encountered task-related keywords or sequences. This bias may cause the model to over-prioritize certain objectives or generate labels based on less relevant data. Section C presents failure case studies that illustrate how these biases affect the model, particularly in edge cases or complex user conditions.

1

Thank you for raising the question of generalizability. CPP’s generalization can be evaluated across three dimensions:

Open-Vocabulary in Natural Language: CPP operates in an open-vocabulary setting, enabling it to handle diverse natural language inputs and outputs, adapting flexibly to user-defined requirements (Tables [1, 2]).

Within-Domain Task Generalization: Within defined domains, we evaluated CPP’s ability to perform on unseen tasks not encountered during training. As shown in the table below, CPP significantly outperforms baseline models on 8 unseen tasks from various domains within the COIN dataset (comprising 180 tasks), demonstrating its strong generalization capability to new tasks within familiar domains.

Cross-Domain Generalization: Generalizing across different domains (e.g., DIY tasks vs. computer graphics) is challenging due to structural data differences. While not the focus of this study, our experiments suggest that domain-specific adaptation, such as fine-tuning or few-shot learning, may be needed for optimal performance. Addressing these challenges is an exciting direction for future research.

2

We agree that using GPT-4o and GEMINI to generate pseudo-labels may introduce biases. However, we have implemented several measures to minimize these biases and ensure label reliability:

1. Limited Reliance on Models:
   While GPT-4o and GEMINI generate customization details, critical elements like plan actions and timestamps are manually annotated, reducing model bias.

2. Rigorous Quality Assessment:
   We manually evaluated the pseudo-labels on subsets of the COIN and CrossTask datasets (Section A), with high scores (e.g., 4.18/5 for customization on CrossTask), confirming label reliability.

3. Independent Models:
   GEMINI extracts customization keywords, and GPT-4o evaluates them, ensuring independence. Both models, trained on diverse datasets, reduce the risk of overlapping biases. We also use objective metrics like aBERT-Score (Table C) for unbiased model evaluation.

4. Error Correction During Training:
   GPT-4o is incorporated into the objective function to iteratively refine predictions, reducing the impact of pseudo-label inaccuracies.

5. Acknowledgment of Limitations:
   We recognize that biases are an inherent challenge in weak supervision. While pseudo-labels enhance model robustness, we will include a more detailed discussion of these limitations in the paper for transparency.

3

Thank you for your feedback. We view the subjectivity of the relevance score as a strength, as customization inherently requires human judgment.

1. Subjectivity by Design:
   Customization quality depends on how well a plan aligns with user-defined keywords, making subjectivity essential to capture nuanced aspects that automated metrics may miss. We formalized this with a structured rubric to ensure consistency while maintaining interpretive flexibility.

2. Balanced Evaluation:
   To complement the relevance score, we use objective metrics like SR, a-mAcc, and aBERT-Score, which measure alignment, order, and similarity. This dual framework ensures a comprehensive evaluation, balancing human satisfaction with structural correctness.

3. Scoring and Validation:
   The rubric includes clear guidelines and examples for each score level (1–5) and is reinforced with few-shot in-context learning (4 examples per rubric rule). Validation tests showed >85% inter-rater agreement, confirming its reliability.

4. Dual Metrics Rationale:
   Combining subjective and objective metrics allows a holistic evaluation of customization quality—capturing both human satisfaction and the model’s alignment and accuracy.

4

Thank you for your suggestion regarding deeper qualitative analysis of human feedback. Our current human evaluation focuses on validating the model's performance by assessing alignment, plan effectiveness, and customization quality using structured rubrics. While this approach demonstrates the model's effectiveness, we agree that a more detailed human evaluation would be valuable for uncovering improvement areas and is an interesting direction for future work.