SAGE: A Unified Framework for Generalizable Object State Recognition with State-Action Graph Embedding

Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.

---

Q1: Viterbi Decoding

For each frame, the model predicts the probability of being in one of three states: initial, transitional, or end. To produce the most probable and temporally consistent sequence of states, we apply Viterbi decoding with a monotonic constraint: the sequence must follow the order initial → transitional → end. The pseudocode for the constrained Viterbi decoding is provided below. We will clarify Viterbi Decoding in the revised version and provide the full algorithm in Appendix.

Input:

• T: total number of frames
• S = {0: initial, 1: transitional, 2: end}
• probs[t][s]: probability of frame t being in state s

Initialization:

• dp[0][0] ← log(probs[0][0])
• dp[0][1] ← -∞, dp[0][2] ← -∞

Recursion:
For t = 1 to T - 1:
 For s in S:
  Let prev(s) be allowed previous states:
   prev(0) = {0}
   prev(1) = {0, 1}
   prev(2) = {1, 2}
  dp[t][s] ← max_{p in prev(s)} [dp[t-1][p] + log(probs[t][s])]
  back[t][s] ← argmax_{p in prev(s)} [dp[t-1][p] + log(probs[t][s])]

Termination:

• y_T ← argmax_s dp[T-1][s]

Backtrace:

• For t = T-2 to 0:
    y_t ← back[t+1][y_{t+1}]

Output:

• Sequence y = [y_0, ..., y_{T-1}] with enforced order: initial → transitional → end

---

Q2: LLM Reliability

To verify the reliability of LLM, we manually annotate the states and concepts generated by LLMs. We randomly sample 100 states and 100 concepts generated by the LLM and manually annotate the precision of these states and concepts. The precision of states is 100% and the precision of concepts is 92%, which shows that the LLM can generate highly accurate descriptions of object states and concepts.

Q3: LLM Sensitivity

To verify that our method is not sensitive to the choice of LLM, we conduct the experiments with another LLM, Qwen3-32B. We compare the performance of SAGE with GPT4 and Qwen3 under the settings of Table 1 and Table 2 on HowToChange dataset. SAGE demonstrates similar performance when combining with different LLMs, showing that our method is not sensitive to the choice of LLM.

Table 1: Object state recognition

Table 2(a): Object state recognition for novel object

Table 2(b): Object state recognition for novel object&action

Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.

---

Q1: The integration of multiple models could make the system challenging to implement and maintain

Integrating LLMs and VLMs is a common practice in multimodal learning, as it provides an easy approach to utilize the web-scale prior knowledge encoded by these models. Notably, all baseline approaches we compare against used VLMs to generate pseudo labels. In SAGE, the LLM is only responsible for proposing relevant visual concepts for each object state. The design is highly modular and not tied to other components in the overall framework, hence easy to implement. We invite the reviewer to check our actual implementation, which was included in the supplementary materials.

Our system is also easy to maintain. Table 2 shows that our approach naturally generalizes to new actions and objects, without the need for re-training.

---

Q2: Proper alignment between the models are needed.

Alignment is not needed due to our modular design. As discussed in Section 3.2, the LLM is responsible for proposing an over-complete concept list, based on its “commonsense” knowledge. The concepts are then refined by a VLM based on how visually discriminative each concept is. As discussed in our response to reviewer LFZv, SAGE’s performance is robust with respect to the choice of the LLM. As shown in Table 1 and 6, SAGE achieves competitive performances when different VLMs are used, where as expected, better VLM leads to higher performance.

---

Q3: "Performance dependent on quality and diversity of the data … high-quality labeled data might not always be available"

As pointed out by reviewer LFZv themselves, “SAGE can learn from unlabeled instructional videos”, which allow us to utilize diverse and large-scale training data, without relying on labels.

---

Q4: LLMs and VLMs require substantial computational resources for both training and inference

The LLM is only used for graph construction at training time. The VLM is used for graph construction at training time, and visual feature extraction at training and test time. The use of the visual encoder in a VLM is a standard practice adopted by all baseline approaches.

Once the visual representations are extracted, only a lightweight temporal Transformer is needed during inference time.

---

Q5: "Overfitting on certain specific objects or actions present in the training set … If the system is exposed to a narrow range of actions or objects during training, it might fail to generalize effectively"

As discussed earlier, our method does not rely on any labeled videos, hence can naturally utilize large-scale training data that covers a wide range of actions and objects. As shown in Table 2, our approach achieves the best performance on generalization towards novel actions and objects.

Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.

---

Q1: Clarity on evaluation setups

At test time, we sample video frames at 1 FPS and feed them to the model. While the object and action ground truth labels are available in the test set, they are optionally provided to the model according to the evaluation setting. When the object and action labels are not provided, our model predicts them according to our method in Section 3.3. In Table 1, checkmarks in the Privileged Info column indicate whether object and action labels are provided. For Table 2, 4, 5, and 6, we follow the standard protocol from prior work (e.g. VidOSC), where action information is provided but the object label is not. Given the information above,the model predicts one of the following labels for each sampled video frame: initial, transitional, end, or background.

To evaluate model performance, the test sets have frame-level annotations. Each label is one of initial, transitional, end, or background. Following the implementation detail of evaluation metrics in Appendix A, we compute State Precision@1 and Transition Precision@1 on the ChangIt dataset and Precision, F-1 score, and Precision@1 on the HowToChange dataset. Note that we use different metrics on two datasets because previous works did so and we want to compare with them. This is not because two datasets have different forms of annotations.

We will include these clarifications in the revised version.

---

Q2: Why is temporal segmentation not evaluated?

We follow the standard evaluation protocols as proposed by each of the original datasets. The original evaluation protocol was introduced by Alayrac et al. in ICCV 2017, where the authors described that “a temporal action localization is said to be correct if it falls within the ground truth time interval”. In practice, the metric is more robust in the presence of background frames that frequently occur in the videos.

---

Q3: Pseudo ground truth generation method as a baseline?

We would like to clarify that all approaches, including SAGE and the baselines, used pseudo ground truth generation as part of their pipelines. We note that there are two approaches where pseudo labels are used, as shown in Table 1: CLIP, VideoCLIP, and InternVideo are VLMs that directly predict pseudo labels on the test videos. This setup is usually referred to as “zero-shot” classification. SAGE, LFC, MTC, and VidOSC all use VLMs to generate pseudo labels on the training videos. These videos are then used to train machine learning models that make predictions on the test videos.

To answer the reviewer’s particular question: If the goal is to understand how SAGE outperforms VLMs that generate pseudo labels on the test videos, then CLIP/VideoCLIP/InternVideo in Table 1 are the proper baselines.

If the goal is to understand how SAGE outperforms models trained on videos with pseudo labels. Table 1/2/3 show the relevant comparisons, where we carefully control the pseudo label generator to be the same (CLIP ViT-L-14). Table 6 shows the comparison where the baselines can choose their own pseudo label generators (e.g. VideoCLIP, which was trained on HowTo100M).

Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.

---

Q1: Scalability for extremely large action sets

We would like to clarify that exhaustive matching between concepts and states is not needed during model inference. As illustrated in Figure 2, we first compute the embedding similarity to identify the video-level action label (unless the video-level action label is given as input) and then the object state recognition is restricted to the three object states and their corresponding concepts related to this action. We observe that recognizing video-level actions does not impose computational overhead since it only needs to be performed once per video (and the action space is moderate, since it is not combinatorial with states and concepts).

---

Q2: Multi-action evaluation

SAGE can technically be generalized to support multi-action videos by replacing the video-level action classification step with a temporal action localization / segmentation module. We agree that evaluations under multi-action settings would further demonstrate the effectiveness of our approach. Unfortunately, to the best of our knowledge there is no publicly available benchmark for recognizing object states from videos with multiple actions and objects. The most relevant existing dataset is Multiple Object States Transition (MOST) by Tateno et al., recently published in WACV 2025. This benchmark still assumes a single object is being manipulated within a video, but expands the list of object states to include those not strictly tied to actions (for example, an egg can be both “cracked” and “raw” after being cracked. In SAGE, we treat “raw” as a possible visual concept).