Visual Attention-Prompted Prediction and Learning

• The paper ignores a body of research on learning with privileged information (e.g., [ref1, ref2, ref3]), which is closely related to the proposed method in terms of incomplete information during inference. In addition, leveraging a two-stream CNN for distilling privileged knowledge is also not particularly new (e.g., [ref3] also encourages alignment between features from the two streams).

• According to Equation 2, the high-quality attention prompt D is never used to modulate the images in the second stream. Is there a particular reason why always using the perturbed attention prompts?

• Related to the previous comment. I have difficulty understanding the design choice of the perturbed masks. Specifically, Equation 2 discards all important/unimportant pixels (lambda=+1 or -1) and only takes into account regions with unknown importance. What do we want the model to focus on regions with higher uncertainty, instead of those more prominent ones?

• I am a bit concerned about the computational overhead of the model. For each image, it increases the inference by N+1 times (N is the number of perturbed masks). The alternative training also makes the cost complicated. I would suggest the authors provide a quantitative measure of the computational cost.

• The experiments are inadequate to demonstrate the advantages of the proposed method, mainly because of two reasons: (1) Tasks and scale of the datasets. The paper only evaluates on two small medical imaging datasets for binary classification, and it is unclear if the method can generalize toward broader domains. Moreover, for each dataset, the authors only sample a small subset of data for experiments (e.g., 100 and 30 images for training on the two datasets, respectively). With such a small scale, it is very hard to validate whether or not the method is truly advantageous. (2) Model selection. The paper only tests with ResNet-18, does the method work on other more advanced backbones? I would suggest the authors experiment with state-of-the-art networks designed for medical imaging tasks (or test on ImageNet classification). Alternatively, for a fair comparison with other visual explanation methods, it would also be reasonable to directly test on the same tasks.

• For a more comprehensive evaluation of the explainability of the method, it is reasonable to perform a quantitative evaluation of the model attention to see how they align with expert annotations.

References:

[ref1] Learning Using Privileged Information: Similarity Control and Knowledge Transfer. JMLR, 2015.

[ref2] Unifying Distillation and Privileged Information. ICLR, 2016.

[ref3] Modality Distillation with Multiple Stream Networks for Action Recognition. ECCV, 2018.

1. The proposed method appears to be overstated. In detail,

• The framework can be seen as a stacked version of two CNN layers, with one leveraging attention annotations while the other does not. In scenarios where attention annotations are not available, it seems that only one branch of the model is necessary, rendering the other redundant.

• The advantages of the proposed attention prompt modification are not clearly conveyed. On one hand, obtaining the modified attention map requires the model to propagate each sample numerous times (equal to the number of augmented images). This can be time-consuming and degrade the efficiency. On the other hand, there have been many methods (e.g., RES) focused on improving noisy attention prompts. Yet, no comparisons are given in this regard.

• The name of “attention-prompt guided multi-task” is confusing. While the authors show that the learning objective includes several components (e.g., classification loss, parameter/feature regularization loss), this should not be treated as “multi-tasks” in my view. Because both network branches still account for the same classification problem.

2. Many design choices are not self-contained. Specifically, why knowledge sharing is necessary for the two CNN networks? Can we just use the prediction from one branch as the final model decision? What are the advantages of this two-branch framework over the one-branch model?

3. The results primarily focus on medical datasets. If this work specifically targets medical analysis, the authors should explicitly state it. Otherwise, it would be crucial to provide additional results on other scenarios (as referenced by RES) to showcase the applicability of the proposed method.

1. Though the ablation experiments are extensive, it seems that there may be a very important ablation that was not performed. From my point of view, the $p$ in Eqn (2) is very important for different attention map generation. However, the authors do not take an in-depth study on this hyper-parameter. I think it is important to perform a targeted ablation about that.

2. Why the incomplete input - Proposed is better than Proposed-P in most cases in Tab. 1 ?

3. It seems that three $lambda$ are very sensitive to the final results as shown in Fig. 4. By introducing too many hyper-parameters, this method is not robust.

4. I am also curious to see what the meaning of final learned attention is, is there any change as compared with the ones trained without an attention map?

5. There are no failure cases, which I believe is very important for the reader to understand the proposed technique.

This work also has the following weaknesses:

1. The method assumes that attention prompts are available or can be generated for all images. However, the process of creating or obtaining these prompts for a vast array of images in real-world datasets could be challenging and is not addressed in detail.
2. The model’s performance seems heavily reliant on the perturbation-based attention prompt map modification. The randomness introduced here and its impact on model stability and consistency over multiple runs is not clearly addressed.
3. There is a potential concern regarding how well the model would generalize to images significantly different from those it was trained on, especially considering that the attention prompts might not be accurate or might introduce bias towards certain features.
4. The model includes several hyperparameters (e.g., the number of perturbed masks $N$, probability $p$, $\lambda_1$, $\lambda_2$, $\lambda_3$). The process of tuning these hyperparameters is not extensively discussed. Despite the experiments shown in Fig. 4, what other insights could there be regarding the hyper-parameter tuning? The performance of the model exhibits sensitivity to the variation in $\lambda_1$, $\lambda_2$, and $\lambda_3$, suggesting that careful tuning is essential for optimal performance.
5. The model's adaptability to other imaging modalities (such as MRI, PET scans, etc.) or different domains is not discussed. This adaptability is vital for the model's utility in a broader context.