Response to reviewer Aam6

We sincerely thank the reviewer for the support of our work. Below we address the reviewer’s concerns and questions:

1. Question:
   How exactly are the uncertainty or controversial guidance signals integrated into the diffusion process?
   Response:
   The uncertainty and controversial guidance signals are integrated into the diffusion process using a classifier guidance method. More details can be found in Appendix A.2.

2. Question:
   Are you using classifier-free guidance diffusion models, and if so, how are the dual-label conditions handled?
   Response:
   We adopted classifier guidance methods for all of our experiments. Although classifier-free guidance diffusion models may offer interesting insights, they were not used in our work. In our approach, all guidance is introduced by the classifiers, and the dual-label conditions are handled by our proposed controversial guidance and uncertainty guidance.

3. Question:
   Which classifiers (architectures and specifics) were tested for generating controversial stimuli?
   Response:
   For generating controversial stimuli using the base models (trained from scratch on the MNIST dataset), all five models are paired with each other. The guidance outcome results are presented in Figure A.14.

4. Question:
   Could you clarify precisely how fine-tuning is conducted (GroupNet, IndivNet, BaseNet), and what are the exact targets for population-level fine-tuning?
   Response:
   We apologize for any confusion. For detailed information on the fine-tuning process, please refer to point 6 of our rebuttal to Reviewer 62LP. The goal of the population-level fine-tuning is to align the models with the population-level characteristics. Starting from GroupNet for the fine-tuning of IndivNet provides the model with a human group-level prior and helps prevent overfitting, particularly since the varMNIST-i dataset contains only a small amount of data.

5. Question:
   What prevents applying the full analysis presented for MNIST to more complex datasets like ImageNet?
   Response:
   Please refer to point 1 of our rebuttal to Reviewer YcpF.

6. Question:
   Can you discuss how the generative model’s biases impact the validity of perceptual variability experiments (versus GAN-based or other types of generative models)?
   Response: We chose classifier-guided diffusion models for their superior performance, stability, and flexibility. While we explored alternatives (e.g., VAEs and prior-free guidance[1], as shown in Figure A.3), none achieved the quality or precise individual alignment that our approach provides. We acknowledge that though a comprehensive comparison of generative model biases would indeed provide more insights, it would require developing new methodologies based on various backbone models, which falls outside the scope of this work.

7. Question:
   The authors assert that their diffusion-based method produces more "natural" controversial stimuli compared to existing approaches. This claim, however, is insufficiently supported, especially given the comparison is largely limited to MNIST rather than photorealistic domains.
   Response:
   We apologize for any confusion in our writing. By "natural," we mean stimuli that are closer to the original distribution of the dataset. For example, in Figure A.3, stimuli generated by methods without a prior would not be recognized as digits by human participants, thereby classifying them as out-of-distribution or "unnatural" in our terms. We deliberately chose this terminology to distinguish our approach from others, such as the one proposed by Golan [1]. Although Golan’s method can generate an image x that causes classifier f1 to label it as y1 and classifier f2 as y2, it fails to produce images that human participants recognize as digits—a shortcoming that our method effectively overcomes.

References:

1. T. Golan, P. C. Raju, and N. Kriegeskorte, "Controversial stimuli: Pitting neural networks against each other as models of human cognition," Proceedings of the National Academy of Sciences, vol. 117, no. 47, pp. 29330–29337, 2020.

Response to reviewer YcpF

We sincerely thank the reviewer for the support of our work. Below we address the reviewer’s concerns and questions:

1. Comment:
   The MNIST dataset is not suitable for examining human perceptual variability, as evident from Figure 4b, where participants' judgments appear to be highly consistent. This suggests that MNIST lacks the complexity necessary to capture meaningful individual differences in perception.
   Response:
   We agree with the reviewer that in the MNIST dataset participants' judgments appear to be highly consistent and are less complex than natural images. We conduct the major experiments using handwritten digits for two main reasons:

   • Efficiency: The models are much easier to train and require less time, which is very important for a large-scale behavioral experiment like ours.
   • Perceptual Variability: As mentioned by Reviewer 62LP and Reviewer YcpF, perceptual variability is harder to evoke on handwritten digits. If we can evoke perceptual variability even on a highly consistent dataset such as MNIST, then it could be applied to more complex datasets where human perceptual variability is more diverse. Based on this assumption, we evaluated our framework on natural images and observed that perceptual variability is indeed larger on natural images than on handwritten digits. Although further experiments on natural images were not conducted, it is reasonable to assume that individual perceptual boundaries are more complex on natural image datasets. This complexity might require additional rounds of experiments or a parameter-efficient method for finetuning(e.g., LoRA) as the models would become bigger and more complex. We plan to conduct more experiments and analysis on natural images in future work. We hope the reviewer can agree with our considerations.

2. Comment:
   In addition to reporting the results, the authors should provide further analysis to explain why fine-tuning performance varies across different model architectures. A deeper investigation into the underlying factors driving these differences would strengthen the study's findings.
   Response:
   We appreciate the reviewer's suggestion, as further analysis of the differences in fine-tuning performance across various model architectures could indeed provide valuable insights. However, the primary focus of this paper is to demonstrate the feasibility of aligning humans and AI using our method and to explore the potential for manipulating perceptual variability within our framework. Given the limited space available, we believe that an in-depth analysis of architectural differences would be better suited for a separate, more focused study.

3. Comment:
   The use of technical terms in the paper lacks consistency and deviates from standard terminology in cognitive science. For example, a numerical category classification task is not typically referred to as "decision making." Additionally, the paper alternates between terms like "decision boundaries of classifiers" and "ANN perception boundary," suggesting that the authors may conflate perception and decision-making processes. The terminology should be clarified and used consistently to avoid confusion.
   Response:
   Thank you for pointing out this inconsistency. We will pay more attention to the use of terms and revise the text to ensure consistency with standard cognitive science terminology.

4. Comment:
   In Figure 4, the colors in the legend do not match those in the bar graph. The authors should ensure consistency between the legend and the figure for clarity.
   Response:
   We thank the reviewer for highlighting this issue. We will revise the figure to ensure that the legend colors match those used in the bar graph.

5. Question:
   Are there specific reasons showing the large error bars in the customized dataset for guidance success rates in Figure 6c?
   Response:
   We admit that the customized manipulation experiment can be difficult to conduct, and the effect of manipulation is largely influenced by the individual status of the participants due to the need for precise alignment with each participant. Under these circumstances, large error bars can occur. Despite the large error bars and relatively low improvement, our statistical analysis confirmed that the results are statistically significant (p < 0.001).

Response to reviewer 62LP

We sincerely thank the reviewer for the positive feedback. Below we address the reviewer’s concerns and questions:

1. Comment: The success rate in selective manipulation of human behavior is relatively low, showing limited success in using the proposed approach. This is especially true for handwritten digits with ~20% success rate. Surprisingly, while the initial experiment using natural images seem to be much more successful, none of the following experiments were conducted using natural stimuli.
   Response: Please refer to point 1 of the rebuttal to Reviewer YcpF.

2. Comment: Section 5.2: The IndivNet only improves the behavioral prediction accuracy by 5% over GroupNet, yet it is claimed that this approach captures individual differences in perceptual judgments. The result seems to suggest that the model training mostly captures the group effect in this behavior.
   Response: We believe the relatively small improvement is due to the similarity between the individual perceptual boundary and the group-level perceptual boundary. We hope the reviewer agree with us that it is reasonable to assume that individuals, in general, are close to the group, resulting in a relatively small improvement.

3. Response to typos:
   We thank the reviewer for pointing out these mistakes. We will reorganize the appendix and revise the text in a future version.

4. Question: Were subjects given extended time or was there a time limit in labeling the images?
   Response: In the major experiments, all subjects were given extended time to reduce random effects that could impact the experiment, such as arbitrary gaze positions or unexpected distractions.

5. Question: (Line 232) The varMNIST-i dataset is not defined. Is that a subset of samples labeled by one individual? If so, was one model trained per individual? The dataset and procedure should be better explained.
   Response: The varMNIST dataset consists of images with multiple labels (labeled by different participants or trials, as described in the paper). Each participant performed around 500 trials on different images. The dataset corresponding to each participant is referred to as varMNIST-i, and one model is trained per individual. We appreciate your valuable suggestions and will clarify the dataset and procedure in future revisions to the main text.

6. Comment: (Line 236) The description of model training is confusing. The first part suggests that the network was jointly trained on two datasets, but the latter part and the term "finetuning" suggest first training on MNIST and then on varMNIST.
   Response: The training process is conducted as follows:

   • Base Model: First, we train a model using only the MNIST dataset.
   • Group Model: Next, we finetune the base model to align it with the human group level. The finetuning dataset is constructed by mixing MNIST data into our varMNIST data, a common technique to prevent overfitting and model forgetting. This process yields the group model.
   • Individual Model: Finally, based on the group model, we perform another finetuning at the individual level to align the model with each individual. This finetuning dataset is created by mixing MNIST, varMNIST, and varMNIST-i together. The training and evaluation sets are carefully divided so that even if there is an overlap between varMNIST and varMNIST-i, nothing in the evaluation set appears in the training set. Parameter details can be found in Appendix C.2.

7. Question: Were there shared images across different individuals? How were the labels for repeated images from different individuals combined?
   Response: Yes, there are shared images across different individuals. Each image-label pair is treated as a trial and used in the finetuning process.

8. Question: It is unclear whether the networks used in the study (VGG, ViT, CORnet) were trained from scratch or if a pretrained network was used.
   Response: The base models are trained from scratch. The group models are finetuned from the base models, and the individual models are finetuned from the group models, as explained in point 6.

9. Comment: There are no references to the models used in the study.
   Response: We sincerely appreciate your suggestion and will add references for these models in future revisions.

10. Comment: Figure 6: It is unclear what the "individually customized dataset" is.
    Response: The individually customized dataset is generated under the guidance of finetuned individual models to better evoke variability and bias participants toward certain directions.

11. Ethics
    Response: We mentioned the ethics approval in Appendix B.2.2. Regarding human data collection, we can provide ethics documents and participant agreements upon request. We will incorporate these details into the main text in future revisions.