We're encouraged by the reviewer's positive comments on the novelty of the problem and our method contribution. Below, we address the concerns raised in your review.

• Datasets: Thanks for the suggestion. We followed your recommendation and added additional experimental results on Fashion MNIST, CIFAR-10, and two UCI datasets (Adult and Mushroom). The results are included in Figures 2 and 3 in the one-page PDF response. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs. Please refer to Response to All and the attached PDF for details.

• References: Thanks for the great references! Indeed, learning to reject and learning to defer (adaptive learning to reject) are closely related to our work. These works consider how to select AI or the human at testing time to decide on an instance while our paper focuses on how to design the active learning strategy with human discretion behaviors. [Perini et al. 2020] studies how to estimate the class prior with a given active learning strategy in the PU learning setup. PU learning also considers a partially observed label setting similar to ours. We will discuss the relationships with these literature in detail in the revised manuscript.

We appreciate your insightful suggestions and have addressed it thoroughly in our revised manuscript to enhance the overall quality of the work. Please do let us know if you have any further concerns, or if this adequately addresses all the issues that you raised with the paper.

We are delighted to learn your acknowledgment of our Bayesian approach to a practical problem in active learning. We address your points of concern in detail below.

• W1 (Estimating $e(x)$): (i) We now add more implementation details of the Bayesian network $e_\phi(x)$ in the appendix. In the simulation, we use the MC dropout as a Bayesian Approximation [Gal & Ghahramani 2016] (we set the number of MC samples as 40 in all experiments). The model architecture is a 3-layer fully connected neural network with LeakyReLU activation function.. (ii) underfitting problem: in L171, we propose to query data points that are informative to the human discretion model in the second point, which aims to address the underfitting issue. More specifically, in Def 4.3-4.5, we include an additional term that quantifies the information gain on the human discretion model to help learn $e(x)$ better. In addition, we propose the UCB and TS variants to guide our label acquisition using the underfitted $e(x)$. As shown in Figure 3, the underfitted $e(x)$ will indeed harm methods like $e$-BALD but our proposed method can get a better performance. (iii) We agree with the reviewer that the true $e(x)$ is unnormalized, i.e. $\int e(x) dx \neq 1$. The discretion model $e_\phi(x) = p(a=1 | x, \phi)$ is a function mapping from the domain $\mathcal{X}$ of $x$ to any value in $[0,1]$, which also does not require normalization over $x$. The Bayesian computation yields the posterior $p(\phi | \mathcal{D})$ of the parameters $\phi$ instead of $e(x)$. So our $e_\phi(x)$ can be applied to approximate any function $\mathcal{X} \to [0,1]$.

• Datasets and Human Behavior Models: Thanks for the suggestion. We added additional experimental results on Fashion MNIST, CIFAR-10, and two UCI datasets (Adult and Mushroom). The results are included in Figures 2 and 3 in the one-page PDF response. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs. Though selective labeling is prevalent in industry use cases, we hope you can understand that it is challenging for us to find public real-world data with recorded human decision-maker judgments on such applications, so we choose to vary the human behavior model in the paper so that we can get a more detailed comparison and insights between different methods.

Please find the results for additional datasets and human behaviors in Response to All and the attached PDF.

• True/Predicted Human Behavior: We kindly note that Figure 3 in the main paper demonstrates the comparison of $e(x)$ and the estimated $\hat{e}_\phi(x)$ for the synthetic data. We will address the typos and notations in the revised version.

We appreciate your insightful suggestions and have addressed it thoroughly in our revised manuscript to enhance the overall quality of the work. Please do let us know if you have any further concerns, or if this adequately addresses all the issues that you raised with the paper.

Thank you very much for your positive comments on our problem, method, and writing. Nevertheless, we understand there are concerns impacting the evaluation.

To address these:

• Computational Complexity: In this paper, we mainly focus on improving the BALD method, therefore the baseline models are also Bayesian models that share the same computational complexity. In addition, we use the MC dropout as a Bayesian Approximation [Gal & Ghahramani 2016] which is not computationally demanding compared to a deterministic model with the same architecture. For other traditional AL metrics that work for simpler models such as Entropy, it is also possible to extend the proposed method for these metrics by replacing the information gain term with the uncertainty term. We demonstrate that in Figure 1 in the response PDF, we implement Joint-Entropy-UCB with entropy measure on the synthetic data. Joint-Entropy-UCB can also improve the traditional entropy objective in this case. We will add the discussion in the revised manuscript.

• Modeling Human Behaviors is Challenging: We agree that human discretion models may be hard to accurately predict, which is the main reason we proposed a UCB and TS variant to better explore the unknown human behaviors. In addition, we can use a non-parametric learner such as Gaussian Process (GP) or BNN (by universal approximation theorem) to learn the human discretion behavior that guarantees the true model can be learned with enough samples. Furthermore, recent studies and real-world applications have demonstrated strong predictive performance of human discretion behaviors [1,2] or human mental models [2,3], which indicates the human discretion model can be learned in practice, which we hope can address your concern.

• Changing Human Discretion Model: We agree that human decision-making processes can change over time. We would like to kindly highlight that we discuss and evaluate this scenario in Appendix B where the true human behavior model changes at a given time step in the instance collection. Our method can still be applied in this setting and we find the Joint-BALD-UCB successfully selects the samples that are likely to be labeled and are informative to the predictive model and human discretion model, and it can better recover the correct decision boundary than baselines.

• Real-world application and Data sets: Thanks for the suggestion. First, we kindly note that one of our experiments is motivated by a financial fraud investigation. Second, we added additional experimental results on Fashion MNIST, CIFAR-10, and two UCI datasets (Adult and Mushroom) in the rebuttal. The results are included in Figures 2 and 3 in the one-page PDF response. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs. Though selective labeling is prevalent in industry use cases, it is challenging for us to find public real-world data with recorded human decision-maker judgments, so we choose to vary the human behavior model in the paper so that we can get a more detailed comparison across different methods.

Please find the results for additional datasets and human behaviors in Response to All.

• Figures and Writings: In the revised paper, we ensure all the figure legends are clear with proper fonts (like the figures in our PDF response). We also add intuitive explanations to the objective terms in the Definitions. We add implementation details of the training discretion model in the Appendix.

• Assumptions Biases: We impose minimum assumption on the human discretion model since $e(x)$ can be any complex function. In practice, we can use a non-parametric learner such as Gaussian Process (GP) or BNN (by universal approximation theorem) to learn the human discretion behavior that guarantees the true model can be learned with enough samples.

• $c_e$ and $c_l$: In the ALIR problem, we assume there is a cost to decide whether to label an instance (examine cost $c_e$) and a cost to actually label the instance (label cost $c_l$) (L130-131 in the main paper). The total cost is the sum of the current examination cost and label cost. We will make this more clear in the revised manuscript.

We appreciate your insightful suggestions and have addressed it thoroughly in our revised manuscript to enhance the overall quality of the work. Please do let us know if you have any further concerns, or if this adequately addresses all the issues that you raised with the paper.

[1] Sun, Jiankun, et al. "Predicting human discretion to adjust algorithmic prescription: A large-scale field experiment in warehouse operations." Management Science 68.2 (2022): 846-865.

[2] Wang, Xinru, Zhuoran Lu, and Ming Yin. "Will you accept the ai recommendation? predicting human behavior in ai-assisted decision making." Proceedings of the ACM web conference 2022.

[3] Bansal, Gagan, et al. "Beyond accuracy: The role of mental models in human-AI team performance." Proceedings of the AAAI conference on human computation and crowdsourcing. Vol. 7. 2019.

[4] Madras, David, Toni Pitassi, and Richard Zemel. "Predict responsibly: improving fairness and accuracy by learning to defer." Advances in neural information processing systems 31 (2018).

We appreciate your recognition of the novelty of the problem and the clarity of our approach.

Other AL Metrics: Thanks for the suggestion! While we mainly focus on the BALD objective in this paper, our methods can be generalized by replacing the mutual information with other metrics in the objective. For example, we can replace the information gain with other uncertainty measures such as entropy.

As shown in Figure 1 in the response PDF, we implement Joint-Entropy-UCB with entropy measure on the synthetic data. Joint-Entropy-UCB can also improve the traditional entropy objective in this case. The BADGE method measures uncertainty at the gradient space. One potential way to apply BADGE is to compute the gradients for the last layer of the predictive model $p(y|x,\theta)$ and discretion model $p(a | x, \phi)$, weighted combine the gradients with the weights $(e(x), 1)$, and then use K-means++, however, this adaptation may be more challenging compared to traditional AL metrics and we believe will be interesting future work. We will discuss the connection with BADGE in the revised version.

Datasets and Human Behavior: Thanks for the suggestion. We added additional experimental results on Fashion MNIST, CIFAR-10, and two UCI datasets (Adult and Mushroom). The results are included in Figures 2 and 3 in the PDF response. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs.

Following your suggestion, we also implemented a Logistic Regression based human behavior model in Figure 5 in the PDF response with similar qualitative conclusions as in the main paper.

Please find the results for additional datasets, human behaviors, and more details in Response to All.

F1 Score: In our experiments, we find F1 score often has a similar qualitative conclusion as the Accuracy. To illustrate that, we include a comparison between Accuracy and F1 score in Figure 4 in the PDF response for your reference. We will add the discussion about the F1 score in the paper.

Total Cost: In the ALIR problem, we assume there is a cost to decide whether to label an instance (examine cost) and a cost to actually label the instance (label cost) (L130-131 in the main paper). The total cost is the sum of the current examination cost and label cost (the cost structure is defined in L212-213 in the paper). We will make this more clear in the revised manuscript.

We appreciate your insightful suggestions and have addressed it thoroughly in our revised manuscript to enhance the overall quality of the work. Please do let us know if you have any further concerns, or if this adequately addresses all the issues that you raised with the paper.