Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting

We thank reviewer Fg3Y for your thorough reviews and valuable comments. Hopefully the following responses can address your concerns.

W1: Classification Results Comparison in Tab. 1

A1: In response to your concern on Table 1, we have re-implemented the benchmarking results on CIFAR-10, and the Deep Ensemble's performance on CIFAR-10 has also been re-evaluated. These results are updated to the revised paper.

The following table shows the updated image classification results on CIFAR-10. All accuracies are given in percentage with ResNet-18/ResNet-34 as backbone. Best score for each metric in bold, second-best italicized. We re-implement all baseline methods using default hyperparameter.

• The observation is similar to the original Tab. 1 in the paper, where Split-Ensemble beats single model significantly without additional computation cost. Compared to previous ensemble methods, Split-Ensemble also achieves similar or even better performance with significantly less cost.

To understand the impact of structure searching methods, we report the results of applying the same sensitivity-based pruning strategy to prune a naive ensemble model towards single model computation cost. We use the SC-OOD benchmark as suggested by Reviewer 37MR

Image classification and OOD detection results on the SC-OOD benchmarks. Models with ResNet18 as the backbone are trained and tested on the CIFAR-10 dataset. Best score for each metric in bold, second-best italicized.

• Note that without the subtask-splitting training objective and parameter sharing enabled by splitting, pruning the naive ensemble to a single model cost significantly impacts its performance. Whereas Split-Ensemble achieves a similar performance to the naive ensemble with significantly less cost.

W2: Performance on ImageNet

A2: Ensemble-based methods are not commonly tested on the ImageNet dataset due to their significant computation cost. As Split-Ensemble achieves single-model computation cost, here we perform an evaluation on ImageNet as suggested. The following table compares image classification results on the large-scale ImageNet1K dataset with ResNet18 as backbone. Best score in bold.

• Our Split-Ensemble model outperforms the single model and the ensemble model by 1.9% and 1.5% respectively, demonstrating the effectiveness of our method on the large-scale dataset. Details on ImageNet experiment settings are available in the appendix of the revised paper.

We thank reviewer FimD for your thorough reviews and valuable comments. Hopefully the following responses can address your concerns.

W1: Optimal subtask number and category grouping

A1: This work focuses on the training objective of performing the subtask-splitting training for OOD awareness and achieving efficient architecture with iterative splitting and pruning. While for subtask number and category grouping we use a heuristic design in the paper. We design the subtask splitting based on the intuition that semantically close classes shall be grouped together (Sec. 3.1) so as to make the ID and OOD data in each subtask more distinguishable, and the results in Table 5 show this choice truly makes a difference. For selecting an optimal number of task splits, the tradeoff is that a larger number of splits enables each submodel to learn its OOD-aware objective more easily with fewer ID classes, therefore leading to better AUROC;. Yet the performance may suffer from aggressive pruning to fit the additional branches within the budget. This is observed in Tab. 6 in Appendix C. We choose the optimal subtask number for experiments through this ablation trial. We leave a more rigorous analysis of the optimal subtask grouping strategy as future work.

W2: OOD-awareness

A2: We take the intuition of OOD-awareness from previous work Outlier-Exposure (OE), where the model will better detect the OOD data if it is exposed to OOD samples during training, thus becoming OOD-aware. As OE and its variants achieve the goal of OOD-awareness with external OOD dataset, our Split-Ensemble utilizes a novel subtask-splitting training objective to make each submodel OOD-aware with only ID data.

W3: Experiment on complex dataset

A3: Following your suggestion, we further compare our approach with baselines on ImageNet1k. Ensemble-based methods are not commonly tested on the ImageNet dataset due to their significant computation cost. As Split-Ensemble achieves single-model computation cost, here we perform an evaluation on ImageNet as suggested. The following table compares image classification results on ImageNet1k with ResNet18 as backbone.

• Split-Ensemble outperforms single model and the $4\times$ larger ensemble model by 1.9% and 1.5% respectively. Details on ImageNet experiment settings are available in the appendix of the revised paper.

Besides ImageNet, we follow the suggestion of Reviewer 37MR to include the new SC-OOD [C1] benchmark and CIFAR-10LT dataset [C2] in our experiments. along with some additional baselines like ODIN [C3], EBO [C4], OE [C5], MCD [C6], UDG [C1] are listed below and have been included in the Appendix of the revised manuscript.

The following two tables show the comparison between previous state-of-the-art methods and ours on the SC-OOD CIFAR10 benchmarks. The results are reported for models with ResNet-18 backbone. Some baseline methods use the Tiny-Imagenet dataset as additional OOD training data. Best score for each metric in bold, second-best italicized.

• The Split-Ensemble model outperforms single-model approaches in OOD detection without incurring additional computational costs or requiring extra training data. Its consistent high performance across key metrics highlights its robustness and efficiency, underscoring its practical utility in OOD tasks.

• The Split-Ensemble model consistently outshines other ensemble-based methods in both image classification and OOD detection, achieving similar or even better performance with $4\times$ less computation cost.

We thank reviewer ZwnE for your thorough reviews and valuable comments. Hopefully the following responses can address your concerns.

W1: Benchmarking Results and Comparison

A1: In response to your concern on Table 1, we have re-implemented the benchmarking results on CIFAR-10 and TinyIMNET, and the Deep Ensemble's performance on CIFAR-10 has also been re-evaluated. These results are updated in the revised paper.

The following table shows the updated image classification results on CIFAR-10/100 and Tiny-ImageNet. All accuracies are given in percentage with ResNet-18/ResNet-34 as backbone. Best score for each metric in bold, second-best italicized. ∗ refers to our own implementation using default hyperparameter.

• The observation is similar to the original Tab. 1 in the paper, where Split-Ensemble beats single model significantly without additional computation cost. Compared to previous ensemble methods, Split-Ensemble also achieves similar or even better performance with significantly less cost.

For Table 2, we follow the suggestion of Reviewer 37MR to include results with recent benchmarks and baselines like Yang et al. [C1] and Cao et al. [C2] as well as some additional baselines like ODIN [C3], EBO [C4], OE [C5], MCD [C6], UDG [C1]. Results have been included in the Appendix of the revised manuscript.

The following two tables show the comparison between previous state-of-the-art methods and ours on the SC-OOD CIFAR10 benchmarks. The results are reported for models with ResNet-18 backbone. Some baseline methods use the Tiny-Imagenet dataset as additional OOD training data. Best score for each metric in bold, second-best italicized.

• The Split-Ensemble model outperforms single-model approaches in OOD detection without incurring additional computational costs or requiring extra training data. Its consistent high performance across key metrics highlights its robustness and efficiency, underscoring its practical utility in OOD tasks.

• The Split-Ensemble model consistently outshines other ensemble-based methods in both image classification and OOD detection, achieving similar or even better performance with $4\times$ less computation cost.

The following table shows the comparison between previous state-of-the-art ensemble-based methods and ours on the SC-OOD CIFAR10-LT benchmarks. The results are reported for models with ResNet-18 backbone. Best score for each metric in bold, second-best italicized.

• The Split-Ensemble model excels at handling challenging long-tailed dataset, as evidenced by its top-tier performance across major metrics in the SC-OOD benchmarks. This achievement is particularly notable given its efficiency, as it attains these results without incurring additional computational costs.

We thank reviewer 37MR for your thorough reviews and valuable comments. Hopefully the following responses can address your concerns.

W1.1: OOD Detection Criteria

A1.1: We follow the practice of ODIN [C1] and outlier exposure [C2] to consider the softmax probability of the model output as an estimation of model confidence, which is used as the OOD detection criteria. Since we train each submodel with a separated OOD-aware objective, as in Eq. (4), we perform the uncertainty estimation with the submodel $f_i$ that contributes to the ensemble output label, as discussed in Sec. 3.3.

W1.2: Post-Split architecture and parameter

A1.2: For the new branch after the splitting, the architecture and parameters are initialized as the exact copy of the original layers it is splitting from. This guarantees the same model functionality before and after the split. The branches will then be updated, pruned, and further split independently after the splitting is performed. We have updated this clarification in Sec. 4.1.

W1.3: Subtask Grouping Experiments

A1.3: This work focuses on the training objective of performing the subtask-splitting training for OOD awareness and achieving efficient architecture with iterative splitting and pruning. We heuristically design the subtask splitting based on the intuition that semantically close classes shall be grouped together (Sec. 3.1) so as to make the ID and OOD data in each subtask more distinguishable, and the results in Table 5 show this choice truly make a difference. We will explain the intuition behind semantically close grouping more clearly in Sec. 3.1 and point to this result. Meanwhile, we leave a more rigorous analysis of the optimal subtask grouping strategy as future work.

W1.4: Computation Budget

A1.4: As stated at the end of Sec. 4.2, the Floating Point Operations (FLOPs) of the original backbone model is used as the computation budget to guide the splitting and pruning of the Split-Ensemble. This is also reflected in Tab. 1 as the FLOPs of Split-Ensemble is reported as 1x, the same as the single backbone model.

W2: Recent Baseline and Benchmark

A2: Please see the next post due to space limitation.

W3: OOD Detection Threshold Determination

A3: Note that with the OOD detection metrics reported in the paper, we do not manually determine the OOD detection threshold for each model. For AUROC and AURP computation, we scan through all the possible thresholds to get the corresponding tradeoff curves. For FPR and detection error evaluation, we report the number with the threshold achieving 95% TPR. We have clarified this in the description of Tab. 2.

W4: Addressing Unknown OOD Data Distribution

A4: One unique feature of the proposed Split-Ensemble method is that we perform OOD-aware training without the need for external OOD data, so no knowledge of the OOD to be faced is required. Specifically, we propose subtask splitting training in Sec. 3, where each submodel is trained with part of the dataset as ID while the other as OOD. With the OOD objective in Eq. (3), each submodel becomes OOD aware, which is generalizable to unknown OOD distributions. This generalizability is verified by our results in Tab. 2, and on the new SC-OOD benchmarks you suggested. Our model, trained with only the ID data, can achieve better performance on different OOD distributions compared to baseline methods, even those trained with additional OOD data.

Thank you for your timely reply!

The motivation of subtask-splitting training is to reduce the redundancy of having multiple ensemble submodels learn the same task, and allow each submodel to have better uncertainty calibration with the OOD-aware training objective. To this end, the OOD-aware training objective in Equ. (2) and (3) allows each submodel to well classifiy the $K$ ID classes assigned to itself, and output low confidence for OOD samples outside of these $K$ classes, including the samples from the other $N-K$ classes of the overall training task and generalizable to external unseen OOD data. Since this OOD detection ability is learnt by each submodel, we use submodel output to perform uncertainty estimation as in Sec. 3.3.

For ensemble output, as introduced in Sec 3.3, we concatenate all the ID logits from submodels to form the ensemble logit for classification. In an ideal case, only the submodel which the correct class falls into will output a high confidence in the ID logits, while all the other submodels output low confidence. So the correct classification can be made with the concatenated ensemble logits. In practice, there may be a mismatch between the range of logits output from different submodels, where a low-confidence logit from one submodel may get a larger value than a high confidence logit of another independently-trained submodel. We therefore introduce Equ. (4) to calibrate logits ranges across submodel with the overall $\mathcal{L}_{CE}$. Note that the CE loss weight $\lambda$ can be as small as 1e-4 to achieve this calibration, so the calibration does not impact the individual performance of each submodel on its subtask, including its ability to detect OOD samples.