Monet: Mixture of Monosemantic Experts for Transformers

Can we mix Horizontal Expert Decomposition and vertical expert decomposition?

Thank you for your suggestions on additional experiments where two orthogonal decomposition methods can be mixed and complement each other. The results are presented as below:

Summary of 8 open-ended LLM benchmarks

Details of 8 open-ended LLM benchmarks

Q1. citation to PEER is missing.
Q2. No proper ablations to study different choices in the architectural design and no insight is provided.
Q3. What does the model start with in table 3?

Respectfully, we would like to correct that

• A1. A citation to PEER was already present in our 1. Introduction section.
• A2. An ablation study on auxiliary loss weights was present in Appendix Section C.1, where orthogonal architectural design choices have been rigorously compared in Section 3 across model sizes and benchmarks.
• A3. Table 3’s full performance was also present in Appendix Section E.

We understand that such a misconception is due to a lack of space in the paper where a fraction of the information had to be moved to the appendix. We would graciously ask you to read our revised manuscript if you could spare your invaluable time. Thank you once again.

[1] Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large Memory Layers with Product Keys. In Advances in Neural Information Processing Systems, volume 32, 2019.
[2] Xu Owen He. Mixture of a million experts. arXiv preprint arXiv:2407.04153, 2024
[3] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, et al. OLMoE: Open Mixture-of-Experts Language Models. arXiv preprint arXiv:2409.02060, 2024.
[4] Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. JetMoE: Reaching Llama2 Performance with 0.1M Dollars. arXiv preprint arXiv:2404.07413, 2024.
[5] Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, et al. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1280–1297, August 2024.
[6] Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, et al. Scaling Laws for Fine-Grained Mixture of Experts. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.

No baseline comparison against PEER and traditional SMoE.

Following your request for a traditional SMoE interpretability baseline, we have included knowledge unlearning of OLMoE [3]. OLMoE LLM with total 6.9B parameters has been selected as the representative baseline for conventional SMoE architectures for two reasons: (1) it has the largest number of experts among the publicly available SMoE LLMs [3-5] and (2) it has been trained with an extensive amount of tokens from various sources.

Monet-VD 1.4B’s Domain Masking Performance Perturbation in MMLU

• Mean of $\Delta$ Target: -2.65
• Mean of $\Delta$ Avg. Other: -0.18
• Mean of $\Delta$ Std. Other: 0.71

OLMoE 6.9B’s Domain Masking Performance Perturbation in MMLU

• Mean of $\Delta$ Target: -4.91
• Mean of $\Delta$ Avg. Other: -2.21
• Mean of $\Delta$ Std. Other: 1.72

Our additional experimentations suggest that OLMoE may be constituted with polysemantic experts. Results can be summarized as the following:

1. In OLMoE, there were extremely few specialized experts for MMLU, based on our criteria of skewness in expert routing score. In the case of Monet, we identified specialized experts if its highest routing score on a particular domain is twice as much as that of the second highest domain. However, OLMoE’s experts’ routing score was evenly distributed, making it difficult to detect specialized experts. We leveraged criteria of occurrences in maximum activation to determine the expert’s domain specialization to obtain the results.
2. OLMoE’s accuracy drop in other domains was significant in unlearning, possibly due to the entangled characteristics of experts since their specializations were only detectable with argmax criteria.
3. We measured delta performances’ mean standard deviation of the other 13 domains, resulting in 0.7 for Monet and 1.7 for OLMoE, differing twice as much, showing disparity in stability of knowledge conservation during unlearning.

We believe that such results suggest that for most of the SMoE architectures [3-6] with 64 experts or less, the expert count is too small to disentangle polysemanticity. Our architecture, on the other hand, has 262,144 experts available, which we believe enable fine-grained specialization, resulting in monosemantic experts that capture mutually exclusive aspects of knowledge. To further address your inquiry, we provide an overview of the unlearning results of Monet, Gemma Scope, OLMoE, and LLaMa in Figure 3 in our revised paper.

Despite the fact we have previously compared the time complexity and space complexity with the PEER baseline, we remind you that additional 100B parameters are needed to constitute a PEER baseline, as we have explained in Section 2 of our paper. Such exorbitant memory requirements are beyond the scope of most researchers (note that PEER was introduced by Google Deepmind), where our contribution is to achieve parameter efficiency, precisely because directly implementing the PEER baseline is infeasible.

We appreciate the comments about the method and are sorry about the confusion.

Q1. Presentation can be greatly improved.
Q2. Which part of the changes over PEER make it superior?
Q3. Incremental proposal on top of PEER, I am uncertain how significant the contributions are.

Please refer to our updated manuscript, where we have improved the readability in sections 2. Preliminaries to through 3. Monet. Based on your feedback Q1, we have also enhanced the presentations of Figure 1 in the revision.

To summarize sections 2 and 3:

1. Inspired by the product key algorithm [1], PEER [2] processes up to a million experts with product key retrieval.
2. Despite its computational efficiency, PEER requires to initialize and store $N$ standalone experts, resulting in memory usage that grows linearly with the number of experts, $O(N)$.
3. In response to Q2, our contribution is partitioning the expert’s MLP network into two different groups of segments and storing them within $O(\sqrt{N})$ memory constraint. During the training or inference, the learned router dynamically composes expert networks to form $N$ combinations of experts.

Below is a comparison of time complexity for expert retrieval and space complexity for expert parameters:

where $d$ is the hidden dimension of the expert, $m$ is the dimension of the individual expert, $k$ is the TopK hyperparameter, and $H$ denotes multi-head of the router.

Regarding Q3, we suggest that our contribution is significant considering that our product key composition has optimized space complexity while maintaining the time complexity of PEER.

Dear Reviewer @YJRi,

Thank you for your insightful feedback. We understand that your primary concern is how our Monet architecture compares to traditional SMoE architectures in terms of the final quality of the model. As practitioners, we agree that assessing model performance is crucial alongside interpretability.

To address your concern, we conducted additional experiments to provide a direct comparison between Monet and the state-of-the-art SMoE architecture, OLMoE [1]. We ensured a fair evaluation by matching both the number of active parameters and the total number of parameters, as well as training both models on the same amount of data.

Total Parameter Matched Comparison

In this setup, both models have a similar total parameter count and are trained on 100 billion tokens.

Overall Performance

Benchmark Results

Zero-shot Performance

5-shot Performance

Active Parameter Matched Comparison

To ensure an apples-to-apples comparison within our limited time frame, we conducted the active parameter matched experiments over a shorter training period. Both models have the same number of active parameters (1.3B) and were trained on 20 billion tokens.

Overall Performance

Benchmark Results

Zero-shot Performance

5-shot Performance

Discussion

The results indicate that Monet consistently outperforms the traditional SMoE model across multiple benchmarks in both zero-shot and 5-shot settings. By matching both the total and active parameter counts, we ensured that the performance gains are attributable to the architectural differences rather than model size or training data volume. These findings demonstrate that Monet not only offers improved interpretability but also delivers superior performance compared to conventional SMoE architectures.

We have revised the manuscript accordingly to include these comparisons and address your feedback. We appreciate your suggestion, as it encouraged us to perform this comprehensive comparison. We hope this addresses your concern regarding the final quality of the model. Please let us know if you have any further questions or suggestions.

[1] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, et al. OLMoE: Open Mixture-of-Experts Language Models. arXiv preprint arXiv:2409.02060, 2024.

We would like to express our gratitude to the reviewer for their constructive response. Below we respond to the weaknesses and questions.

Q1. The explanation in the methodology section is poor and hard to understand.
Q2. Are there any trade-offs for adopting Monet over traditional MoE?
Q3. The intuition behind the architecture design is unclear.
Q4. What's the reason of choosing $512^2$ experts?

Please refer to our updated manuscript, where we have improved the readability in sections 2. Preliminaries to through 3. Monet. We appreciate your comments about the clarity, and we are sorry about the confusion.

To summarize sections 2 and 3:

1. Inspired by the product key algorithm [1], PEER [2] processes up to a million experts with product key retrieval.
2. Despite its computational efficiency, PEER requires to initialize and store $N$ standalone experts, resulting in memory usage that grows linearly with the number of experts, $O(N)$.
3. In response to Q1, our contribution is partitioning the expert’s MLP network into two different groups of segments and storing them within $O(\sqrt{N})$ memory constraint. During the training or inference, the learned router dynamically composes expert networks to form $N$ combinations of experts.

Below is a comparison of time complexity for expert retrieval and space complexity for expert parameters to address Q2:

where $d$ is the hidden dimension of the expert, $m$ is the dimension of the individual expert, $k$ is the TopK hyperparameter, and $H$ denotes multi-head of the router. Individual expert dimension $m$ can be any value in our architecture, but PEER had to use a fixed value of $m=1$ because of a memory bottleneck.

• Regarding Q3, our purpose is to optimize space complexity while maintaining the time complexity of PEER.
• For Q4, we have followed the product key counts as mentioned in [1] of $512^2$ for our product key composition.

Thank you for your thoughtful feedback that has helped refine our paper. We welcome any further questions or suggestions that could enhance the contribution of our work to the field.

[1] Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large Memory Layers with Product Keys. In Advances in Neural Information Processing Systems, volume 32, 2019.
[2] Xu Owen He. Mixture of a million experts. arXiv preprint arXiv:2407.04153, 2024.

Lack of baselines
Q1. I think it is important to maybe compare the Monet method with standard MoEs
Q2. Would for instance fine-grained MoEs work in this case?
Q3. Is it the fact that we have a lot of experts that is responsible for more “monosemantic” experts?

Following your request of a fine-grained SMoE interpretability baseline in Q1 and Q2, we have included knowledge unlearning of OLMoE [3]. OLMoE LLM with total 6.9B parameters has been selected as the representative baseline for conventional SMoE architectures for two reasons: (1) it has the largest number of experts among the publicly available SMoE LLMs [3-5] and (2) it has been trained with an extensive amount of tokens from various sources.

Monet-VD 1.4B’s Domain Masking Performance Perturbation in MMLU

• Mean of $\Delta$ Target: -2.65
• Mean of $\Delta$ Avg. Other: -0.18
• Mean of $\Delta$ Std. Other: 0.71

OLMoE 6.9B’s Domain Masking Performance Perturbation in MMLU

• Mean of $\Delta$ Target: -4.91
• Mean of $\Delta$ Avg. Other: -2.21
• Mean of $\Delta$ Std. Other: 1.72

Our additional experimentations suggest that OLMoE may be constituted with polysemantic experts. Results can be summarized as the following:

1. In OLMoE, there were extremely few specialized experts for MMLU, based on our criteria of skewness in expert routing score. In the case of Monet, we identified specialized experts if its highest routing score on a particular domain is twice as much as that of the second highest domain. However, OLMoE’s experts’ routing score was evenly distributed, making it difficult to detect specialized experts. We leveraged criteria of occurrences in maximum activation to determine the expert’s domain specialization to obtain the results.
2. OLMoE’s accuracy drop in other domains was significant in unlearning, possibly due to the entangled characteristics of experts since their specializations were only detectable with argmax criteria.
3. We measured delta performances’ mean standard deviation of the other 13 domains, resulting in 0.7 for Monet and 1.7 for OLMoE, differing twice as much, showing disparity in stability of knowledge conservation during unlearning.

We believe that such results suggest that for most of the SMoE architectures [3-6] with 64 experts or less, the expert count is too small to disentangle polysemanticity. Our architecture, on the other hand, has 262,144 experts available, which we believe enable fine-grained specialization, resulting in monosemantic experts that capture mutually exclusive aspects of knowledge. To further address your inquiry of Q3, we provide an overview of unlearning results of Monet, Gemma Scope, OLMoE, and LLaMa in Figure 3 in our revised paper.

We sincerely appreciate your thorough review and valuable suggestions, which have helped strengthen our manuscript substantially. We remain available to address any additional questions or concerns you may have.

[3] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, et al. OLMoE: Open Mixture-of-Experts Language Models. arXiv preprint arXiv:2409.02060, 2024.
[4] Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. JetMoE: Reaching Llama2 Performance with 0.1M Dollars. arXiv preprint arXiv:2404.07413, 2024.
[5] Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, et al. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1280–1297, August 2024.
[6] Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, et al. Scaling Laws for Fine-Grained Mixture of Experts. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.

We would like to express our gratitude for your positive feedback on our paper's idea and the effort you invested in its assessment. In the following response, we will address each of the weaknesses and questions you have raised.

Section 3 should be more clear.
Q1. Why is there any memory savings (compared to the PEER approach)?
Q2. Why is each expert of dimension m (while in PEER, it is a single neuron)?
Q3. I also recommend the authors to do a complexity calculation, Section 3.2 to be fully transparent on the memory/computation complexities.
Q4. I think that this drawing should be improved.

Please refer to our updated manuscript, where we have improved the readability in sections 2. Preliminaries to through 3. Monet. We appreciate your comments about the clarity, and we are sorry about the confusion.

To summarize sections 2 and 3:

1. Inspired by the product key algorithm [1], PEER [2] processes up to a million experts with product key retrieval.
2. Despite its computational efficiency, PEER requires to initialize and store $N$ standalone experts, resulting in memory usage that grows linearly with the number of experts, $O(N)$.
3. In response to Q1, our contribution is partitioning the expert’s MLP network into two different groups of segments and storing them within $O(\sqrt{N})$ memory constraint. During the training or inference, the learned router dynamically composes expert networks to form $N$ combinations of experts.

Below is a comparison of time complexity for expert retrieval and space complexity for expert parameters:

where $d$ is the hidden dimension of the expert, $m$ is the dimension of the individual expert, $k$ is the TopK hyperparameter, and $H$ denotes multi-head of the router.

• Regarding Q2, dimension $m$ can be any value in our architecture, but PEER had to use a fixed value of $m=1$ because of a memory bottleneck.
• Regarding Q3, specific complexity calculation is present in Appendix A.2 in our updated manuscript, where the table above provides a brief overview and comparison.
• Based on your feedback Q4, we have also enhanced the presentations of Figure 1 in the revision.

[1] Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large Memory Layers with Product Keys. In Advances in Neural Information Processing Systems, volume 32, 2019.
[2] Xu Owen He. Mixture of a million experts. arXiv preprint arXiv:2407.04153, 2024.

How exactly were the top experts by subdomain chosen for the Gemma-2B SAEs? Note that SAEs have no notion of probability over the "experts", unlike the MONET model, and I could not find this addressed in the paper. Do you pass the hidden SAE activations through a softmax first?

We referred to the steering methods with SAEs, such as clamping the feature activations [7, 8] based on their logit values. To adhere to the conventional logit-based steering, we analyzed the skewness of SAE’s logit values, where we determine the feature is specialized in the particular domain only when its highest logit value is at least twice higher than that of the second most activated domain.

The only relevant baseline here is using SAEs at the MLP layers, because this matches the MONET setup; so, the residual stream SAEs seem irrelevant for this work?

While SAEs at the MLP layers correspond to the Monet's fine-grained experts, we chose to include residual stream SAE results for comprehensiveness. The MLP-based comparisons demonstrate the core architectural benefits, while the residual stream results provide context within the broader landscape of interpretability research. This allows readers to evaluate Monet's effectiveness against both the most directly comparable baseline and current common practices in the field.

What is the scale in figure 2?

Regarding the scale and full performance of each Monet (ours), Gemma Scope, OLMoE, and LLaMa in MMLU domain unlearning, we have listed in Appendix E’s Table 11 through Table 14 for the specifics. Please refer to the revised manuscript, and if you have additional inquiries, we are happy to respond to further questions and comments.

• For example, the only interpretability method used as a baseline is patching reconstructions from SAEs for Gemma-2B. However, it is not reported what sparsity these SAEs achieve compared to the (effective?) sparsity of MONET. This makes it difficult to make sense of the results.
• The primary goal of SAEs is to find interesting concepts used by the model, and reconstruction is secondary to that (and being able to chain SAE reconstructions is even more secondary). So, ideally the baseline would compare the "monosemanticity" of MONET features vs SAE ones.

We employed Gemma Scope with 262K features at $L_0 = 263$, its maximum provided sparsity setting. However, direct sparsity comparisons between Monet and SAE models are not methodologically sound due to fundamental architectural differences. While MoE models use top-k routing for sparse expert activation, this mechanism differs fundamentally from SAE's $L_0$ sparsity measure.

Nevertheless, our Monet's theoretical sparsity would be $L_0$ is 512, derived from $|\mathcal{K}_h^1 \times \mathcal{K}_h^2| = 64$ across 8 multi-head routings. Despite this higher $L_0$ value, which traditionally would suggest lower monosemanticity, Monet achieves superior disjoint unlearning performance, as demonstrated in Figure 3 in our revised manuscript. This indicates that routing-based sparsity may be more effective at isolating and controlling specific knowledge domains compared to traditional SAE approaches.

We thank you again for your constructive comments and for your efforts to improve the quality of our paper. Please let us know if you have any further questions or if we can provide further clarification.

[1] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, volume 35, pp. 17359–17372.
[2] Dmitrii Kharlapenko, neverix, Neel Nanda, and Arthur Conmy. Self-explaining SAE features. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/8ev6coxChSWcxCDy8
[3] Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscope: A Unifying Framework For Inspecting Hidden Representations of Language Models. arXiv preprint arXiv:2401.06102, 2024.
[4] Haozhe Chen, Carl Vondrick, and Chengzhi Mao. SelfIE: Self-Interpretation of Large Language Model Embeddings. arXiv preprint arXiv:2403.10949, 2024.
[5] John Hewitt, John Thickstun, Christopher D. Manning, and Percy Liang. Backpack Language Models. In Annual Meeting of the Association for Computational Linguistics, 2023.
[6] Alex Tamkin, Mohammad Taufeeque, and Noah D Goodman. Codebook Features: Sparse and Discrete Interpretability for Neural Networks. arXiv preprint arXiv:2310.17230, 2023
[7] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024.
[8] Adly Templeton*, Tom Conerly*, Jonathan Marcus, Jack Lindsey, Trenton Bricken, et al. Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread, 2024, URL https://transformer-circuits.pub/2024/scaling-monosemanticity

We sincerely thank the reviewer for your helpful and constructive suggestions. In the following response, we will explicate the changes that have been made to the manuscript and a new version uploaded.

Have you tried running the MONET features through an automated interpretability pipeline?

We express gratitude for suggesting such valuable feedback, where we have reflected the changes in Figure 2 and in the 4.3 Qualitative Results section in our revised manuscript.

The attached example, sae-auto-interp has its significance in generating explanations of SAE features via external LLM or through the compatible API. We agree that features within the model should be able to be described in natural languages, considering its significance in controlling and managing LLMs.

Taking your advice, we decided to take a step further, referring to the Self-explaining SAE features [2]. It states that it has advantages over sae-auto-interp of the following: no max activating dataset examples are needed, and it’s cheaper by using the model subject of study to generate about its own features rather than relying on a larger model like GPT-4.

Without using external LLMs or APIs, we adapted an automated interpretation framework as Self-explained Experts, where Monet-1.4B CHAT generates a description for its own experts. We have referred to the work of Patchscope [3] and SelfIE [4], where both works prompt LLM to answer “Q: What is the meaning of the word X? A: Sure! The meaning of the word X is ”, where X serves as a placeholder for target token embedding for the analyses. Similarly, we averaged token embeddings activated for the targeted expert and have inserted them into the aforementioned placeholder. Our Monet-1.4B CHAT generated a description for its experts, like explaining the Expert 232,717 as “Cartilage” and the Expert 51 as “Expertise”, as stated in our revised manuscript.

Q1. The paper would benefit from a discussion of, and comparison with, related work, such as backpack language models and codebook features.
Q2. Perhaps adding extra bells and whistles like instruction tuning or multimodality distracts from the main goal of the paper, which is to establish the usefulness of the new architecture for interpretability.

We thank your opinion on the paper’s related works and its main goal.

• We have reviewed Backpack LLMs [5] and Codebook Features [6] according to your advice Q1, where we found encoding interpretable weights in LLM during pretraining shares similar philosophy in achieving interpretable models. In our 1. Introduction section, we have reflected the change accordingly.
• Furthermore, we value your advice Q2 and took out the examples of multimodal experts (Figures 9 and 10) from the main text and moved to the appendix section. The rationale for staying in the paper is that it is yet unknown whether it is generalizable for fine-grained experts to specialize in and capture monosemantic concepts across modalities with finetuning. We would appreciate it if you could reconsider the significance of analyzing the expandability of our method in LLM’s multimodal integration to remain in the paper’s appendix section.
• In the case of instruction tuning, the process was a precursor of the automated interpretability pipeline. Adhering to your suggestion, we have excluded the specifics regarding instruction tuning and moved to the Appendix, but we discussed its role in Self-explained Experts as we mentioned above in the previous response.

A baseline using the ordinary MLP neurons of the LLaMA model would be very valuable to make the point that MONET discovers more interpretable structure compared to the neuron basis.

Thank you for your insightful suggestion. In response, we have included the LLaMA unlearning baseline in Figure 3 and in 5.1 Domain Masking section of our revised manuscript.

In our experiments, we suppressed domain-specific MLP neurons based on first-layer activations. Inspired by the ROME [1] treating MLP as key-value pairs, we identified neurons with domain specialization based on GELU activations. Specifically, if the highest activation of a particular domain is twice as much as that of the second highest activated domain, we consider that neuron a specialized neuron.

For the results, LLaMA displays an average 6% of neurons to be specialized in each domain compared to Monet's 2.2%, suggesting possible feature entanglement and resulting in significant performance degradation across unrelated domains during knowledge removal. We measured delta performances’ mean standard deviation of the other 13 domains, resulting in 0.7 for Monet and 1.4 for LLaMa, differing twice as much in stability of knowledge conservation during unlearning. Such results highlight Monet’s monosemanticity, where experts encapsulate disentangled parametric knowledge across domains.

Thank you for your support and for endorsing the acceptance of our paper. In the following responses, we address each of the concerns you have raised and have updated the manuscript accordingly (*changes highlighted in magenta).

the autointerpretability evaluation is exploratory and does not demonstrate metrics showing the improved interpretability of MONET compared to other methods;

Thank you for raising this important point about quantitative evaluation of automated interpretability. We want to clarify that primary objective of self-explained experts was to leverage LLMs' internal knowledge and capabilities by eliminating dependencies on external LLMs, rather than to demonstrate superiority in automated interpretation frameworks. We note that concurrent work in quantitatively measuring automated interpretability of SAEs [1,2] are still in its early stage of development, which we view as an opportunity to develop more comprehensive evaluation protocols.

While tools like sae-auto-interp provide valuable pipelines for generating and evaluating feature explanations, their quantitative evaluation frameworks are currently designed to compare different explanation methods between each LLM, rather than enabling direct comparisons between SAE models. We plan to prioritize developing more robust comparative frameworks in our future work to provide additional numerical assessment of Monet's automated interpretation framework.

the results described in Figure 3 are very interesting, but they are a bit hard to read due to the unclear scale. In particular, looking at Appendix E, I found the last two rows of each table (Δ Target and Δ Others) to be most helpful in making sense of this dense data. I would advise somehow surfacing this in the camera ready if the paper is accepted;

Thank you for your positive feedback on Figure 3. We appreciate your feedback that, while the results are very interesting, the unclear scale made them hard to read. We agree that relying on Appendix E for clarity might be inconvenient for readers. In response to your suggestion, we have updated Figure 3 in the revised manuscript to include precise scales. We believe these editorial changes will enhance the readability of our results. Thank you for bringing this to our attention, and we apologize for any inconvenience the original presentation may have caused.

I feel that the exploration of methods for picking experts was insufficient. I would love to see future work/revisions more thoroughly tuning the choice of experts for each baseline as well as MONET.

Thank you for your valuable feedback. We acknowledge that our exploration of methods for selecting experts was insufficient and agree that more thorough tuning is necessary.

In our current work, we used the skewness of the routing score to determine experts' domain specialization and identified toxic experts using the Pearson correlation coefficient between the toxicity score and the routing score. We recognize that these criteria are basic and minimal.

Our primary contribution lies in making the LLM transparent, enabling researchers to observe routing scores and directly manipulate the parametric knowledge. We believe that the routing scores of monosemantic experts allow researchers to observe patterns for retrieving intrinsic knowledge, which were previously opaque in polysemantic LLMs. We are optimistic that such observations can lead to addressing research questions related to hallucinations (e.g., "Is the model confident in retrieving internal knowledge?") and lifelong learning in LLMs (e.g., "How can we incorporate additional knowledge into the model?").

Based on your feedback, we have added a "Limitations" section to our paper, summarizing the discussions above. Thank you once again for your insightful comments, which have been invaluable in guiding the future direction of our research.

[1] Jack Lindsey, Hoagy Cunningham, and Tom Conerly. Interpretability Evals for Dictionary Learning. Transformer Circuits Thread, 2024, URL https://transformer-circuits.pub/2024/august-update/index.html#interp-evals

[2] CHAUDHARY, Maheep; GEIGER, Atticus. Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small. arXiv preprint arXiv:2409.04478, 2024.