Filtered not Mixed: Filtering-Based Online Gating for Mixture of Large Language Models

Responses to Questions

[Q1] How does the performance of MoE-F degrade with an increasing number of experts, and what are the computational scaling implications?

The theoretical runtime of our filtering algorithm is $O(N^2)$, but in practice, the number of experts is typically limited (~7 in our work). Thus, the (algorithmic) scaling implications are negligible for realistic use cases.

Scaling concerns primarily stem from the computational cost of expert inference, especially with large-scale, billion-parameter LLMs. To elucidate, simultaneously hosting a stream of 7 billion-parameter LLM experts requires significant inference compute infrastructure. However, MoE-F itself introduces no significant overhead, as it serves as a lightweight filtering harness applicable to a heterogeneous mix of experts.

To address potential scalability concerns:

• A top-k experts strategy can maintain a fixed computational budget, with less effective experts relegated and replaced periodically by offline or new experts.

• Cost-effective classical models (e.g., ARIMA) can complement large-scale experts in low-resource settings. This modularity ensures that MoE-F adapts to a wide range of computational constraints.

[Q2] What is the impact of different market regimes (high volatility vs. low volatility) on the filtering performance?

Market volatility estimation, while challenging, does not directly impact our method. Unlike traditional filtering setup where the measurement process reflects stock prices and volatility is the unobservable, latent signal process affecting the stock price. This does not apply here because we consider a single non-random path (similarly to the rough path integration, Rough Differential Equation (RDE) literature). MoE-F evaluates the performance of experts, with the "hidden signal" being the identity of the best expert at any time.

Nevertheless, if regime changes occur rapidly (e.g., small occupation time of the hidden Markov process, i.e. frequent switching of the best expert), any statistical method, including MoE-F, may struggle. Conversely, in stable regimes where the expert performance ratios remain consistent, MoE-F is well-suited to approximate the best expert. Our mechanism relies on the stability of these performance patterns to function effectively, which aligns well with situations where market dynamics are not too erratic in nature.

---

Finally, we would like to respectfully request that you reconsider the ranking of our paper. Given its originality and the strength of its theoretical and practical contributions, we believe it could be a valuable addition to the oral or highlight sessions at the conference. Presenting in such a format would enable us to reach a wider AI/ML audience, sparking broader discussions about the fusion of classical control theory with modern machine learning paradigms. We would be deeply grateful if you could consider this when finalizing your assessment.

Thank you once again for your valuable feedback and consideration.

Dear Reviewer,

We deeply appreciate your thoughtful feedback and strong endorsement of our work, particularly your recognition of its originality and creativity in fusing classical control theory with modern machine learning. This positive assessment strengthens our commitment to ensure our work permeates and benefits the wider AI/ML research community and beyond.

Responses to Weaknesses:

[W1] The financial market experiments focus primarily on a single dataset (NIFTY), and broader validation across different types of time series problems would strengthen the claims of generality.

We thank you for pointing this out. As highlighted in our submission, we deliberately designed our experiments to span two fundamentally different time-series forecasting (TSF) tasks: the financial market movement (FMM) task and the electric grid long-term load forecasting (LTSF) task. These tasks were chosen for their orthogonality in dimensions such as domain, features, observation windows, forecasting horizons, stationarity, and evaluation metrics, illustrating the broad applicability of our approach.

If we understand your concern correctly, it pertains to the usage (or lack thereof) of additional datasets within the FMM task. While the financial domain indeed offers diverse datasets (e.g., commodities, cryptocurrencies, economic indicators), we deliberately opted to focus on demonstrating the orthogonal use case of LTSF. This decision was driven by two factors:

1. Illustration of two loss paradigms: Adding LTSF allowed us to showcase our method's efficacy across both cross-entropy loss (classification in FMM) and MSE, mean squared error (regression in LTSF) cases.

2. Avoiding financial domain bias: Incorporating more financial datasets could overshadow our primary contribution - the relative performance guarantees and flexibility of the MoE-F framework - by the overt focus on downstream financial domain-specific results that are usually means to an end.

Nevertheless, we acknowledge that additional FMM datasets could further validate our claims within this domain. We plan to incorporate these datasets in future work and appreciate your suggestion.

Dear Reviewer,

We are sincerely grateful for your helpful feedback and kind words of encouragement. We have used your feedback to further enhance our manuscript for clarity and exposition. Below, we outline the specific actions we have taken to address the weaknesses you identified:

Response to Weaknesses:

[W1] The writing can be improved in some sections, specifically the implications of theorem 1 and 2 need to be explained better.

We appreciate the feedback and agree that some of the writing could be better harmonized. As you will see in our updated manuscript, we have improved the exposition by streamlining the introductory sections to improve the clarity and flow better. (specific changes detailed below). These changes, along with our additional feedback-based writings refactorings, should significantly enhance the flow and clarity of our manuscript.

[W2] In the TSF application it is unclear at what granularity of time-steps is the filtering applied. For instance if the horizon is 720, do the individual experts predict every 60 time-steps and then their 60 step look-ahead predictions are combined and then we move to the task of predicting the next 60 time-steps. Or do they predict one step at a time for 720 time-steps?

This is a great question - thank you for pointing out this ambiguity. The two plausible scenarios you described are essentially what the granularity categories in LTSF literature are dichotomized into:

• (i) Iterated multi-step (IMS): predicting one step at a time over the horizon, $H$: $\hat{x}_{t+1}$, and

• ii) direct multi-step (DMS): predicts the entire future window, $H$ in a single operation: $\hat{x}_{t+1:t+H}​$.

In our experiments, we adopt the Direct Multi-Step (DMS) strategy for filtering, where each expert predicts the entire horizon, $H$, in one step. The look-back window (or, lag period), $L$, is set to 720 time-steps, and our method follows a channel-independent (CI) strategy. This approach aligns with state-of-the-art (SOTA) Long-Term Series Forecasting (LTSF) architectures. With this setup, the Mean Squared Error (MSE) loss is then calculated across all channels $C$, as outlined by the equation in S5.2 [Ln. 455] of our manuscript.

To ensure clarity for all readers, we have added Appendix Section E.2 [Ln. 1713] to the revised manuscript, where we discuss these distinctions in detail, including the differences between IMS and DMS strategies and their implications for our experimental setup.

Changes to the Manuscript:

In our updated manuscript:

• We have streamlined the presentation of notations, main steps, and helper functions definitions w.r.t. the theorems and algorithms to improve exposition and flow. Specific changes include, moving Figure 2 later in section 3, and incorporating the main steps (1,2) descriptions to within the paragraps from caption. The helper functions are defined later in S3, following preliminaries (notations, loss functions), adjacent to the main algorithm's presentation for easier juxtaposition and reference.

• We added Appendix Section E.2 to discuss the granularity of time-steps in our time-series forecasting (TSF) experiments. This section explains our use of DMS and the channel-independent approach, as well as the rationale behind these choices.

We are confident that these additions and revisions address your concerns and make our methodology and contributions more transparent. If you have further questions or need more clarification, please let us know.

Once again, we greatly appreciate your constructive feedback and hope that our improved manuscript meets your expectations. Thank you very much for your time and for highlighting points that have allowed us to improve our work.

Dear Reviewer,

We firstly thank you for acknowledging the strengths of our work for its novelty of methodology and significance of contribution. We would like to respectfully address the concerns you raised regarding the structure and presentation of our manuscript.

While the overall review is unexpected/unusual, upon careful examination, it appears the concerns raised primarily pertain to the structure and presentation of the manuscript, rather than an evaluation of its contributions or the soundness of our methodology. In our revised version, we have made significant efforts to address these presentation issues and hope this allows for a fair assessment of the work's core methodologies.

Responses to Weaknesses

[Major Comments] Clarity, notations and soundness

We recognize that expectations around the presentation style can be subjective, and while we understand your preference for a different structure, we would like to highlight that our manuscript follows a standard format commonly seen in AI/ML conference submissions. If it was indeed a case of highly unorthodox presentation for ICLR, the issue would be reflected, flagged across the reviews immediately.

Nevertheless, with our aim to make our work accessible, we have taken your feedback seriously and meticulously addressed all the raised concerns to improve the clarity of our manuscript within the constraints of the conference format. Specifically:

1. Clarity and Flow: We have restructured the introductory sections 1 to 3 to create a more linear and cohesive flow. Notably, we moved Figure 2 to Section 3 and integrated its caption content into the main text, making it easier to follow. We also ensured that important definitions and steps are positioned close to their usage, thereby reducing the need for forward referencing and improving readability. For example, the main two steps are now contiguous within Section 3, adjacent/close to the main algorithm, allowing easier juxtaposition and referencing.

2. Notation and Definitions: In response to your concerns about undefined or poorly defined notation, we have added explicit explanations earlier in the manuscript. Section 2 has been revised to include a clear introduction to the necessary notation and concepts in continuous time, ensuring consistency throughout the paper. The helper functions are now introduced following adequate preliminaries (like conceptual overview, loss functions) in section 3.

3. High-Level Explanation: We have added a high-level overview at the beginning of Section 3 to provide context for the filtering algorithm and its purpose. Additionally, we included a remark in Section 2 pointing readers to Appendix D for more in-depth background on the filtering methodology, for those interested in further details.

Notations

• The differing versions of $\pi_n$ are minor typos meaning the same $\pi^{(n)}_t$ - now fixed.
• $Q^{(n:i,j})$ and $Q^{(n)}$ (line 174) are deprecated notation, it should read $Q^{i,j}_t$ as updated in our revised draft (line 125).
• You are correct, this is a typo: $Y_t$ has to be given to $\ell$ as second argument.
• With $P^{(n)}$ we meant the n-th row of the matrix $P$. We changed to the more standard notation $P_{(n,\cdot)}$.
• $P_{w_{t+1} | w_{t-1+H}}$ should read 'on the time window $[t-H+1,t]$'

Soundness As you have acknowledged, you were unable to do a fair assessment of the soundness based on raised presentation issues. We hope our revised draft resolves this impediment, and we very much look forward to your updated thoughts.

[Minor comments and Typos]

While we respectfully do not agree with every minor comment raised, we have nonetheless addressed and fixed all these (minor comments, typos) concerns. We are happy to address any further cascading/remaining concerns.

Dear Reviewer 5hye,

We appreciate your constructive feedback and agree that more motivation and high-level intuition for the MoE-F algorithm would enhance clarity. We have revised the introduction to increase the transparency of our MoE-F algorithm and its motivation. Additionally, we have moved the formal definitions of the signal, measurement process, and loss to Section 3, as these elements formalize our setup but do not sufficiently convey the natural intuition behind it, which we aim to address earlier in the paper.

Specifically, the updates to clarify the motivation and intuition for the MoE-F algorithm can be found on lines 56-83 and 145-150, with all modifications highlighted in blue.

Distinguishing Static vs. Dynamic/Temporal Problems (Updated Lines 61-66)

Classical Mixtures of Experts (MoEs) are typically designed for non-temporal (static) prediction tasks, where a fixed gating mechanism is used to route inputs. In temporal prediction tasks, however, there is continuous feedback on the experts' performance, which can be leveraged to dynamically update the gating mechanism since this information indicates which expert model may be relevant and which may not be. This key distinction is crucial in understanding how MoE-F operates.

Rationale for a Filtering Approach in Temporal Settings (Updated Lines 68-74)

In the temporal setting, the user constantly estimates the mixture coefficients, with their objective being the minimization of a prediction loss based on each expert's measured performance. The mixture coefficients represent a partially observed signal, inferred indirectly through the experts' observed performance. Hence, predicting these coefficients aligns naturally with the continuous-time finite state-space stochastic filtering problem, as described in [1].

How the MoE-F Algorithm Works

The MoE-F algorithm leverages this inherent stochastic filtering structure in temporal prediction problems. It operates in two stages: first, $N$ parallel stochastic filters predict the optimal mixture weights over time based on the historical performance of each expert model. These predictions are then aggregated into a single set of mixture weights by solving a robust optimization problem similar to those used in PAC-Bayes theory [2].

We really appreciate your feedback on our writing style and we genuinely do believe it has strongly improved the quality of our presentation.

Much appreciation,

The Authors

References

[1] Wonham, W. Murray. "Some applications of stochastic differential equations to optimal nonlinear filtering." Journal of the Society for Industrial and Applied Mathematics, Series A: Control 2.3 (1964): 347-369.

[2] Alquier, Pierre. "User-friendly introduction to PAC-Bayes bounds." Foundations and Trends® in Machine Learning 17.2 (2024): 174-303.

Thank you for your replies.

I have had a look at the revision and I will increase my score since some of my comments (especially the typos, although I am still finding some (other) typos) have been addressed.

However, some of the main points in my "Clarity needs to be drastically improved" comments have not been satisfactorily addressed. See especially the last recommendation about starting from a high-level explanation of the idea behind the algorithm. At the moment, the paper simply states an algorithm without sufficient intuition for why this algorithm would be valid or how it was derived. I don't think this is good enough.

Dear Reviewer,

We thank you for recognizing the originality and strength of our work. We are especially grateful for your astute observation regarding the F1-measure discrepancies related to precision and recall values in Table 2 and Table 3 of our experimental results for the financial market movement (FMM) prediction task.

[W1] This concern comes from the strange behavior of F1 score in Table 2 and Table 3. In some columns (e.g., most columns in Table 3), the value of F1 is smaller than both Precision and Recall. This is really strange. In my understanding, F1 is a harmonic mean of Precision and Recall, and hence it cannot be less than both Precision and Recall.

For the FMM task (evaluated on the NIFTY test split), the evaluation falls under the category of multi-class classification with three labels: "fall," "neutral," and "rise," for which the support is imbalanced across the labels (support: 73, 143, 101, respectively). This class imbalance necessitates a careful approach to evaluation metrics, particularly in averaging methods for precision, recall, and F1-score.

We use the standard scikit-learn​​ library methods for evaluation. The seemingly unexpected behavior of the F1-score being lower than both precision and recall arises from our use of weighted averaging—a behavior that has been noted and documented in several discussions and issues 1, 2.

Under the 'weighted' averaging scheme, metrics for each class are computed separately, then averaged based on the class support (i.e., the number of instances for each label). In imbalanced settings, this approach can produce a global average where F1 is lower than both precision and recall due to the influence of poor performance on minority classes, effectively weighting down F1-score.

To illustrate this phenomenon and provide additional clarity, we have added Appendix E.1 section [Ln. 1679] in the revised manuscript, which includes the confusion matrices corresponding to Table 2. These confusion matrices clearly show how the performance varies across different label classes, providing a better visual understanding of the model's behavior. The F1-score's sensitivity to poor performance in any class is evident, and this sensitivity explains the observed values being lower than both precision and recall in some cases.

Our updated manuscript now include updated Table caption [Ln. 378] and this explanation [Ln. 1681-1688], aiming to assist other readers with this astute observation. We have also updated our supplementary materials repository [1] with a script (generate_results.sh) that allows for the complete reproduction of the experiment results (with both weighted and micro averaging schemes), in alignment with our commitment to reproducibility as per our "Reproducibility Statement."

We hope that our detailed explanation and improvements to the manuscript sufficiently address your concern. If so, we kindly request that you reconsider your overall evaluation, as we believe our work, if allowed to present in a highlighted session at the conference, will significantly bolster our contributions to reach a wider AI/ML audience and future works in this promising new direction of better harnessing the ever-growing list of LLMs.

Reference:

[1] https://anonymous.4open.science/r/moe-f

Once again, thank you for your detailed feedback and for highlighting an important point that has allowed us to strengthen the clarity of our work.