An Architecture Search Framework for Inference-Time Techniques

Thank you for taking the time to read our paper! We appreciate your feedback and comments.

We’d like to address each of your concerns individually:

• Weaknesses: -The biggest weakness of the method, in my opinion, is its requirement to get specific benchmarks to optimize the architecture for. While the setting is realistic for certain scenarios where we know exactly what's the type of data we'll encounter, a lot of scenarios require a less specialized approach. With that being said, I'm fine with the paper limiting its scope to the aforementioned subset of scenarios.:

  • With regards to architecture optimization, we do currently use a dev set of at least 20 queries for optimizing the Archon architecture learned (Section 3.3).
  • However, the learned architecture can be quite generalizable. For example, by the development sets of the seven benchmarks explored, we were able to create generalized Archon architectures that preserved 90-95% the performance of specialized Archon architectures (Table 1).
  • This suggests that the learned Archon architectures generalize beyond any individual set of queries and perform well on unseen domains.
  • To test this hypothesis, we ran an additional set of experiments and evaluated the generalized Archon architectures on three new benchmarks: GPQA, MMLU, and MMLU Pro. These benchmarks cover a variety of topics ranging from science to business to the humanities.
  • Note: We did not use any ground truth labels from these datasets for developing Archon architectures.
  • We find that our generalized Archon architectures are capable of preserving 91 to 95% the performance gains of specialized Archon architectures trained exclusively for these benchmarks individually.
  • The generalized ADAS and AFlow architectures only achieve 66% and 73% of their specialized architecture performance, respectively.
  • We highlight these added results in Table 2 of our revised paper: https://storage.googleapis.com/anonymous-files/archon.pdf.

• It's unclear to me how the computation budget constraints are enforced during inference and optimized for during the search:

  • We impose the constraints by excluding any architecture that would exceed the inference call, input token, or output token budgets from the search space:
  • Multiple restrictions could be added. For example, you can filter out architectures with more than 20 inference calls or more than 20,000 input tokens.
  • This prevents our Bayesian optimization algorithm from even considering these invalid architectures in our architecture search (Appendix A.7).
  • To address your concern, we added a new section to discuss this in Section 3.3 called “Adding Search Restrictions”.

• We agree that the name Archon might elicit concepts surrounding model architectures. However, in our application, we use the term “archon” to highlight that we are using an architecture of LMs, rather than an architecture of neurons as in NAS.

• Also, thank you for highlighting potential avenues for improving the Archon construction rules!

  • Given an infinite inference budget, future work can further leverage existing techniques, such as generation and fusion, while even developing new components to better translate additional inference compute to improved performance, such as expanded verification systems.

We welcome any further questions. Thank you again for your feedback and comments! In light of these additional experiments and clarifications to address your comments, we would really appreciate it if you would re-examine our paper and consider raising your score.

Thank you for taking the time to read our paper! We appreciate your feedback and comments.

However, I find that the major analyses/conclusions are not presented in Section 3.:

• We agree that understanding the utilities of inference-time techniques is central to our contributions. We've enhanced the explanation of ablation findings in our revised draft (highlighted green in Sections 3.1, 3.2): https://storage.googleapis.com/anonymous-files/archon.pdf
• Our key findings (Table 5, Figures 4, 7, 8) show the following performance increases:
  • Generation: 10.5% across all tasks when increasing from 1 to 10 generators
  • Ranking: 5.7% for instruction-following, reasoning, and math tasks
  • Fusion: 13.2% across all tasks
  • Critiquing: 8.1% when added before fusion for instruction-following, reasoning, and math
  • Verification / Unit Testing: 5.4% improvement across all tasks
• Combining components leads to compounded gains—ranker+critic+fusion yields 15.1% average improvement across all benchmarks and models, while 10+ generators with verification/unit testing boosts math/coding performance by 12.1% (Tables 5-9).
• These results come from extensive ablation studies across 7 benchmarks and 3 model classes detailed in Appendix A.3 (Tables 18-22).

For efficiency, it seems inappropriate to use the token budgets:

• We used input/output tokens and inference calls for compute-matching Archon because:
  • True FLOPs of closed-source models used by ADAS, AFlow, and frontier LMs can only be estimated
  • FLOPs comparisons between different-sized models can be misleading
  • API providers price by tokens, correlating with their inference FLOP costs
• Overall, Archon is 20.0% more inference-call efficient, using 15.1% less input tokens and 13.5% less output tokens than ADAS/AFlow while delivering 15.1% better performance (Table #1; Figure #5).
• Importantly, Archon uses open-source LMs ≤72B parameters, while competitors use GPT-4o with estimated 200B active parameters: https://epoch.ai/gradient-updates/frontier-language-models-have-become-much-smaller
• To further address your comment, we've added PFLOPs estimates in Table 1 using the estimated active parameters from EpochAI. In unrestricted settings, Archon uses 32.1% fewer FLOPs with models 53% smaller than GPT-4o. In restricted budget settings, Archon achieves equal performance with 38.9% less budget (Figure 6).

My major concern is that the paper heavily relies on the appendix. If accepted, how can you reorganize the paper for better readability while fitting page limits?:

• We appreciate your concerns about appendix reliance. To address your concern, we've:
  • Added more ablation highlights in the main paper to better explain Archon's design decisions (highlighted in green, Sections 3.2, 3.3)
  • Added table of contents for easier navigation of Appendix (A.1)
  • Fixed misordering of Appendix references to corresponding tables/figures
• These changes make the paper more self-contained while still adhering to final conference page limits.

How does such a design achieve optimality? How does it address challenges/limitations of previous works?:

• The three intellectual contributions underlying Archon are:
  • Observing that existing inference techniques can be naturally combined, studying their interactions, and optimizing model selection to maximize technique efficacy
  • Providing an automatic approach for optimal inference technique combinations, yielding significant improvements over SOTA baselines over restricted and unrestricted compute budgets.
  • Archon architectures generalize to unseen benchmarks, outperforming alternative frameworks like ADAS and AFlow
• Point (1) motivated Archon's design through ablation studies across 7 benchmarks and 5 model classes (Sections 3.1, 3.2; Appendix A.3). Existing frameworks can't effectively search the inference design space while balancing cost-performance tradeoffs—please note any comparable baselines we missed.
• Point (2) had never been pursued by previous inference-time papers, allowing us to exceed SOTA LMs and emerging frameworks by +15% accuracy and +30% FLOP efficiency (Table 1; Figures 5-6). While ADAS and AFlow plateaued at 30 PFLOPS per query, Archon continues improving beyond 50 PFLOPs per query.
• For Point (3), we evaluated generalized Archon architectures on three new out-of-domain benchmarks: GPQA, MMLU, and MMLU Pro. Without using any ground truth labels for architecture adaptation, our generalized architectures preserve 91-95% of the performance gains from specialized architectures trained exclusively for these benchmarks (Table 2). In contrast, generalized ADAS and AFlow architectures only achieve 66% and 73% of their specialized performance.

Thank you again for your feedback and comments! In light of these additional experiments and clarifications to address your comments, we would really appreciate it if you would re-examine our paper and consider raising your score.

Thank you for taking the time to read our paper! We appreciate your feedback and comments.

The claim that Utilities of Inference-Time Techniques is not well studied:

• We agree that understanding the utilities of inference-time techniques is central to our contributions. We've enhanced the explanation of ablation findings in our revised draft (highlighted green in Sections 3.1, 3.2): https://storage.googleapis.com/anonymous-files/archon.pdf
• Our key findings (Table 5, Figures 4, 7, 8) show the following performance increases:
  • Generation: 10.5% across all tasks when increasing from 1 to 10 generators
  • Ranking: 5.7% for instruction-following, reasoning, and math tasks
  • Fusion: 13.2% across all tasks
  • Critiquing: 8.1% when added before fusion for all tasks
  • Verification / Unit Testing: 5.4% improvement across all tasks
• Combining components leads to compounded gains—ranker+critic+fusion yields 15.1% average improvement across all benchmarks and models, while 10+ generators with verification/unit testing boosts math/coding performance by 12.1% (Tables 5-9).
• These results come from extensive ablation studies across 7 benchmarks and 3 model classes detailed in Appendix A.3 (Tables 18-22).

Will shifting position for different modules cause performance degradation?:

• Our findings in Appendix A.3 (Tables 5-9) show:
  • Generator components are most effective at architecture start
  • Critic components work best before ranker/fusion components
  • Critic, ranker, and fuser components can be stacked sequentially for improvement
  • Verifier and unit testing components are most effective at architecture end
• These findings informed our "Rules of Construction" (Section 3.2) to prevent performance-degrading placements. We've highlighted these findings in green in the revised paper and added pointers to the ablation study.

Lack of explanation about searched framework. How architecture differs across tasks is not well demonstrated:

• The all-source generalized Archon architecture is included in Figure 3 (Section 3.2). All generalized and specialized architectures from
• Table 1 are described in detail in Appendix A.9 (Figures 12-15) and our supplementary code.
• In Section 4.3 ("Archon by Task"), we discuss task-specific architecture distinctions with these key trends:
  • Instruction-following tasks benefit from additional generator models and deeper fusion layers (17.8% improvement in win-rate from 1 to 10 generators).
  • For reasoning, task-specific architectures show meaningful improvements (10.1% improvement for specialized vs. generalized architectures).
  • For coding, unit testing and increased sample scaling significantly improve performance (44.3% boost in Pass@1).
  • Instruction-following and reasoning architectures use multiple critique-rank-fuse layers with diverse LMs, while math/coding architectures use repeated samples from a single LM before applying unit-testing or verification.

Incremental novelty. This paper is mainly combining different strategies without strong motivation or insights:

• The three intellectual contributions underlying Archon are:
  • Observing that existing inference techniques can be naturally combined, studying their interactions, and optimizing model selection to maximize technique efficacy
  • Providing an automatic approach for optimal inference technique combinations, yielding significant improvements over SOTA baselines over restricted and unrestricted compute budgets.
  • Archon architectures generalize to unseen benchmarks, outperforming alternative frameworks like ADAS and AFlow
• Point (1) motivated Archon's design through ablation studies across 7 benchmarks and 5 model classes (Sections 3.1, 3.2; Appendix A.3). No existing frameworks effectively search this large inference architecture space while balancing cost-performance tradeoffs—please inform us if we've overlooked comparable baselines.
• Point (2) had never been pursued by previous inference-time papers, allowing us to exceed SOTA LMs and emerging frameworks by +15% accuracy and +30% FLOP efficiency (Table 1; Figures 5-6). While ADAS and AFlow plateaued at 30 PFLOPS per query, Archon continues improving beyond 50 PFLOPs per query.
• For Point (3), we evaluated generalized Archon architectures on three new out-of-domain benchmarks: GPQA, MMLU, and MMLU Pro. Without using any ground truth labels for architecture adaptation, our generalized architectures preserve 91-95% of the performance gains from specialized architectures trained exclusively for these benchmarks (Table 2). In contrast, generalized ADAS and AFlow architectures only achieve 66% and 73% of their specialized performance.

We welcome any further questions! Thank you again for your feedback and comments! In light of the additional experiments and clarifications to address your comments, we would really appreciate it if you would re-examine our paper and consider raising your score.

Thank you for taking the time to read our paper! We appreciate your feedback and comments.

However, compute budgets are not always controlled for... controlling only for token budget as in Table 1 and Figure 5 doesn't give a clear picture:

• We appreciate your concerns regarding compute matching experiments. For our comparisons, we originally utilized input/output tokens and inference calls because:
  • We can only speculate on the true FLOPs of the closed-source models used by baseline approaches (i.e. o1 and GPT-4o for ADAS and AFlow)
  • Comparing FLOPs between larger and smaller models can be misleading when evaluating efficiency (e.g. 1 PFLOPs with open-source 70B LMs vs. 1 PFLOPs with closed-source 200B LMs)
  • The API providers, such as OpenAI, Anthropic, Google, Together, etc., price their APIs by input and output tokens per query, correlating directly with their FLOP costs at inference.
• With our measurements, Archon is 20.0% more inference call efficient, using 15.1% less input tokens and 13.5% less output tokens compared to alternative frameworks (ADAS and AFlow), while delivering 15.1% better performance across all benchmarks (Table #1; Figure #5).
• Importantly, Archon achieves these gains using open-source LMs of 72B parameters or less, while ADAS and AFlow use GPT-4o with an estimated 200B active parameters per forward pass (https://epoch.ai/gradient-updates/frontier-language-models-have-become-much-smaller).
• To further address your comment, we've added a column in Table 1 showing PFLOPs per query using estimated active parameters from Epoch AI: https://storage.googleapis.com/anonymous-files/archon.pdf. When benchmarked against ADAS and AFlow in an unrestricted budget setting, Archon architectures use 32.1% less FLOPs while using models that are, on average, 53% smaller than GPT-4o.
• In a restricted budget setting, this gap increases as we scale the allowed budget, allowing Archon to achieve the same performance as ADAS and AFlow with 38.9% less budget across various settings (Figure 6).

The improvements shows in Table 1 are quite predictable and not surprising:

• We'd like to respectfully push back on this claim. Additional inference compute does not always translate to better task performance (Figure 5). Unlike ADAS and AFlow, which plateau at 30 PFLOPs per query using GPT-4o, Archon continues to improve beyond 50 PFLOPs across all task types (Figure 6): https://storage.googleapis.com/anonymous-files/archon.pdf
• Even with unlimited budget for inference-time architectures (Table 1, Figures 5-6), Archon architectures outperform alternatives by +15% accuracy while using 32.1% less FLOPs, 20.0% less inference calls, 15.1% less input tokens, and 13.5% less output tokens.

The contributions are closely related to recent trends in scaling test time compute, although they don't provide significant novelty:

• The three intellectual contributions underlying Archon are:
  • Observing that existing inference techniques can be naturally combined, studying their interactions, and optimizing model selection to maximize technique efficacy
  • Providing an automatic approach for optimal inference technique combinations, yielding significant improvements over SOTA baselines over restricted and unrestricted compute budgets.
  • Archon architectures generalize to unseen benchmarks, outperforming alternative frameworks like ADAS and AFlow
• Point (1) motivated Archon's design through ablation studies across 7 benchmarks and 5 model classes (Sections 3.1, 3.2; Appendix A.3). No existing frameworks effectively search this large inference architecture space while balancing cost-performance tradeoffs—please inform us if we've overlooked comparable baselines.
• Point (2) had never been pursued by previous inference-time papers, allowing us to exceed SOTA LMs and emerging frameworks by +15% accuracy and +30% FLOP efficiency (Table 1; Figures 5-6). While ADAS and AFlow plateaued at 30 PFLOPS per query, Archon continues improving beyond 50 PFLOPs per query.
• For Point (3), we evaluated generalized Archon architectures on three new out-of-domain benchmarks: GPQA, MMLU, and MMLU Pro. Without using any ground truth labels for architecture adaptation, our generalized architectures preserve 91-95% of the performance gains from specialized architectures trained exclusively for these benchmarks (Table 2). In contrast, generalized ADAS and AFlow architectures only achieve 66% and 73% of their specialized performance.

We welcome any further questions. Thank you again for your feedback and comments! In light of these additional experiments and clarifications to address your comments, we would really appreciate it if you would re-examine our paper and consider raising your score.