Taishi Nakamura

MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive Text Sources

Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks

Rewriting Pre-Training Data Boosts LLM Performance in Math and Code

Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search

Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization

Agent Skill Acquisition for Large Language Models via CycleQD