Rio Yokota

Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks

Rewriting Pre-Training Data Boosts LLM Performance in Math and Code

Improving LoRA with Variational Learning

Vertical Attention: Automatic Exploration of Inter-Layer Connections in Transformer-based Language Models

Variational Learning Finds Flatter Solutions at the Edge of Stability

Masked Gated Linear Unit

Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation

Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization