Shiwei Liu

Diffusion Language Model Knows the Answer Before It Decodes

GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching

Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution

When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs

Layerwise Learning Rate in the Era of Large Language Models

Neural Sum-of-Squares: Certifying the Nonnegativity of Polynomials with Transformers

Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing

Rewiring Experts on the Fly: Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert models

LOST: Low-rank and Sparse Pre-training for Large Language Models

Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning

Stable-SPAM: How to Stably Train Large Language Models in 4-Bit

Training with Dynamic Sparse Heads as the Key to Effective Ensembling

ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity

Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More

GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling

The Curse of Depth in Large Language Models

AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs

LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning

Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications

SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training

Composable Interventions for Language Models

Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning

OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning

From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients