Zhengfu He

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition

Evolution of Concepts in Language Model Pre-Training

Attention Layers Add Into Low-Dimensional Residual Subspaces

Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures

Automatically Identifying and Interpreting Sparse Circuits with Hierarchical Tracing