Graham Neubig

EditBench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits

Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents

OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety

RefineBench: Evaluating Refinement Capability of Language Models via Checklists

Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering

Prompt-MII: Meta-Learning Instruction Induction for LLMs

VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge

The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think

How can we assess human-agent interactions? Case studies in software agent design

PPTArena: A Benchmark for Computer-Use Agents on PowerPoint Tasks

Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning

Go-Browse: Training Web Agents with Structured Exploration

MIP-Bench: Can LLMs Implicitly Personalize Responses Using Long-Term Memory?

TOM-SWE: User Mental Modeling For Software Engineering Agents

The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

Accumulating Context Changes the Beliefs of Language Models

Shepherd: Pattern-Guided Trajectory Selection for Coding Agents on SWE-Bench

TowerVision : Understanding and Improving Multilinguality in Vision-Language Models

Scaling Evaluation-time Compute with Reasoning Models as Process Evaluators

Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities

Midtraining Bridges Pretraining and Posttraining Distributions

Better Instruction-Following Through Minimum Bayes Risk

Checklists Are Better Than Reward Models For Aligning Language Models

Overtrained Language Models Are Harder to Fine-Tune

Demystifying Long Chain-of-Thought Reasoning

M-Prometheus: A Suite of Open Multilingual LLM Judges

OpenHands: An Open Platform for AI Software Developers as Generalist Agents

Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages

Inducing Programmatic Skills for Agentic Tasks

Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering

Harnessing Webpage UIs for Text-Rich Visual Understanding

Repetition Improves Language Model Embeddings

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios

RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems

Agent Workflow Memory

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems

Training Task Experts through Retrieval Based Distillation

Training Software Engineering Agents and Verifiers with SWE-Gym

Agent Workflow Memory

Beyond Browsing: API-Based Web Agents