Skip to yearly menu bar Skip to main content


Workshop Fri, Oct 9, 2026 • 8:30 AM – 6:00 PM PDT Continental Ballroom 5

AI Measurement Science: Toward Rigorous AI Evaluation

Sang Truong ⋅ Sanmi Koyejo ⋅ Olawale Salaudeen ⋅ Berivan Isik ⋅ Luke Guerdan
Project Page

Abstract

Chat is not available.

Schedule

Time zone: Conference local time zone
8:30 AM
Opening Remarks
Sang Truong
8:45 AM
Invited Talk: Daniel Kang
Daniel Kang
10:15 AM
Invited Talk: Serena Wang
Serena Wang
10:45 AM
Poster Session 1
→ Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark
Christopher M. Stewart ⋅ Preston Botter ⋅ Natalie Sarabosing ⋅ Muye Zhang ⋅ Rachel Phinnemore ⋅ Shalini Ghosh ⋅ Hong Shen ⋅ Hoda Heidari
→ The Future of Facts: Tracing the Factual Generation-Verification Gap
Tim R Davidson ⋅ Anja Surina ⋅ Caglar Gulcehre
→ LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs
Sanchit Kabra ⋅ Nikhil Abhyankar ⋅ Saaketh Desai ⋅ Prasad Iyer ⋅ Chandan Reddy
→ Towards Context-Aware Cloud Anomaly Detection: Benchmark, Agent, and Analysis
Xinkai Zou ⋅ Xuan Jiang ⋅ Ruikai Huang ⋅ Haoze He ⋅ Parv Kapoor ⋅ Hongrui Wu ⋅ Yibo Wang ⋅ Jian Sha ⋅ Xiongbo Shi ⋅ Zixun Huang ⋅ Jinhua Zhao
→ DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots
Jared Moore ⋅ Andrea Mock ⋅ Yifan Mai ⋅ Jacy R Anthis ⋅ Ryan Louie ⋅ Ashish Mehta ⋅ Kevin Klyman ⋅ Percy Liang ⋅ Nick Haber ⋅ Eric Lin ⋅ Desmond Ong
→ Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models
Vyzantinos Repantis ⋅ Ameya Gawde ⋅ Harshvardhan Singh
→ Doubly Robust Predictive AI Evaluation
Aishwarya Mandyam ⋅ Antoine Maechler ⋅ Sang Truong ⋅ Sanmi Koyejo
→ Do LLMs Encode Event Co-Evolution Beyond Textual Similarity?
Chongrui Ye ⋅ Yuxiang Liu ⋅ Haofei Yu ⋅ Jiayu Liu ⋅ Shanyong Wang ⋅ Jiaxuan You
→ You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass
Yinuo Yang ⋅ Zixian Ma ⋅ Manasi Ganti ⋅ Jieyu Zhang ⋅ Ranjay Krishna
→ API Benchmarks Do Not Reliably Predict Chatbot Behavior
Jennifer Wang ⋅ Joachim Baumann ⋅ Sanmi Koyejo ⋅ Daniel Ho
→ Chasing the Public Score: User Pressure and Evaluation Exploitation in Coding Agent Workflows
Hardy Chen ⋅ Nancy Lau ⋅ Haoqin Tu ⋅ Shuo Yan ⋅ Xiangyan Liu ⋅ Zijun Wang ⋅ Juncheng Wu ⋅ Michael Q Shieh ⋅ Alvaro A Cardenas ⋅ Cihang Xie ⋅ Yuyin Zhou
→ Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset
Mayank Sharma ⋅ Savira D Nadela ⋅ Tyler Matteson
→ From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs
Itay Itzhak ⋅ Eliya Habba ⋅ Gabriel Stanovsky ⋅ Yonatan Belinkov
→ The AI Evaluation Ecosystem
Yash Satish Dave ⋅ Sang Truong ⋅ Serena Wang ⋅ Sanmi Koyejo
→ Evaluation Awareness in Language Models: Mapping Behavior to Mechanism
Bilal Siddiqui ⋅ Marcus Deng ⋅ Alec Saenz ⋅ Jianhang Hong ⋅ Vasu Sharma ⋅ Aditya Shah ⋅ Maheep Chaudhary
→ Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw
Zijun Wang ⋅ Haoqin Tu ⋅ Letian Zhang ⋅ Hardy Chen ⋅ Juncheng Wu ⋅ Xiangyan Liu ⋅ Zhenlong Yuan ⋅ Tianyu Pang ⋅ Michael Q Shieh ⋅ Fengze Liu ⋅ Zeyu Zheng ⋅ Huaxiu Yao ⋅ Yuyin Zhou ⋅ Cihang Xie
→ Where did the ambiguity go? Examining how multimodal models interpret polysemous words
Jasin Cekinmez ⋅ Addison J. Wu ⋅ Raja Marjieh ⋅ Thomas Griffiths
→ Removal Cost as a Measurement Axis for Open-Weight Language Model Safety
Nedim M Sezer ⋅ Ahmet Rıfat Öztürk ⋅ Yağız E Dalar ⋅ Ömer F Aksoy ⋅ Feyzi A Salihoğlu
→ Can Generalist Agents Automate Data Curation?
Feiyang Kang ⋅ Hanze Li ⋅ Adam Nguyen ⋅ Mahavir Dabas ⋅ Jiaqi Ma ⋅ Frederic Sala ⋅ Dawn Song ⋅ Ruoxi Jia
→ CertJudge: Evaluating Lean Formal-Code With Falsifiable Properties
Ethan S Hersch ⋅ Brando Miranda ⋅ Elyas Obbad ⋅ Srivatsava Daruru ⋅ Zhanke Zhou ⋅ Kirill Acharya ⋅ Sanmi Koyejo
12:00 PM
Lunch Break
2:00 PM
Invited Talk: Jacob Steinhardt
Jacob Steinhardt
2:30 PM
Invited Talk: Elham Tabassi
Elham Tabassi
→ Public Benefits Bench: Can AI Help People Navigate SNAP Benefits?
Meghana Kotcherlakota ⋅ Omar Almatov ⋅ Rayan Krishnan
→ Verbalized Confidence Across Autoregressive and Diffusion Language Models
Edward Palacci ⋅ Ian Yoon ⋅ Ryan Xu ⋅ Rohan Balaji ⋅ Tongmeng Xie ⋅ Sean Wu
→ The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations
Emma Yanyang Kong ⋅ JJ Tan ⋅ Ishan Gupta ⋅ Lars Olds ⋅ Velislava P Balin ⋅ Claire Campbell ⋅ Rohan Gosain ⋅ Minsu Jang ⋅ David E Fagnan ⋅ Louis Garcia
→ Spiking the training data to correct for test set contamination
Johnny Wei ⋅ Jerry Li ⋅ Ameya Godbole ⋅ Robin Jia
→ Skill-based Agentic Evaluation for Real-time Data Science Tasks
Aniruddha Tamhane ⋅ Raghav Addanki ⋅ Ayushi Aggarwal ⋅ Aditya Bansal ⋅ Rui Wang ⋅ Charles Menguy ⋅ Swati Jain
→ Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior
Rafal Kocielnik ⋅ Pengrui Han ⋅ Peiyang Song ⋅ Myrl G Marmarelis ⋅ Ramit Debnath ⋅ Dean Mobbs ⋅ Anima Anandkumar ⋅ R. Michael Alvarez
→ Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal
Xiaoxue Yang ⋅ Bozhidar Stevanoski ⋅ Matthieu Meeus ⋅ Yves-Alexandre de Montjoye
→ Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration
Eliya Habba ⋅ Itay Itzhak ⋅ Asaf Yehudai ⋅ Yotam Perlitz ⋅ Elron Bandel ⋅ Michal Shmueli-Scheuer ⋅ Leshem Choshen ⋅ Gabriel Stanovsky
→ Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents
Katrina Drozdov ⋅ Oliver Chen ⋅ Langston Nashold ⋅ Rayan Krishnan
→ Code Migration Benchmark: Evaluating AI Models' Autonomous Code Migration Performance
Dilhan Salgado ⋅ Hung Tran ⋅ Langston Nashold ⋅ Rayan Krishnan
→ Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation
Nikhil R Pottanigari ⋅ Ramin Fahimi ⋅ Noah Bolger ⋅ Sepideh Kharaghani ⋅ Ying Zhang
→ LLM-as-a-Verifier: A General-Purpose Verification Framework
Jacky Kwok ⋅ Shulu Li ⋅ Pranav Atreya ⋅ Yuejiang Liu ⋅ Yixing Jiang ⋅ Chelsea Finn ⋅ Marco Pavone ⋅ Ion Stoica ⋅ Azalia Mirhoseini
4:15 PM
Panel Discussion
Berivan Isik
AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs
Pranay Goel ⋅ Aahana Basappa ⋅ Anusri Karra ⋅ Anish Karra ⋅ Kevin Zhu
Looking Again: Measuring Sycophancy in the Reasoning Chains of Multimodal Models Under Pressure
Mahir N Islam ⋅ Gakuto Okuyama ⋅ Nikolaus Siauw ⋅ Shivank Garg ⋅ Madhur Panwar
Decodability is Not Causality: Dissociating Probe Readouts from Behavioral Drivers via SAE Decomposition
Devesh Tiwari ⋅ Camille Davis ⋅ Shivank Sinha ⋅ Talia Weaver ⋅ Aditya Shah ⋅ Maheep Chaudhary
Measuring the Reliability of Natural Language Autoencoders
Abhinav Boda ⋅ Alexa Tran ⋅ Manouchehr Rimer
LegalScope: Measuring Exam-to-Case Transfer in LLM Legal Reasoning
Hongyu Wang ⋅ Rilyn R Han ⋅ Yilun Zhao ⋅ Xuandong Zhao ⋅ Arman Cohan
When Does Uncertainty Measure RAG Hallucination Risk?
Rajeel Ahmad Ansari ⋅ Tautik Agrahari