Justin Zhao

I'm currently an AI researcher at Meta Superintelligence Labs with a focus on evaluations for trust and safety.

Before Meta, I was at Google Research (2015-2021) where I worked on the first text generation model for the Google Assistant. I left to build an enterprise platform for LLM post-training that eventually got acquired. During some time off 2024-2025, I became interested in LLM judges and published the Language Model Council, a democratic twist on LLM evals.

Outside of tech, I enjoy CrossFit and discovering new food spots. I also play the cello, run a sheet music store, and compete in basketball. I founded two music ensembles: String Theory and Columbia Pops.

LinkedIn Twitter GitHub YouTube Instagram Spotify

Technical Work

March 2026 · EACL
Balanced Accuracy: The Right Metric for Evaluating LLM Judges — Explained through Youden's J statistic
Stephane Collot, Colin Fraser, Justin Zhao, William F. Shen, Timon Willi, Ilias Leontiadis
paper demo
March 2025 · NAACL
Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks
Justin Zhao, Flor Miriam Plaza-del-Arco, Amanda Cercas Curry
paper talk slides demo
Nominated for best paper
May 2024 · Preprint
LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, Devvret Rishi
paper fine-tuning index (archived) talk
December 2023
LLM Distillation Playbook
Justin Zhao, Wael Abid
github talk
600+ GitHub stars
August 2023
How to Use LLMs with Tabular Data Using TabLLM and Predibase
Timothy Wang, Justin Zhao
blog (archived)
December 2022 · EMNLP
CLSE: Corpus of Linguistically Significant Entities
Justin Zhao*, Aleksandr Chuklin*, Mihir Kale
paper data
May 2022
Ludwig AutoML for Text Classification
Anne Holler, Justin Zhao, Avanika Narayan, Travis Addair, Devvret Rishi, Piero Molino
blog (archived)
February 2022
Ludwig AutoML for Deep Learning
Anne Holler, Avanika Narayan, Justin Zhao, Shreya Rajpal, Daniel Treiman, Devvret Rishi, Travis Addair, Piero Molino
blog (archived)
11.7K GitHub stars
July 2021 · Preprint
Using Machine Translation to Localize Task Oriented NLG Output
Scott Roy, Cliff Brunk, Kyu-Young Kim, Justin Zhao, Markus Freitag, Mihir Kale, Gagan Bansal, Sidharth Mudgal, Chris Varano
paper
October 2017 · Talk
Natural Language Generation at Google Research
Justin Zhao, Yufeng Guo
interview
100K+ views