LLM-as-a-Judge: Evaluating with language models is more nuanced than it seems

Using a language model to evaluate the output of another language model is no longer experimental practice—it's the dominant method in production evaluation pipelines. The problem is that many teams adopting it don't fully understand its structural biases or how complications arise when multimodal inputs enter the picture.

This week, an article by Yingzhe Lan circulated on Hacker News offering a systematic introduction to the topic: Introduction to (Multimodal) LLM-as-a-Judge. It's worth pausing on because, while presented as an introduction, it captures with precision several problems that academic literature has been discussing for months and that product teams typically ignore until the system fails.

What exactly is LLM-as-a-Judge

The pattern is straightforward: instead of paying human evaluators or relying on classic automated metrics like BLEU or ROUGE, you ask a model, the judge, to score or compare responses generated by another model. In practice, this is used to evaluate chatbot response quality, align models via RLHF, build comparative leaderboards, and validate outputs in agentic pipelines.

The obvious advantage is cost and scale: an LLM judge can process thousands of pairs in minutes. The less obvious disadvantage is that the judge inherits biases from the base model and adds its own from the evaluation prompt.

The biases that matter most in practice

The article identifies several well-documented bias patterns. The most well-known is position bias: when presented with two responses to compare, the judge tends to favor the first or last depending on the model. Studies show variations of up to 15 percentage points just from reordering options.

Another relevant bias is verbosity bias: LLM judges typically score longer responses higher even when they're not more correct or useful. This is especially problematic when the goal is to train more concise models.

The third, less discussed, is familiarity bias: a model tends to score higher responses that resemble its own generation style. If you use Claude Opus 4.8 as a judge to evaluate responses generated by the same model, correlation with human preferences can be artificially high and fail to generalize to other models.

The multimodal layer adds real complexity

When inputs include images, graphics, or video, the problem amplifies. The article notes that multimodal judges have additional difficulty calibrating whether a response accurately describes an image or whether a technical explanation of a diagram is accurate. The judge model's visual perception errors blur with the evaluated model's reasoning errors, and separating both error sources is non-trivial.

This matters for any team building evaluation systems for vision applications: document analysis, code assistants with screenshots, or visual content moderation pipelines. Using a multimodal judge without validating its own visual perception error rate is adding noise to your evaluation signal without realizing it.

Who should read this

The article is written for technical profiles who already know LLM fundamentals but haven't gone deep into evaluation methodology. It doesn't require familiarity with specific papers, though it references some. It's especially useful for:

• Engineers building automated evaluation pipelines in production.
• Teams using Claude Code with sub-agents who need to validate output quality between system layers.
• Applied researchers wanting a mental map before diving into more technical literature on reward models and preference learning.

What's left out

The article doesn't detail how to calibrate LLM judges against human annotations, nor techniques like Constitutional AI scoring or structured rubrics for reducing variance. It also doesn't address cost management when the judge is a frontier model like Claude Opus 4.8 and you're evaluating millions of pairs. These are reasonable gaps for an introduction, but worth keeping in mind if you're seeking direct implementation.

---

From our perspective, the LLM-as-a-Judge pattern seems useful and probably inevitable at scale, but the level of naivety with which many teams deploy it remains concerning. An introduction like this doesn't solve the problem, but at least it names the risks properly.