SpotDraft Eval Harness
Built an evaluation framework from scratch that surfaced real customer failures in a multi-agent system and drove systematic prompt and tool improvements.
Pranav Pandey
Senior Applied AI Engineer at SpotDraft
From their time as
Senior Applied AI Engineer
SpotDraft • 2025
Overview
When SpotDraft's multi-agent AI system was in production, there was no systematic way to track where it was failing or to prevent regressions as the team made changes. Pranav built the evaluation harness from scratch to solve two distinct problems: making it easy for any product team to define and run evals without writing boilerplate, and surfacing the specific failure points in the multi-agent system from real customer usage.
The Story
When SpotDraft's multi-agent AI system was in production, there was no systematic way to track where it was failing or to prevent regressions as the team made changes. Pranav built the evaluation harness from scratch to solve two distinct problems: making it easy for any product team to define and run evals without writing boilerplate, and surfacing the specific failure points in the multi-agent system from real customer usage.
He designed the framework so teams could define a task and expected outcome and run evaluations with minimal setup. For the multi-agent system, he worked with in-house reviewers to curate a dataset of real customer queries, including cases flagged by customers and cases identified internally where tool usage was incorrect or the model hallucinated. Each failure was added to the eval dataset and tracked over time.
As the team made changes to prompts, tools, and agents, the dataset served as the regression baseline. Pranav also built an automated prompt optimization loop on top of the framework, which identified failing cases and iterated on prompts automatically to address them. The result was a closed loop from failure identification to fix to validation.