SmartTest: Rebuilding the Path to Product Value
Reworked the prediction approach so customers could evaluate SmartTest from their first test runs.
- AI Product Strategy
- Time to Value
- Enterprise SaaS
- Sales Enablement
Samyak Rai
AI Product Manager at CloudBees
From their time as
AI Product Manager
CloudBees • 2025
Overview
Samyak rebuilt SmartTest's evaluation path around immediate prediction value. The next enterprise deal closed in 40 to 45 days, down from sales cycles of roughly 90 to 100 days.
The Story
Samyak took end-to-end ownership of SmartTest, a CloudBees product that helps enterprise QA and development teams shorten test execution and developer feedback cycles. Predictive test selection already produced strong results for existing users, while new evaluations needed about 30 days of training data before prospects could see prediction quality.
He framed the product challenge around reducing time to value and creating a repeatable sales motion. The existing training-dependent ML model made that difficult during an enterprise evaluation.
Samyak and the team replaced that model within the sales-cycle experience with an approach using OpenAI embeddings, vector similarity, and cosine similarity. This preserved an AI-based relevance layer while removing the learning period that had delayed demonstrations of value.
Customers could see prediction quality from day one and test the product without a month-long wait. The next sales-heavy deal closed in about 40 to 45 days, compared with prior cycles of roughly 90 to 100 days, and sellers had a more practical motion to take to prospects.
Ownership Snapshot
Broad role
Product manager with end-to-end ownership of SmartTest.
Goal
Create a repeatable evaluation and sales motion for the product.
Direct ownership
Owned product strategy, prediction experience, analytics, monitoring, and evaluation flow.
Team execution
Worked with engineering, AI, sales, and customers to redesign and validate the approach.