Building an Evaluated Memory System for an AI Assistant
Expanded a Python memory service and built its dedicated evaluation pipeline.
Karan Mungra
Founding Engineer, profound.me
From their time as
SDE
Krutrim • Jan 2024 – Mar 2026
Overview
I built a Python memory system that grew from tracking simple facts into maintaining richer user context. I adapted the design to sequentially store memory documents in a vector database and created a dedicated evaluation pipeline using about 1,000 production queries.
The Story
The initial requirement was narrow: retain enough information to stop the assistant from asking the same questions again. I built that memory component in Python.
The requirement later expanded toward a broader user persona. I evolved the design into sequential memory documents stored in a vector database. The newer behavior extended the original structure, so most of the system remained useful while the generation logic changed.
The organization already had evaluation infrastructure for other components, but the memory system needed its own setup. I designed an evaluation pipeline around about 1,000 real production queries. For each case, we had an expected view of the memory that should be generated and the memory that should later be served.
The evaluation checked whether the system captured the right information and retrieved the appropriate memory. That gave us a repeatable way to test changes as the product requirement moved from simple recall toward richer user context.
Ownership Snapshot
Broad role
Built the memory component and its evaluation process for an AI assistant.
Goal
Help the assistant retain useful user context and avoid repeated questions.
Direct ownership
Owned the Python implementation, evolving memory design, and dedicated evaluation pipeline.
Team execution
Used the organization's broader evaluation infrastructure while adapting it for the memory system.
Topics
- AI Agents
- Memory Systems
- Python
- Vector Databases
- Evaluation
Talk to Karan about this story
Send a message