Work
Selected systems, integrations, and educational work.
I work where applied AI meets developer infrastructure: building systems, defining how to evaluate them, and turning what works into tools and explanations other people can use.
Current work
Agent engineering
Leading work on a background coding agent and the infrastructure around it: internal harness benchmarks, memory, and systems that help agents learn from prior experience. The work combines product engineering with empirical evaluation—making progress measurable rather than anecdotal.AgentsEvaluationLeadership
Leading work on a background coding agent and the infrastructure around it: internal harness benchmarks, memory, and systems that help agents learn from prior experience. The work combines product engineering with empirical evaluation—making progress measurable rather than anecdotal.AgentsEvaluationLeadership
NVIDIA NeMo Agent Toolkit × W&B Weave
Worked on evaluation and observability support for agentic workflows built with NVIDIA’s framework-agnostic toolkit, making it easier to trace, compare, and improve multi-agent systems.Technical reportAgentsObservability
Worked on evaluation and observability support for agentic workflows built with NVIDIA’s framework-agnostic toolkit, making it easier to trace, compare, and improve multi-agent systems.Technical reportAgentsObservability
LLM systems and evaluation
Wandbot: an evaluation-driven refactor
Reworked W&B’s open-source documentation assistant around systematic evaluation. Answer accuracy rose from 72% to 81%, latency fell by 84%, and a full evaluation run went from roughly two hours to under ten minutes. Nearly 50 experiments made the trade-offs between quality, speed, and cost visible.Read the case study · SourceRAGEvaluationProduction
Reworked W&B’s open-source documentation assistant around systematic evaluation. Answer accuracy rose from 72% to 81%, latency fell by 84%, and a full evaluation run went from roughly two hours to under ten minutes. Nearly 50 experiments made the trade-offs between quality, speed, and cost visible.Read the case study · SourceRAGEvaluationProduction
Evaluating, comparing, and optimizing LLM systems
Built a practical workflow for comparing system configurations with human labels, LLM judges, retrieval metrics, and parameter search. This work became a foundation for later production systems and courses.Technical reportLLMOpsEvals
Built a practical workflow for comparing system configurations with human labels, LLM judges, retrieval metrics, and parameter search. This work became a foundation for later production systems and courses.Technical reportLLMOpsEvals
LlamaIndex × W&B
Built an integration for inspecting retrieval and LLM traces, debugging complex query pipelines, and versioning indexes and embeddings. The accompanying walkthrough showed how those primitives support evaluation-driven RAG development.WalkthroughLlamaIndexTracing
Built an integration for inspecting retrieval and LLM traces, debugging complex query pipelines, and versioning indexes and embeddings. The accompanying walkthrough showed how those primitives support evaluation-driven RAG development.WalkthroughLlamaIndexTracing
Tools and integrations
My public GitHub history includes 200 merged pull requests. Much of that work made experiment tracking, model evaluation, and application tracing feel native inside the tools developers already used.
A new generation of W&B callbacks for Keras
Added dedicated callbacks for metrics logging, model checkpointing, and rich model-prediction tables, alongside fixes for SavedModel and distributed-training workflows.Metrics logger · Model checkpointing · Evaluation tables · Distributed trainingKerasTensorFlow
Added dedicated callbacks for metrics logging, model checkpointing, and rich model-prediction tables, alongside fixes for SavedModel and distributed-training workflows.Metrics logger · Model checkpointing · Evaluation tables · Distributed trainingKerasTensorFlow
Native experiment-tracking hooks across the OpenMMLab stack
Built dedicated W&B integrations for object detection, image classification, and semantic segmentation, then followed them with maintenance fixes and runnable examples.MMDetection · MMClassification · MMSegmentationComputer visionOpenMMLab
Built dedicated W&B integrations for object detection, image classification, and semantic segmentation, then followed them with maintenance fixes and runnable examples.MMDetection · MMClassification · MMSegmentationComputer visionOpenMMLab
Instrumentation and evaluation for emerging LLM stacks
Added W&B instrumentation to LlamaIndex and integration support to EleutherAI’s LM Evaluation Harness; updated OpenAI fine-tuning, Cohere v2, and Wandbot evaluation workflows as their underlying APIs evolved.LlamaIndex · LM Evaluation Harness · OpenAI · Cohere · WandbotLLMOpsEvaluation
Added W&B instrumentation to LlamaIndex and integration support to EleutherAI’s LM Evaluation Harness; updated OpenAI fine-tuning, Cohere v2, and Wandbot evaluation workflows as their underlying APIs evolved.LlamaIndex · LM Evaluation Harness · OpenAI · Cohere · WandbotLLMOpsEvaluation
Tracing and evaluation across agent frameworks
Extended support for DSPy, MCP, LlamaIndex, CrewAI, Verifiers, and NVIDIA NeMo Agent Toolkit, including fine-grained tracing and evaluation inside external framework repositories.NeMo tracing · NeMo evaluation · MCP · DSPy · Verifiers · CrewAIAgentsTracingEvals
Extended support for DSPy, MCP, LlamaIndex, CrewAI, Verifiers, and NVIDIA NeMo Agent Toolkit, including fine-grained tracing and evaluation inside external framework repositories.NeMo tracing · NeMo evaluation · MCP · DSPy · Verifiers · CrewAIAgentsTracingEvals
I also helped build W&B’s Kaggle Ambassador program, connecting practitioners, reusable notebooks, and experiment-tracking education. Along the way I became a Kaggle Notebooks Master, with notebooks spanning object detection, medical imaging, reproducibility, and interactive visualization.
Research publication
Feature extraction and classification of phonocardiograms using convolutional neural networks
With Devjyoti Chakraborty, Snehangshu Bhattacharya, Aritra Roy Gosthipaty, and Chira Datta. We studied an automated approach to classifying normal and abnormal heart sounds: transforming phonocardiogram recordings into spectrograms, addressing class imbalance, and using deep convolutional neural networks for feature extraction and classification.Read on IEEE Xplore · Google Scholar profileMedical MLSignal processingCNN
With Devjyoti Chakraborty, Snehangshu Bhattacharya, Aritra Roy Gosthipaty, and Chira Datta. We studied an automated approach to classifying normal and abnormal heart sounds: transforming phonocardiogram recordings into spectrograms, addressing class imbalance, and using deep convolutional neural networks for feature extraction and classification.Read on IEEE Xplore · Google Scholar profileMedical MLSignal processingCNN
Courses
LLM apps: Evaluation
Created with Anish Shah, with case studies from Paige Bailey and Graham Neubig. Covers programmatic evaluators, LLM judges, evaluator alignment, and real-world evaluation workflows.Take the course
Created with Anish Shah, with case studies from Paige Bailey and Graham Neubig. Covers programmatic evaluators, LLM judges, evaluator alignment, and real-world evaluation workflows.Take the course
RAG++: From POC to Production
Created with Bharat Ramanathan, with contributions from Meor Amer and Charles Pierse. Distills nearly two years of production RAG work into evaluation, data ingestion, query enhancement, hybrid retrieval, reranking, agentic RAG, and response synthesis.Take the course
Created with Bharat Ramanathan, with contributions from Meor Amer and Charles Pierse. Distills nearly two years of production RAG work into evaluation, data ingestion, query enhancement, hybrid retrieval, reranking, agentic RAG, and response synthesis.Take the course
Training and Fine-tuning LLMs
Co-taught with Darek Kłeczek and guest instructors Jonathan Frankle, Weiwei Yang, and Mark Saroufim. Covers model foundations, evaluation, data preparation, training, and fine-tuning.Take the course
Co-taught with Darek Kłeczek and guest instructors Jonathan Frankle, Weiwei Yang, and Mark Saroufim. Covers model foundations, evaluation, data preparation, training, and fine-tuning.Take the course
Earlier work
Faceless
Built a multitask face-analysis pipeline with TensorFlow and OpenCV for face detection and attribute prediction. It was an early bridge between independent ML projects, technical writing, and my work at W&B.TensorFlowOpenCV
Built a multitask face-analysis pipeline with TensorFlow and OpenCV for face detection and attribute prediction. It was an early bridge between independent ML projects, technical writing, and my work at W&B.TensorFlowOpenCV
American Sign Language classifier
Built and documented an end-to-end CNN project for recognizing ASL letters—one of my first attempts to explain a complete computer-vision system in public.Read the articleCNNWriting
Built and documented an end-to-end CNN project for recognizing ASL letters—one of my first attempts to explain a complete computer-vision system in public.Read the articleCNNWriting
Autonomous robots
Built a progression of line-following, maze-solving, object-detection, and grid-navigation robots. One system avoided obstacles, explored a grid, and retraced the shortest route. These projects gave me the systems instinct I later brought to ML infrastructure and agent engineering.Embedded systemsNavigation
Built a progression of line-following, maze-solving, object-detection, and grid-navigation robots. One system avoided obstacles, explored a grid, and retraced the shortest route. These projects gave me the systems instinct I later brought to ML infrastructure and agent engineering.Embedded systemsNavigation