AI Automation Test Lead

We are seeking an AI Automation Test Lead to drive the quality strategy for our clients AI/LLM-powered products. You will build and scale test automation frameworks that validate not just functionality, but also model behavior, data quality, and non-deterministic outputs. This role combines deep test automation expertise with hands-on experience in AI/ML systems, leading a team to ensure our AI features are reliable, safe, and performant in production. Key Responsibilities 1. Test Strategy & Leadership Define and own the end-to-end QA strategy for AI/LLM products including chatbots, agents, RAG pipelines, recommendation, and CV/NLP models Establish best practices for testing non-deterministic systems: prompt evaluation, hallucination detection, bias/safety testing, latency & cost regression Lead, mentor, and grow a team of SDETs/automation engineers. Set quality gates for CI/CD and release readiness Partner with Product, Data Science, and Engineering to shift-left quality and define acceptance criteria for AI features 2. AI Test Automation Architecture Architect automation frameworks for AI systems: prompt regression suites, golden dataset evaluation, synthetic data generation, LLM-as-judge pipelines Build tooling to test RAG quality: context relevance, grounding, citation accuracy, retrieval latency Automate testing of model APIs, vector DBs, embedding pipelines, and fine-tuning workflows Implement eval harnesses using frameworks like DeepEval, RAGAS, LangSmith, Promptfoo, or custom solutions 3. Data & Model Quality Design tests for data pipelines feeding AI: schema validation, drift detection, feature consistency between training/serving Own offline/online eval pipelines. Track metrics: accuracy, faithfulness, toxicity, P50/P95 latency, token cost Build canary & shadow testing for model deployments. Define rollback criteria based on guardrail violations 4. Traditional + AI System Testing Drive API, UI, and integration test automation for services hosting AI models Performance, load, and chaos testing for LLM inference endpoints and real-time features Security testing for prompt injection, jailbreak, data leakage, and PII handling 5. Governance & Reporting Create quality dashboards: model eval trends, defect leakage, flaky rate, coverage for AI scenarios Drive root cause analysis for AI incidents. Feed learnings back into dataset curation and test design Ensure compliance with AI safety, privacy, and regulatory requirements Required Qualifications 8+ years in software QA/test automation, with 2+ years leading teams 3+ years hands-on testing AI/ML systems, LLMs, or data-intensive platforms Strong coding in Python for test framework development. Java/Go is a plus Experience with test automation: Pytest, Playwright, Selenium, REST/GraphQL , CI/CD with GitHub Actions, Jenkins Deep understanding of LLM/RAG concepts : prompts, embeddings, vector DBs, chunking, eval metrics Hands-on with Flink/Spark, SQL, Hive for validating data pipelines Experience with cloud + K8s : AWS/GCP, Docker, Kubernetes, model serving on GPU/CPU Built eval pipelines using LangSmith, Langfuse, Weights & Biases, MLflow , or similar Strong grasp of statistics for A/B testing, significance, and measuring non-deterministic systems Preferred Qualifications Prior experience testing multi-agent systems, tool use, function calling Knowledge of red-teaming, AI safety evals, bias/fairness testing Contributions to open-source AI eval or testing frameworks Experience with Doris, ClickHouse, Elasticsearch, Druid for test data analysis Background in FinTech, E-commerce, or Search domains with real-time requirements