LLM Evaluation specialist

Job Description Job Title: LLM Evaluation Specialist / AI Quality Analyst (NLP, RAG, Generative AI) Location: Bangalore / Pune / Gurgaon Job Summary: We are hiring an experienced LLM Evaluation Specialist to work on evaluation and optimization of Generative AI / LLM-based systems . The role involves improving model accuracy, reducing hallucinations, and enhancing response quality in real-world applications. Key Responsibilities: Design and execute LLM evaluation frameworks (offline & online) Define and track metrics like: Faithfulness, Groundedness, Relevance Precision@K, Recall@K Identify issues such as: Hallucination Retrieval failure Response inconsistency Create test datasets, adversarial prompts, and evaluation benchmarks Work on LLM-as-judge and automated evaluation techniques Analyze user feedback and improve model performance Collaborate with product, engineering, and AI teams Required Skills: 69 years experience in AI / Machine Learning / NLP Strong knowledge of: LLMs, Prompt Engineering, RAG systems Hands-on experience with tools like: Ragas / DeepEval / TruLens (or similar) Strong analytical and problem-solving skills Experience in AI model evaluation or deployment Qualification: Bachelors / Masters in Computer Science, AI, Data Science or related field Preferred: Experience in AI copilots / conversational AI Domain exposure in airline / transport / enterprise systems Salary: Best in Industry How to Apply: Interested candidates can share their updated CV at: [ hidden_email ] OR Whatsapp on hidden_mobile