Senior AI Application Test Engineer (m/f/d)

We are seeking a highly experienced Senior AI Application Test Engineer to join our team. This role requires deep expertise in testing AI-powered applications, particularly conversational AI and RAG (Retrieval-Augmented Generation) based systems. You will be responsible for ensuring the quality, scalability, and reliability of our AI solutions across the entire application lifecycle.Key ResponsibilitiesDesign and execute comprehensive test strategies for AI/ML applications, with focus on conversational AI and RAG-based systemsLead end-to-end testing of AI applications from development through production deployment and scalingDevelop and implement testing frameworks for evaluating LLM outputs, retrieval accuracy, and response qualityTest AI model performance, accuracy, bias, hallucinations, and edge casesValidate data pipelines, vector databases, and embedding quality in RAG architecturesConduct scalability and performance testing for AI applications handling large-scale dataEstablish quality metrics and KPIs specific to AI application testingCollaborate with data scientists, ML engineers, and developers to define testable requirementsMentor junior QA team members on AI testing methodologiesRequired Technical SkillsAI/ML Testing: 5+ years of hands‑on experience testing AI applications, conversational AI systems, chatbots, and RAG-based solutionsAzure Expertise: Deep knowledge of Azure AI services including Azure OpenAI, Azure ML, Cognitive Services, Azure AI Search, and related componentsTesting Frameworks: Proficiency in automation testing tools and frameworks (Selenium, Playwright, PyTest, etc.)Programming: Strong coding skills in Python; experience with test automation scriptingAI Testing Tools: Experience with LLM evaluation frameworks, prompt testing, and AI‑specific quality metricsData Testing: Knowledge of testing data pipelines, ETL processes, and vector databasesPerformance Testing: Experience with load testing AI endpoints and monitoring model inference performanceCI/CD: Familiarity with Azure DevOps, GitHub Actions, and automated testing pipelinesPreferred QualificationsExperience testing Azure OpenAI implementations and custom GPT applicationsKnowledge of MLOps practices and model monitoringUnderstanding of NLP, embeddings, and semantic search conceptsFamiliarity with prompt engineering and LLM best practicesExperience in Agile/Scrum methodologiesPrevious work in enterprise‑scale AI deploymentsSoft SkillsExcellent analytical and problem‑solving abilitiesStrong communication and collaboration skillsAbility to work in fast‑paced, innovative environmentsDetail‑oriented with a quality‑first mindsetSelf‑motivated and proactive in identifying testing gapsHalian GroupWith over 28 years of experience, we have come to understand that innovation is the only way to provide agile, practical solutions that transform businesses and careers. Our resourcing and smart services help you to realize tomorrow’s potential. Discover the amazing things possible when you bring the right people and the right technologies together. At Halian, we recognize that diversity, equity, and inclusion (DEI) are essential to building high‑performing teams for our clients. We are committed to connecting organizations with top talent from all backgrounds, ensuring that every individual feels valued, respected, and empowered to contribute their unique perspectives. We encourage applications from all qualified candidates, regardless of race, gender, disability, or any other characteristic that makes them unique. By fostering diverse and inclusive workplaces, we help our clients drive innovation, enhance collaboration, and better reflect the communities they serve.#LI-CC1#J-18808-Ljbffr