AI QA Tester

RemoteHace 2 meses

Tecnología
Tiempo completo
Remoto
Semi-senior
Entrevista IA

Descripción

We are looking for an AI QA Engineer to join one of our projects and help ensure the quality, reliability, and safety of advanced AI systems. In this role, you will test and validate LLM-based applications, agentic workflows, and AI-powered systems, working across software, infrastructure, and hardware components.

You will play a critical role in identifying issues, validating model behavior, and building automated testing frameworks to support the deployment of high-quality, production-ready AI solutions.

Requisitos

Experience
3–5+ years of experience in QA engineering
1–2+ years testing AI/ML or LLM-based systems
Experience delivering QA validation for at least one production AI system (chatbot, agentic workflow, RAG pipeline, or similar)
Hands-on experience with prompt engineering and generative AI output evaluation
Technical Skills
Strong proficiency in Python for test scripting and automation
Experience with LLM evaluation tools such as: OpenAI Evals, RAG evaluation frameworks, Weights & Biases
Familiarity with agent frameworks (LangChain, LangGraph, CrewAI, AutoGen) to trace and test agent behavior
Experience with API testing
Understanding of NLP concepts such as intent recognition and entity extraction
Familiarity with vector databases and RAG architectures for retrieval accuracy testing
Experience with version control systems (Git) and issue tracking tools (Jira, Linear)
Preferred Qualifications
Experience in AI safety testing, red-teaming, or adversarial testing
Skills in evaluation rubric design, bias and fairness auditing, and grounding verification
Experience testing AI systems in regulated industries (healthcare, fintech, legal)
Background in NLP, computational linguistics, or data science
Familiarity with RLHF (Reinforcement Learning from Human Feedback) and model alignment concepts
Security certifications such as OSCP or OSWA
Coursework or experience related to AI safety

Habilidades requeridas

PythonPyTestOpenAI EvalsWeights & BiasesLangChainGitJiraAPI TestingNLP

AI QA Tester