AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
ABOUT THE ROLE
We are looking for a
Senior QA Engineer
to perform manual testing of web applications and validate AI-generated outputs as part of a product team building LLM-integrated features. You will design and execute test cases across UI, API, and AI behavior scenarios using TestRail, Postman, and JIRA, working within an Agile environment. The role requires a detail-oriented engineer who can evaluate model quality metrics and ensure accuracy, completeness, and alignment across prompt–response workflows.
WHAT YOU WILL DO
- Perform manual testing of web applications to identify bugs, glitches, and inconsistencies;
- Develop, maintain, and execute detailed manual test cases based on product requirements and specifications;
- Perform thorough cross-browser UI testing to ensure consistent behavior across multiple browsers;
- Validate AI-generated test cases, summaries, and recommendations produced by integrated LLMs;
- Design and execute test scenarios to verify AI behavior, including prompt-response consistency and contextual relevance;
- Evaluate AI model quality using defined acceptance metrics such as precision, recall, hallucination rate, and duplication rate;
- Conduct manual API testing to validate request and response accuracy, error handling, and integration;
- Use TestRail to create, manage, and track test plans, test cases, and test execution results;
- Use Jenkins CI/CD pipelines;
- Report and document defects clearly using JIRA;
- Participate in daily stand-ups, sprint planning, and other Agile team meetings;
- Review user stories and acceptance criteria to understand the scope of testing;
- Estimate own tasks and report risks to the team lead;
- Collaborate with developers and other QA team members to reproduce issues and verify fixes;
- Maintain and update test documentation as needed.
MUST HAVES
-
4+ years
of experience with manual testing;
- Basic understanding of methodologies and SDLC and STLC models;
- Knowledge of testing and quality assurance processes;
- Strong experience with test design activities to cover a set of product requirements;
- Knowledge of test design techniques and testing methodologies;
- Defect analysis skills and the ability to write detailed and useful defect reports;
- Experience with Postman for API testing and browser developer tools;
- Experience with UI testing;
- Experience with TestRail as a test management system;
- Basic understanding of CI/CD processes;
- Intermediate English level.
NICE TO HAVES
- Understanding of LLM evaluation principles, including prompt engineering, output validation, and model fine-tuning cycles;
- Familiarity with AI and ML concepts, APIs, and frameworks such as OpenAI API, Azure OpenAI, or LangChain;
- Knowledge of data protection and compliance standards for AI systems, including GDPR, zero-data retention, and PII anonymization;
- Experience with SSO testing;
- Experience in performance or load testing.
PERKS AND BENEFITS
-
Professional growth
: Mentorship, TechTalks, and personalized growth roadmaps.
-
Competitive compensation
: USD-based pay with education, fitness, and team activity budgets.
-
Exciting projects
: Modern solutions with Fortune 500 and top product companies.
-
Flextime
: Flexible schedule with remote and office options.
