Senior Technical Architect (DPO)

Job Summary

Agentic QA Solution Architect
Quality Solution Architect for AI Agents & Agentic Automation
About the Role
AI agents are non-deterministic: the same input can produce different outputs across runs. Testing them requires a fundamentally different discipline than scripted test automation - evaluation frameworks, scoring functions, and statistical judgment instead of exact-match assertions. As an Agentic QA Engineer, you will bring that discipline directly to our customers.
This is a forward-deployed, customer-facing engineering role. You will work side by side with enterprise customers to design, build, test, and harden agentic workflows in their live environments - combining the hands-on coding of a software engineer, the quality mindset of a test automation expert, and the end-to-end design perspective of a solution architect. You will own the question every customer asks before putting an AI agent into production: "How do we know we can trust it?"

Key Responsibilities

What You Will Do
•
Deploy and validate agentic solutions at customers. Work with enterprise customers to implement, integrate, and operationalize AI agents and agentic workflows in production environments - including APIs, data pipelines, and integration with existing customer systems.
•
Design testing agents and evaluation strategies for deterministic and non-deterministic systems. Define what "quality" means for each agentic use case and turn it into measurable signals: evaluation suites, scoring functions, regression baselines, and pass/fail criteria that account for variability in agent behavior.
•
Build and run evaluation frameworks. Implement automated evals (LLM-as-judge, semantic similarity, structured-output checks, human annotation rubrics) that track accuracy, latency, and cost across agent runs, and compare every release against baselines to catch regressions.
•
Perform adversarial and safety testing. Probe agents for failure modes, hallucinations, prompt-injection vulnerabilities, and out-of-scope actions; validate guardrails, human-in-the-loop checkpoints, and escalation paths before go-live.
•
Automate the testing lifecycle. Build and maintain automated test suites for both deterministic components (integrations, workflows, UI) and agentic components, wired into CI/CD so quality gates run on every deployment.
•
Act as a trusted technical advisor. Translate customer requirements into solution and test architecture; explain evaluation results, risks, and trade-offs to audiences ranging from QA engineers to non-technical executives.
•
Architect for scale. Apply solution-architecture judgment to design quality strategies that span the full solution - agents, RPA/workflow/tools automation, integrations, and human-in-the-loop steps - and that customers can operate and extend themselves.
•
Close the loop with product and engineering. Feed field findings - failure patterns, missing capabilities, evaluation gaps - back into the product roadmap and internal best practices, and contribute to reusable test assets and accelerators.

Skill Requirements

What You Bring (Required)
•
Software engineering foundation. 7+ years of hands-on software development experience in Python, C#, VB.NET, or comparable languages, with the ability to write production-quality code and test harnesses.
•
Testing and QA depth. 3+ years in software testing / test automation, including test strategy design, automated test frameworks, and CI/CD-integrated quality gates.
•
Hands-on experience with AI agents. Practical experience building, deploying, or testing LLM-based agents and agentic workflows (e.g., tool/function calling, RAG, multi-step orchestration, frameworks such as LangChain/LangGraph or the Model Context Protocol), ideally in customer or production settings.
•
LLM evaluation literacy. Working understanding of non-determinism, prompting, grounding, and hallucination, and experience setting up evals - including LLM-as-judge approaches, rubric-based scoring, and regression testing of model or prompt changes.
•
Customer-facing experience. Proven track record in a forward-deployed, professional services, implementation, or consulting role: leading technical workshops, managing stakeholder expectations, and delivering under real-world constraints in customer environments.
•
Solution architecture background. Experience designing end-to-end solutions across applications, APIs, and automation platforms - able to see the whole system, not just the component under test.
•
Communication. Excellent written and verbal communication; comfortable presenting technical findings and quality risks to executive stakeholders. Fluent English.

Other Requirements

What Sets You Apart (Preferred)
•
Experience with agentic testing or AI-augmented QA platforms (e.g., UiPath Test Cloud / Test Manager, or comparable AI-powered testing tools) and with automation platforms more broadly (RPA, workflow orchestration).
•
Experience with evaluation and observability tooling for LLM applications (e.g., LangSmith, Braintrust, Promptfoo, or custom pytest-based eval harnesses) and with tracing/monitoring agent behavior in production.
•
Statistical literacy - able to distinguish signal from noise in non-deterministic outputs and reason about sample sizes, baselines, and confidence.
•
Experience with test management and ALM tooling (e.g., ADO, Jira - Xray, TestRail, Test Manager) and requirements-to-test traceability.
•
Knowledge of AI governance, safety, and compliance considerations for deploying agents in regulated industries.
•
Degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
•
Willingness to travel to customer sites as needed.
How Success Is Measured
•
Customers move agentic use cases from pilot to production with documented, repeatable evaluation and test coverage.
•
Evaluation suites catch regressions before customers do - measured by escaped-defect and eval-coverage trends.
•
Customer stakeholders trust and understand the quality evidence behind every go-live decision.
•
Field learnings are converted into reusable test assets, patterns, and product feedback.