Research Engineer, Benchmarks

About the Role

Join a small, technical team working on AI evaluation and reinforcement learning infrastructure. You will design and build rigorous benchmarks for assessing AI agents on realistic, domain-specific workflows, helping research and engineering teams understand how well agents perform in practice.

What You'll Do

• Design, implement, and maintain benchmarks for evaluating AI agents on domain-specific tasks.

• Work with subject-matter experts to translate real workflows into practical tasks and evaluation criteria.

• Build reliable infrastructure to run models and agents against benchmark tasks at scale.

• Develop metrics and analyses to assess benchmark difficulty, reliability, and failure modes.

• Validate benchmark results against real-world evaluations and user needs.

• Write clear technical documentation and reports that explain methods and results.

What We're Looking For

• Two to four years of experience in software engineering, machine learning engineering, or research, including at least two years building AI benchmarks, evaluation infrastructure, or agent environments.

• Hands-on experience designing and running evaluations or benchmark environments for AI agents or large language models.

• Strong proficiency with Python, Docker, and Linux, including building and operating reliable infrastructure.

• Experience developing metrics, statistical analyses, or validation studies for evaluation systems.

• Experience collaborating with subject-matter experts and understanding workflows across technical or business domains.

• Strong technical writing, careful attention to detail, and the ability to work independently in an early-stage, fast-moving environment.

• Research publications or technical writing on benchmarking, model evaluation, or failure modes are beneficial. Experience with reinforcement learning pipelines or public benchmark projects is also useful.

Compensation & Benefits

Base salary: $100,000 to $170,000 USD annually. Visa sponsorship is available.

Location

On-site in Singapore, Singapore.