Apertus Engineer: Evaluations

Join Our Evaluation Team

We are looking for a talented engineer to engage in our evaluation efforts, contributing to the development and maintenance of the evaluation codebase and pipelines that guide our training and release processes. This role necessitates strong Python engineering skills, practical experience in large language model (LLM) evaluation, and the ability to thrive in a collaborative research setting.

Join a team that is at the forefront of AI development, training open foundation models with extensive parameters on Europe's largest AI-ready supercomputers. You will be part of a dedicated team of engineers and researchers from prestigious institutions like EPFL and ETH Zürich, working to create multilingual, multimodal AI models that are fully open and responsibly trained.

Key Responsibilities

• Build and maintain the evaluation codebase and pipelines, ensuring consistent results from training through to deployment.
• Implement evaluations that run efficiently and at scale on advanced infrastructure, utilizing parallel processing and effective caching techniques.
• Identify and resolve discrepancies between evaluations during training and serving phases.
• Integrate various types of evaluations, including text, image, and audio, into cohesive pipelines.
• Provide comprehensive evaluation results, reports, and dashboards that aid in training and release decisions.
• Collaborate closely with engineers focused on safety, deployment, and community needs to integrate evaluations they develop.

Essential Qualifications

• Master’s or Doctorate in Computer Science, Data Science, Artificial Intelligence, Machine Learning, or related fields. Outstanding Bachelor’s degree candidates with significant engineering experience will also be considered.
• Proficient in Python and software engineering, with experience in building robust data or evaluation pipelines.
• Familiarity with LLM evaluation frameworks and custom benchmarking tools.
• Strong teamwork and communication capabilities, with a history of working collaboratively across research and engineering disciplines.
• Practical experience in relevant domains, ideally through direct projects or studies.
• Adaptability to shifting priorities, tools, and daily tasks driven by training schedules and rapid advancements in the field.
• Experience executing evaluations on GPU clusters and with inference engines.

Preferred Qualifications

• A keen understanding of statistical rigor and variance in evaluations.
• Published research or engagement with recent studies relevant to this role.
• Experience with LLM-as-judge pipelines and their calibration.
• Knowledge of benchmark contamination practices and effective data visualization techniques.

What We Offer

• A stimulating academic environment located at leading technical universities.
• Access to top-tier supercomputing resources.
• Collaborative opportunities with leading researchers in the field.
• Attractive employment conditions and comprehensive benefits, including pension plans.
• Flexible work arrangements with remote work options.
• Support for professional development, including conference participation and specialized training.
• The chance to participate in impactful open-source projects.
• Engagement in initiatives of national significance within Switzerland’s AI development sector.

This role can be based in either Lausanne at EPFL or Zürich at ETH Zürich. We look forward to welcoming dedicated professionals who are eager to make a meaningful contribution to the field of AI.