Airbnb is seeking a Senior Software Engineer for the AI Assistant Product Evaluation team. You will design and build scalable data systems and evaluation platforms to support Airbnb’s agentic AI products, working at the intersection of backend engineering, data systems, AI evaluation, and product quality. You will help define how Airbnb measures, monitors, and improves AI quality at scale by turning model outputs, human feedback, and evaluation results into reliable signals for product and model development. You will partner closely with modeling teams, product managers, data scientists, and operations to translate ambiguous AI evaluation needs into robust systems that enable trustworthy AI experiences.
Airbnb is seeking a Senior Software Engineer for the AI Assistant Product Evaluation team. You will design and build scalable data systems and evaluation platforms to support Airbnb’s agentic AI products, working at the intersection of backend engineering, data systems, AI evaluation, and product quality. You will help define how Airbnb measures, monitors, and improves AI quality at scale by turning model outputs, human feedback, and evaluation results into reliable signals for product and model development. You will partner closely with modeling teams, product managers, data scientists, and operations to translate ambiguous AI evaluation needs into robust systems that enable trustworthy AI experiences.
A Typical Day: - Design and productionize scalable data systems that support AI evaluation, metric computation, observability, and feedback loops for agentic AI products. - Build data models, schemas, and processing pipelines for agentic AI interactions, enabling reliable logging, retrieval, metric computation, and long-term evaluation dataset management. - Work with Core Modeling engineers to address pain points in the LLM evaluation process, developing LLM-as-a-judge solutions and data pipelines to meet metric challenges at scale. - Collaborate with machine learning infrastructure teams to evolve evaluation frameworks for Airbnb’s Conversational AI products. - Lead all phases of software development, including architecture design, implementation, and testing. - Collaborate with product managers, operations, and data scientists to identify opportunities for business impact
Track similar jobs
Get email alerts when new roles like this are posted.
Based on: Senior Software Engineer
See how this role matches your resume
Sign in to get an AI match score and personalized feed.
A Typical Day: - Design and productionize scalable data systems that support AI evaluation, metric computation, observability, and feedback loops for agentic AI products. - Build data models, schemas, and processing pipelines for agentic AI interactions, enabling reliable logging, retrieval, metric computation, and long-term evaluation dataset management. - Work with Core Modeling engineers to address pain points in the LLM evaluation process, developing LLM-as-a-judge solutions and data pipelines to meet metric challenges at scale. - Collaborate with machine learning infrastructure teams to evolve evaluation frameworks for Airbnb’s Conversational AI products. - Lead all phases of software development, including architecture design, implementation, and testing. - Collaborate with product managers, operations, and data scientists to identify opportunities for business impact
Track similar jobs
Get email alerts when new roles like this are posted.
Based on: Senior Software Engineer
See how this role matches your resume
Sign in to get an AI match score and personalized feed.
© 2026 JobMatcher. All rights reserved.
© 2026 JobMatcher. All rights reserved.
Backend Software Engineer I
Tebra
Supply Account Manager
Life360
Cloud Infrastructure Engineer
RefinedScience
Software Engineer
LIV
Copy Editor
Guardian US
Inside Sales Representative - Account Management
The Duckhorn Portfolio
Backend Software Engineer I
Tebra
Supply Account Manager
Life360
Cloud Infrastructure Engineer
RefinedScience
Software Engineer
LIV
Copy Editor
Guardian US
Inside Sales Representative - Account Management
The Duckhorn Portfolio
