Deepgram is seeking a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech, audio, and multilingual models before release. You will define evaluation methodologies and build automated evaluation pipelines for batch and streaming workloads, including metrics such as WER, latency, time-to-first-byte, and anomaly detection to guard against regressions. The role involves developing scalable, reproducible evaluation infrastructure—canaries, orchestration, and result-aggregation pipelines—that run against production models and, when needed, large GPU clusters. You will translate Research benchmarks into automated pass/fail gates and embed evaluation and quality checks into CI/CD so quality is verified continuously. You’ll collaborate with Research, model training, inference, and product teams to provide trusted evaluation signals that inform release and optimization decisions, and work with DevOps/Infra to establish ephemeral test environments and reliable results-aggregation infrastructure. The ideal candidate will have 5+ years of professional software or QA engineering experience and a strong track record of shipping test infrastructure or evaluation systems, with solid backend/scripting skills in Python, Rust, Go, or similar. Nice-to-have qualifications include hands-on experience evaluating modern AI systems (LLMs, RAG pipelines, agents, multimodal models), familiarity with metrics like WER, MOS, or latency/TTFB, and experience building evaluation frameworks used by other teams. This is a remote-friend

Deepgram is seeking a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech, audio, and multilingual models before release. You will define evaluation methodologies and build automated evaluation pipelines for batch and streaming workloads, including metrics such as WER, latency, time-to-first-byte, and anomaly detection to guard against regressions. The role involves developing scalable, reproducible evaluation infrastructure—canaries, orchestration, and result-aggregation pipelines—that run against production models and, when needed, large GPU clusters. You will translate Research benchmarks into automated pass/fail gates and embed evaluation and quality checks into CI/CD so quality is verified continuously. You’ll collaborate with Research, model training, inference, and product teams to provide trusted evaluation signals that inform release and optimization decisions, and work with DevOps/Infra to establish ephemeral test environments and reliable results-aggregation infrastructure. The ideal candidate will have 5+ years of professional software or QA engineering experience and a strong track record of shipping test infrastructure or evaluation systems, with solid backend/scripting skills in Python, Rust, Go, or similar. Nice-to-have qualifications include hands-on experience evaluating modern AI systems (LLMs, RAG pipelines, agents, multimodal models), familiarity with metrics like WER, MOS, or latency/TTFB, and experience building evaluation frameworks used by other teams. This is a remote-friend