Machine Learning Engineer, LLM Systems

Build the evaluation, reasoning, and retrieval infrastructure underneath our seventeen agents — the parts that shouldn't guess.

About this role

The parts of CMO Owls that shouldn't guess, don't. You will build and harden the model reasoning, retrieval, and evaluation infrastructure underneath our agents: how they ground themselves in real customer data, how we measure whether an agent's output is actually good, and how we catch quality regressions before a customer ever sees them.

What you will do

• Design and maintain evaluation pipelines and benchmarks for agent output quality
• Build and improve retrieval-augmented generation (RAG) systems grounding agents in real customer data
• Work on prompt and context engineering for our production agent fleet
• Investigate model quality, latency, and failure cases across our seventeen agents
• Partner with the Agentic AI Engineering team on architecture for new agents

What we are looking for

• Bachelor's or Master's in CS, ML, or a related field, or equivalent real-world experience
• Strong Python and real ML/NLP or information-retrieval fundamentals
• Hands-on experience with LLM applications, RAG, or embeddings, not just coursework
• Comfort reading research and turning it into a working, production system

Nice to have

• Experience with evaluation frameworks, RL environments, or agent tooling
• Familiarity with vector search and structured-output techniques for LLMs