Artificial Analysis, Inc.

Solutions Engineer — Language Models

United States (Remote)

Sponsorship not specifiedDetected 41 days ago

PythonGitDatadogMachine LearningData EngineeringNLPLLMsCommunication

About the role

It's about running a sophisticated existing stack exceptionally well, consistently and reliably, while being the trusted technical face of Artificial Analysis to our customers.

Operate and maintain our Python-based language model benchmarking pipeline end-to-end: onboard new models, configure evaluations, execute benchmark runs, and validate results
Maintain documentation of processes, known issues, and model-specific configurations
Collaborate with the engineering team to flag pipeline improvements and contribute to process refinements

Strong Python proficiency and comfort working with complex codebases you didn't write
Hands-on experience working with AI/ML model APIs (OpenAI, Anthropic, Google, Meta, etc.)
Excellent debugging skills - you can trace issues across APIs, data pipelines, and code
Strong written and verbal English communication skills, with the ability to explain technical concepts clearly to technical stakeholders

Experience with AI evaluation, benchmarking, or testing methodologies
Familiarity with LLM inference infrastructure (tokenization, latency measurement, throughput metrics)
Experience working in or with AI labs or model providers
Background in B2B SaaS or developer tools
WHY ARTIFICIAL ANALYSIS?
Shape how AI gets built: The leading AI labs track our benchmarks and use them to guide their development priorities.
Your work will directly influence the direction of AI.
Become a world expert in AI: You will evaluate every major model, across every major capability, as they are released.

you'll add new models to our evaluation pipeline, run and debug benchmarks, and serve as the primary technical point of contact for AI lab customers - explaining results, fielding methodology questions, and resolving API endpoint issues over Slack and video calls.
This is not a software engineering role focused on building new systems.
Serve as the primary technical contact for AI lab customers: communicate benchmarking results clearly, explain methodology, field technical questions, and troubleshoot integration issues via Slack and video conferencing

This listing is sourced directly from Artificial Analysis, Inc.'s careers page and normalized into a canonical job model.