Solstice

Solstice

Member of Technical Staff (QA Engineer - Agentic Systems)

New York City · Staff+

Sponsorship not specified$160k-$300kDetected 26 days ago
TypeScriptPythonReactBackend DevelopmentCI/CDRESTMachine LearningLLMsAgentic AILLMOpsMLOpsLangGraphPlaywrightTest AutomationManual Testing

About the role

  • Our platform generates regulated pharmaceutical marketing content for the brands we work with, so when the output is wrong, say an unsupported claim or a missing safety disclosure, it becomes a real compliance problem and not just a bug to file.
  • What makes this hard is that the system is probabilistic.
  • The same prompt can return different answers, "correct" is often a judgment call, and the exact-match assertions that traditional QA relies on don't apply.

Responsibilities

  • We've raised from investors like Transformation Capital, Twelve Below, Virtue and the founders of Datavant, Commure, and Paradigm to supercharge our growth and build an elite team of engineers and operators.
  • Building anything great requires commitment and dedication.
  • Build our evaluation systems. Because we can't check an output against a single correct answer, you'll design the evals that score quality instead and decide, with evidence, what is good enough to ship.
  • Test the agents for the ways they actually fail. Agents drift off their goal, loop on the same tool call, pick the wrong tool, or get hijacked by a malicious instruction buried in a document we ingest. Those are the cases you'll design for.
  • Protect the compliance-critical paths. The checks that keep an unsupported claim or a missing disclosure out of a finished asset are the ones that matter most, and you'll own how we test them, including verifying claims against approved source material.
  • Own end-to-end testing across the app. Build and maintain a Playwright suite that exercises the real user flows, from login through content creation and review, so a frontend or API change can't quietly break something a customer depends on.
  • Independent thinking and a strong sense of ownership. We can teach specifics, but you should be able to make calls on your own and build a quality function from nothing.
  • We're hiring our first dedicated QA lead to own quality for the AI that powers Solstice.
  • Relocation support

Requirements

  • Comfort with evaluation methods: golden datasets, LLM-as-judge (rubric, pairwise, reference-based), and calibrating those judges against human or expert labels.
  • You can explain technical risk to non-technical people, and you both give and take direct feedback.
  • Hands-on experience with agent or LLM frameworks (LangChain, LangGraph) and a feel for how agentic systems break.
  • Experience with eval and LLM-observability tools (LangSmith, Langfuse, Arize Phoenix, Braintrust, RAGAS, Promptfoo, OpenAI Evals, or similar).
  • Comfort in a modern frontend stack (TypeScript and React) so you can write meaningful UI tests and reproduce bugs quickly.
  • Backend experience with async Python services and task queues.

Compensation

  • This role offers a highly competitive NYC salary in the $160,000 to $300,000 range, calibrated to reflect experience, seniority, and the level of ownership you'll take on as part of the early team.
  • We benchmark against top-tier startups and established tech companies to ensure our compensation is both fair and compelling, and we maintain flexibility at the upper end of the range for exceptional candidates who can meaningfully accelerate our product and engineering roadmap.

Benefits

  • Health, dental, and vision insurance
  • Ground-floor equity opportunity
  • Professional growth stipend
  • Experience in a regulated industry such as pharma, healthcare, or finance.

Company info

  • Solstice is redefining how life sciences organizations commercialize their therapeutics.
  • We are building a commercial engine that allows pharmaceutical marketers to launch campaigns at 100x the speed.
  • Just as much of what keeps customers happy is ordinary product reliability: that the app does what it should, and that a frontend tweak or a new feature doesn't quietly break something in production.
  • Your work will reach across the whole product, from the backend services to the frontend customers use every day.
  • We're looking for someone who has the fire to prove others wrong.

Visa & Work Authorization

  • Visa sponsorship (O-1, H-1B, TN) available
Apply directly at Solstice →Create a free account for alerts like thisView Solstice immigration profile

This listing is sourced directly from Solstice's careers page and normalized into a canonical job model.