Etchedai

Inference Software Engineer

Cupertino, CA

Sponsorship not specifiedDetected 118 days ago

PythonRustC++Distributed SystemsNLPResearch

About the role

We do not have boundaries between engineering and research, and we expect all of our technical staff to contribute to both as needed.

Contribute to the architecture and design of the Sohu host software stack
Implement high-performance, modular code across the complete Etched software stack, consisting of a mix of Rust, C++ and Python.
Work with AI model researchers and product-facing teams building out the Etched serving front-end.
Build scheduling logic for handling continuous batching and real time inference
Implement inference-time acceleration techniques such as speculative decoding, tree search, KV cache sharing, etc.
Implement distributed networking primitives for efficient multi-server inference
Relocation support for those moving to Cupertino
We think most of the progress in the AI field has come from using more FLOPs to train and run models, and the best way to get more FLOPs is to build model-specific hardware.
Larger and larger training runs encourage companies to consolidate around fewer model architectures, which creates a market for single-model ASICs.

Experience with C++ and Python
Familiarity with transformer model architectures and inference serving stacks (vLLM, SGLang, etc.) or experience working in distributed inference/training environments
Experience working cross-functionally in large software and hardware organizations
Experience with Rust
Familiarity with GPU kernels, the CUDA compilation stack and related tools, or other hardware accelerators
You may be a good fit if you have

We are a fully in-person team in Cupertino, and greatly value engineering skills.

This listing is sourced directly from Etchedai's careers page and normalized into a canonical job model.