ReflectionAI

Member of Technical Staff - Web Crawl Engineer

San Francisco · Staff+

H1B sponsorship availableDetected 33 days ago

HTMLDistributed SystemsSparkLLMsA/B TestingSystems EngineeringResearchCommunication

About the role

The web is one of the most important sources of information for frontier AI systems.
The quality, coverage, freshness, and diversity of web data directly influence model capabilities.

Build and operate web-scale crawling infrastructure capable of continuously collecting data across billions of URLs
Design and optimize URL discovery, prioritization, scheduling, and crawl orchestration systems
Develop distributed crawlers that efficiently acquire content while respecting site constraints and operational requirements
Build systems for content extraction, rendering, parsing, and normalization across diverse web formats
Design infrastructure for large-scale recrawling, change detection, and incremental updates
Develop specialized crawlers for high-value domains, dynamic websites, and difficult-to-access content sources
Build observability, monitoring, and reliability systems for large-scale crawl operations
Sponsorship support: We sponsor visas to help exceptional talent join our team and support long-term immigration pathways where applicable.
Team building: We have regular off-sites, happy hours, and team celebrations.
Joining Reflection means building from the ground up as part of a talent-dense team.

Strong understanding of crawling architectures, URL frontier management, scheduling, and distributed crawl coordination
Experience with large-scale distributed systems using technologies such as Ray, Spark, Beam, Flink, or similar frameworks
Familiarity with content extraction, HTML parsing, browser automation, rendering systems, and modern web technologies
Experience operating systems that process petabyte-scale datasets
Excellent communication skills and the ability to reason clearly about system tradeoffs and operational constraints
Experience building large-scale web crawling, search indexing, content acquisition, or internet-scale data collection systems
Strong systems engineering skills, including reliability, observability, performance optimization, and debugging
Experience designing experiments and using data to improve crawl quality, coverage, and efficiency

Familiarity with anti-bot systems, dynamic web content, browser automation, and large-scale extraction pipelines
Understanding of how web data is used in training and evaluating large language models
Experience with distributed storage systems, content deduplication, and web-scale dataset management

Salary and equity structured to recognize and retain our talent globally.
Health & wellness: Comprehensive medical, dental, vision, and life, with an annual wellness allowance.

Top-tier compensation: Salary and equity structured to recognize and retain our talent globally.
Stock options: Everyone who joins and contributes to Reflection's success gets to share in the upside through stock options.
Health & wellness: Comprehensive medical, dental, vision, and life, with an annual wellness allowance.
Life & family: 22 weeks paid parental leave for all new birthing and non-birthing parents, including adoptive and surrogate journeys.
Vacation days: Unlimited paid time off in the U.S. and 30 days in the U.K.

make intelligence open and accessible to all.
Reflection is a research lab making intelligence open and accessible for everyone to use, customize, and build on.
We build open models that let anyone control their intelligence and help shape the future of AI.

We sponsor visas to help exceptional talent join our team and support long-term immigration pathways where applicable.

This listing is sourced directly from ReflectionAI's careers page and normalized into a canonical job model.