Inception
Member of Technical Staff, Data Infrastructure
San Mateo, USA · Staff+
Sponsorship not specifiedDetected 134 days ago
PythonSQLNoSQLBigQueryVector DatabasesMachine LearningTensorFlowPyTorchSparkAirflowLLMsResearch
About the role
- We seek experienced engineers to architect and scale the core infrastructure behind distributed training pipelines and petabyte-scale data catalogs.
Responsibilities
- Design, build, and operate scalable, fault-tolerant infrastructure for LLM research: distributed compute, data orchestration, and storage across modalities.
- Develop high-throughput systems for data ingestion, processing, and transformation - including training data catalogs, deduplication, quality checks, and search.
- Build systems for web crawling, data ingestion, and real-time data processing to support model training operations.
- Develop tools and frameworks for efficient data storage, retrieval, and versioning across distributed systems.
Requirements
- Experience with distributed computing and large-scale data storage systems (HDFS, S3, BigQuery).
Nice to have
- distributed compute, data orchestration, and storage across modalities.
- Ensure data collection adheres to privacy regulations.
- 3+ years of experience building data processing pipelines at scale, particularly with AI/ML applications.
- Strong proficiency in Python and experience with data processing frameworks (Apache Spark, Beam, Airflow).
- Familiarity with synthetic data generation techniques and data augmentation strategies.
- Familiarity with web scraping, crawling technologies, and Common Crawl datasets.
- Experience with SQL and NoSQL databases for managing structured and unstructured data.
- Preferred Skills Experience with large language models and understanding of tokenization, embeddings, and model architectures.
Benefits
- Qualifications BS/MS/PhD in Computer Science, Machine Learning, or a related field (or equivalent experience).
- Solid understanding of machine learning fundamentals and experience with ML frameworks (PyTorch, TensorFlow).
Apply directly at Inception →Create a free account for alerts like thisView Inception immigration profile
This listing is sourced directly from Inception's careers page and normalized into a canonical job model.