FAL
Software Engineer, Infrastructure
San Francisco
Sponsorship not specified$180k-$250kDetected 61 days ago
PythonTerraformAnsibleLinuxComplianceProcurementEmbedded SystemsBGP/OSPFCommunication
About the role
- You are a hands-on engineer who builds the software and processes that keep a large fleet of GPU servers healthy and productive. You write systems and tooling for managing 1000s of servers including provisioning, health monitoring, error detection, and recovery — and when something breaks that automation can't fix, you drive resolution with partners.
- - Build and maintain Python fleet tracking system that manages the full lifecycle of servers including contracting and procurement, target use, pricing, availability, health, RMAs, etc
Responsibilities
- Leverage AI to an extreme level to build tools and automate alerting and recovery
- Implement and enforce OS-level security: hardening baselines, SELinux/AppArmor policies, SSH key management, vulnerability scanning, and compliance automation
- Manage and optimize distributed and local storage systems supporting model weights, checkpoints, and ephemeral scratch: NVMe arrays, NFS, parallel file systems, and object storage
- Develop a suite of automated error detection and recovery processes
- Work with partners to solve technical issues
Requirements
- 3+ years experience managing bare-metal and cloud based server fleets at scale (100+ nodes)
- Strong experience with configuration management and infrastructure-as-code: Ansible, Terraform, cloud-init
Nice to have
- Regular team events and offsites
Skills
- Familiarity with hardware diagnostics and failure modes (GPUs, NVMe, NICs, memory)
- Self-starter who executes quickly, takes ownership, and constantly seeks improvement
- LVM, RAID, NVMe, NFS, Lustre or GPFS, and Linux I/O stack tuning
- Nice to have
- Familiarity with network configuration and diagnostics (VLAN, VXLAN, ECMP, BGP, tcpdump)
- Experience with NVIDIA GPU infrastructure: driver management, health monitoring, DCGM, NVLink/NVSwitch diagnostics, RDMA, InfiniBand/RoCEv2
- Experience with AMD GPUs
- Experience with bare metal and VM provisioning (PXE/iPXE, Kickstart, libvirt, Qemu/KVM)
- Experience with compliance frameworks relevant to cloud providers (SOC 2, ISO 27001)
- Compensation
- $180,000-250,000 plus equity + benefits
- Interesting and challenging work
Compensation
- $180,000-250,000 plus equity + benefits
- San Francisco, CA (we are open to remote in the US for Senior and Staff levels)
- Interesting and challenging work
- We are offering relocation assistance to San Francisco.
- We offer relocation assistance to San Francisco.
- Regular team events and offsites
Benefits
- Build and maintain Python fleet tracking system that manages the full lifecycle of servers including contracting and procurement, target use, pricing, availability, health, RMAs, etc
- Build server management tooling that automates provisioning, health checks, GPU diagnostics, recovery and alerting
- Create and maintain metrics, dashboards, and alerting for hardware health across the fleet (GPU errors, disk failures, network issues, thermals)
- A lot of learning and growth opportunities
- Health, dental, and vision insurance (US)
This listing is sourced directly from FAL's careers page and normalized into a canonical job model.