Benton Partners

Senior Workflow Orchestration Engineer (Airflow & Scheduling Platforms)

New York City, New York · Senior

Sponsorship not specifiedDetected 1 day ago
PythonJavaGoBashNode.jsBigQuerySnowflakeRedshiftDatabricksAWSGCPAzureKubernetesTerraformHelmCI/CDPrometheusGrafanaSite Reliability EngineeringSparkAirflowdbtAI OrchestrationIncident Response

About the role

  • We're seeking a seasoned engineer to design, operate, and scale our workflow orchestration platform with a primary focus on Apache Airflow.
  • You'll build automation infrastructure and partner across data, trading, and engineering teams to deliver mission-critical pipelines at scale.

Responsibilities

  • Build automation infrastructure: Terraform modules and Helm charts with GitOps-driven CI/CD for environment provisioning, upgrades, and zero-downtime rollouts
  • Implement comprehensive observability: metrics collection, dashboards, distributed tracing, SLA/latency monitoring, intelligent alerting, and runbook automation
  • Enable resilient workflow patterns: build idempotency frameworks, retry/backoff strategies, deferrable operators and sensors, dynamic task mapping, and data-aware scheduling
  • Ensure reliability at enterprise scale: architect and tune resource allocation (pools, queues, concurrency limits) to support high-throughput workloads; optimize large-scale backfill strategies; develop comprehensive runbooks and lead incident response/postmortems
  • Partner with teams across the organization to provide enablement, documentation, and self-service tooling
  • Mentor engineers, contribute to platform roadmap and technical standards, and drive engineering best practices
  • DAG design and testing, idempotency, deferrable operators/sensors, dynamic task mapping, task groups, datasets, pools/queues, SLAs, retries/backfills, cross-DAG dependencies.
  • You'll own the Airflow control plane and developer experience end-to-end-architecture, automation, security, observability, and reliability-while also evaluating and operating complementary schedulers where appropriate.

Requirements

  • 3+ years running Airflow in production at scale (hundreds-thousands of DAGs and high task throughput).
  • Data platform experience with at least one major cloud (AWS/Azure/GCP) and systems like Snowflake/BigQuery/Redshift, Databricks/Spark, EMR/Dataproc
  • Proven incident leadership, runbook creation, and platform roadmap execution

Nice to have

  • Experience operating alternative orchestrators (Prefect 2.x, Dagster, Argo Workflows, AWS Step Functions) and leading migrations to/from Airflow.
  • OpenLineage/Marquez adoption
  • Great Expectations or other data quality frameworks
  • dbt Core/Cloud orchestration patterns (state management, artifacts, slim CI).
  • Cost optimization and capacity planning for schedulers and workers
  • spot instance strategies.
  • Multi-region HA/DR for Airflow metadata DB
  • backup/restore and disaster drills.
Apply directly at Benton Partners →Create a free account for alerts like thisView Benton Partners immigration profile

This listing is sourced directly from Benton Partners's careers page and normalized into a canonical job model.