Nityo Infotech Corporation
Lead Data Engineer
Irvine, California, USA · Contract
Sponsorship not specifiedDetected 54 days ago
PythonPostgreSQLAWSDockerCI/CDDevOpsAirflowData EngineeringAI OrchestrationCommunicationCollaborationProblem SolvingMentoring
About the role
- The ideal candidate will have a strong background in data engineering, with extensive experience in Python, PySpark, AWS services, PostgreSQL, Apache Airflow and Docker.
- This role requires a professional with a proven track record of designing, implementing, and maintaining robust data pipelines and architectures.
- Workflow Orchestration: Use Apache Airflow to schedule and monitor complex data workflows.
Responsibilities
- Design and Develop Data Pipelines: Create and maintain scalable data pipelines using Python and PySpark to process large volumes of data efficiently.
- Cloud Integration: Utilize AWS services (such as S3, CloudWatch, ECS, ECR, Lambda) to build and manage cloud-based data solutions.
- Database Management: Design, implement, and optimize PostgreSQL databases to ensure high performance and reliability.
- Containerization: Implement and manage Docker containers to ensure consistent and reproducible environments for data processing tasks.
- Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
- Create and maintain scalable data pipelines using Python and PySpark to process large volumes of data efficiently.
- Implement and manage Docker containers to ensure consistent and reproducible environments for data processing tasks.
- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
Requirements
- Bachelor s or Master s degree in Computer Science, Engineering, or a related field.
Nice to have
- Familiarity with CI/CD pipelines and DevOps practices.
- Knowledge of data warehousing concepts and ETL processes.
Skills
- Proficiency in Python and PySpark.
- Extensive experience with AWS services (S3, CloudWatch, ECS, ECR, Secrets Manager, Cloud9 IDE).
- Strong knowledge of PostgreSQL and database optimization techniques.
- Hands-on experience with Apache Airflow for workflow orchestration.
- Proficiency in Docker for containerization.
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration abilities.
- Ability to work in a fast-paced, dynamic environment.
- Soft Skills: Excellent problem-solving and analytical skills.
- 10+ years overall, 3-4 years in specified technologies
Company info
- We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team.
Apply directly at Nityo Infotech Corporation →Create a free account for alerts like thisView Nityo Infotech Corporation immigration profile
This listing is sourced directly from Nityo Infotech Corporation's careers page and normalized into a canonical job model.