AIRFLOW TECH LEAD

Job title: Airflow Tech Lead

Location: Whitehouse, NJ

Duration: 12+ months

• Architect/Design the Apache Airflow implementation.

• Provide SME in Airflow administration

• Single point of contact and liaison

• Build, maintain, monitor, and orchestrate workflows or data pipelines

• Provide environment/integration support to data platform/ tools

• Provide inhouse KT and training to other resources.

• Bring in best practices with respect to Airflow implementation.

• Guide team in Airflow implementation and orchestration of data pipelines. 3.Experience 8to12Yrs 4.Required Skills ,Kafka,Apache Airflow,Azure Synapse 5.Nice to have skills ,Python,Azure Kubernetes Service 6.Technology -Not Applicable 7.Shift Day 8.Roles & Responsibilities

• 3 to 5 years of industry experience as an Apache Airflow developer/architect

• Experience with data pipeline and workflow management tools: Airflow

• Experience with developing guidelines for Airflow clusters and DAGs

• Experience with Performance tuning of the DAG and task implementation

• Develop DAG - data pipeline to on-board and change management of datasets

• Experience installing Apache Airflow, configuring, and monitoring Airflow cluster

• Expertise in Python and its frameworks

• Understanding of airflow rest services and integration of airflow of platform eco-system

• Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.

• Experience building and optimizing - big data- data pipelines, architectures, and data sets.

• Experience of working with Azure Kubernetes/ Databricks/ Kafka/ Jenkins

• Working experience with CI-CD tools like Jenkins and Github code repositories

• Strong analytic skills related to working with unstructured datasets.

• Build processes supporting data transformation, data structures, metadata, dependency and workload management.

• Proficient in at least one and experience with modern programming languages (Java, C#, Python, JavaScript/TypeScript) and open-source technologies

• Possess professional software engineering practices and best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations

• Fast learner and a team player

• Working knowledge in Azure is required.

• Apache Airflow Fundamentals

• Agile/Scrum principles

• Excellent troubleshooting and debugging skills

• Ability to work independently as well as with multi-disciplinary teams

Last updated on Jul 28, 2023