Lead Data Engineer

 Job Title: Lead Data Engineer

 Location: India

 Job Type: Full-Time

 Experience Level: 6+ Years

 

About the Role:

As a Data Engineer at Material, you will play a crucial role in building and maintaining our advanced data infrastructure. You will be responsible for designing, implementing, and optimizing data solutions that are critical to the success of our data-driven initiatives. This role requires a deep understanding of cloud technologies, particularly AWS, and a strong foundation in data processing and workflow management tools like Airflow and PySpark.  You should be comfortable jumping into existing codebases and maintaining pre-defined coding standards and frameworks.

 

Key Responsibilities: 

·       Design, build, and maintain scalable and reliable data pipelines using AWS services, Airflow, and PySpark. 

·       Develop and optimize data models to support our data warehousing and analytics needs. 

·       Implement and manage continuous integration and deployment (CI/CD) pipelines for data operations using Git and Docker. 

·       Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet these needs. 

·       Lead the automation of data processes and contribute to the development of best practices in data engineering. 

·       Stay updated with emerging technologies and industry trends to continuously improve our data infrastructure. 

·       Document existing and new processes while being able to communicate advancements in pipelines and repositories. 

·       Engage in developing QA processes for API and Pipeline developments. 

 

Minimum Qualifications: 

·       Minimum of 6 years of experience in data engineering or a related field.

·       Strong proficiency in Python, PySpark, AWS cloud services, and Docker.

·       Expertise in designing and implementing data workflows using Airflow.

·       Experience with Git or similar version control systems.

·       Proven ability to design and implement large-scale data solutions in a cloud environment.

·       Excellent problem-solving skills and the ability to work independently or as part of a team.

·       Strong communication skills and the ability to work effectively with both technical and non-technical stakeholders.

·       Experience in working with semi-structured data.

 

Additional nice-to-haves: 

·       Experience in leading data engineering projects and teams.

·       Knowledge of advanced analytics and machine learning concepts.

 

•

Last updated on Oct 16, 2024