Data Engineer with Python /Pyspark

• Develop and implement data pipelines for ingesting and collecting data from various sources into a centralized data platform.
• Develop and maintain ETL jobs using AWS Glue services to process and transform data at scale.
• Optimize and troubleshoot AWS Glue jobs for performance and reliability.
• Utilize Python and PySpark to efficiently handle large volumes of data during the ingestion process.

• Design and implement scalable data processing solutions using PySpark to transform raw data into a structured and usable format.
• Apply data cleansing, enrichment, and validation techniques to ensure data quality and accuracy.

• Create and maintain ETL processes using Python and PySpark to move and transform data between different systems.
• Optimize ETL workflows for performance and efficiency.

• Collaborate with data architects to design and implement data models that support business requirements.
• Ensure data structures are optimized for analytics and reporting.

• Work with distributed computing frameworks, such as Apache Spark, to process and analyze large-scale datasets.

• Manage and optimize databases, both SQL and NoSQL, to support data storage and retrieval needs.
• Implement indexing, partitioning, and other database optimization techniques.

• Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver effective solutions.
• Work closely with software engineers to integrate data solutions into larger applications.

• Implement monitoring solutions to track data pipeline performance and proactively identify and address issues

• Ensure compliance with data privacy regulations and company policies.

• Stay abreast of industry trends and advancements in data engineering, Python, and PySpark.

Requirements

• Proficiency in Python and PySpark.
• Strong knowledge of data engineering concepts and best practices.
• Hands-on experience with AWS Glue and other AWS services.
• Experience with big data technologies and distributed computing.
• Familiarity with database management systems (SQL and NoSQL).
• Understanding of ETL processes and data modeling.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
• Bachelor's degree in Computer Science, Information Technology, or a related field.

Last updated on Dec 20, 2023