Data Engineering Lead

Position Summary: 

The Data Engineering team within Data Services is responsible for creating and maintaining data pipelines, assembling complex datasets, and extracting, transforming, and loading data from a variety of data sources.  

Data Engineers work closely with Software Engineers, Product Managers, Data Scientists, and teams across the enterprise to deliver product features, build data pipelines, and architect our Product and Analytics data infrastructure for optimal performance.  

Data Engineers ensure that Truckstop has hygienic, accurate, data available at the point of need and partner with the System Architecture and DevOps teams to troubleshoot data pipeline production issues and respond to incidents. 

The SR/Lead Data Engineer role works directly with the Director of Data Services to oversee the Data Engineering team, handle the most complex issues, and provide coaching and guidance to more junior Data Engineers. The Sr/Lead DE possesses expert working knowledge of technical subject matter, as well as the ability to strategize and problem solve on data infrastructure, data migration tooling, data modeling, and complex troubleshooting.  

Essential Job Functions: 

• Create, automate, and maintain complex data pipelines from multiple source systems using migration and orchestration technology, employing architecture best practices, leveraging leading edge technology, and utilizing multiple programming languages. 
• Extract, transform, and load data from a variety of data sources using SQL and Python and cloud data technologies.  
• Interact with API and Kafka technology to ingest and transform data into Snowflake for Product and Data Science team end uses.  
• Write and optimize queries and data processes, write orchestration functions, perform source to target mapping and data modeling. 
• Work directly with Product to engineer performant data pipelines required to support Product analytics, reporting, and strategy. 
• Work with data architects, database administrators, Infosec and software engineering solution architects to continually improve data security, data capture, data pipelines, and data infrastructure.  
• Work with IT operations to resolve data related technical issues and respond to data related major incidents. 
• Perform incident resolution and root cause analysis of critical outages. Implement solutions to systematic failures. Provide on-call support, including after-hours. 
• Assist with documentation of the environments that support our products in Confluence and Atlan 

Position Requirements: 

• Bachelor's degree or equivalent professional experience required; Computer Science or Engineering preferred. 
• Demonstrated expertise in SQL and Python required. 
• Minimum of +6 years of experience required. 
• Experience with Snowflake, MS SQL, and PostgreSQL required. 
• Experience building and optimizing reliable, idempotent data pipelines (streaming or batch) using ETL tools such as Matillion, Apache Airflow, FiveTran, Kafka and Spark required. 
• A successful history applying software engineering principles to build data infrastructure tools/libraries to automate and scale data pipelines. 
• Experience with supporting Data Science applications, including machine learning data pipelines; Azure ML experience a plus 
• Understanding of applying logging and metrics to monitor and detect data pipeline performance issues. 
• Advanced understanding of data warehousing architecture including performance optimization and tuning, specifically within Snowflake and Azure.  
• Working knowledge of dbt data modeling and source to target mapping with a demonstrated understanding of data schema design, Snowflake clustering and micro-partitioning 
• Experience working in Azure and AWS cloud-based environment 

Last updated on Aug 21, 2024