Principal Data Engineer- GCP / Newton MA ,( Onsite), 6 months Contract

• Design, develop, and implement end-to-end data solutions (storage, integration, processing, access) in Google Cloud Platform (GCP);
• Create ETL/ELT pipelines that transform and process terabytes of structured and unstructured data in real-time;
• Design data models for optimal storage and retrieval to support sub-second latency;
• Deploy and monitor large database clusters that are performant and highly available;
• Deeply immerse in our Priority Engine Data Pipelines, to acquire full understanding and ability to converse with business representatives and individual contributors alike; and
• Incorporate solid understanding of cloud-based offerings in the space.
• Some upcoming technical challenges include scaling our data ingestion pipelines across a growing number of GCP, AWS, and data center-based data sources, reducing the latency of our product data ingestion pipelines through moving batch jobs into a streaming architecture, and extending our data lake architecture for the growing ecosystem of data ingestion and creation tools.
• Be a hands-on contributor on the team;
• Collaborate with product/application architects to develop holistic solutions;
• Implement operational procedures (logging, monitoring, alerting, etc.) for dependable running of pipelines/jobs; and
• Be autonomous. You own what you work on. You move fast, take ownership, and get things done. The Data lake/warehouse is a central and essential service for multiple initiatives, and you need to be able to multitask and juggle priorities on a regular basis with a deep understanding of business impacts of prioritization decisions.

• BS Degree in Computer Science or a related field,
• 10+ years industry experience, 5+ years of experience building real-time and distributed system architecture, from whiteboard to production,
• Strong programming skills in Python and SQL,
• Versatility: 8+ years' experience across the entire spectrum of data engineering, including:
• Cloud data stores (Example: GCP BigQuery, GCP BigTable, GCP FireStore, GCP CloudSQL, ELK)
• Data pipeline and workflow orchestration tools (e.g., Dataflow, Pentaho, Airflow, Azkaban)
• Data processing technologies (e.g., GCP BigQuery, Spark)
• Data messaging technologies (e.g., GCP PubSub, Kafka)
• Deployment and monitoring large database clusters in public cloud platforms (e.g., Docker, Terraform),
• Unix/Linux Shell scripting,
• Demonstrated knowledge of industry trends and standards,
• Ability to think through multiple alternatives and select the best possible solutions for strategic and tactical business needs, and
• Excellent communication. You will need communicate complex ideas effectively to both technical and non-technical audiences, and both verbally and in writing.

• Experience with GCP
• Java Programming

Last updated on Feb 24, 2023