Big Data Engineer

Big Data Engineer
12+Month Contract
Preference for onsite in Phoenix office, open to remote

Responsibilities:

• Lead and manage engineering teams, drawing from 5+ years of software development experience.
• Employ hands-on expertise in Map-Reduce, Hive, and Spark (core, SQL, and PySpark) for data processing.
• Write and interpret complex SQL (Hive/PySpark-dataframes) for optimizing joins when handling extensive datasets.
• Utilize UNIX shell scripting for data operations.

Requirements:

• Strong grasp of Data Warehousing concepts.
• Knowledge of the Financial reporting ecosystem is a plus.
• Experience with Data Visualization tools like Tableau, SiSense, and Looker.
• Expertise in Distributed ecosystems.
• Hands-on programming skills in Python/Scala.
• In-depth understanding of Hadoop and Spark Architecture and their working principles.
• Ability to design and develop optimized Data pipelines for both batch and real-time data processing.
• Effective communication skills for collaboration with internal and external business partners.

Preferred Qualifications:

• 5+ years of software development experience, including leading teams of engineers and Scrum teams.
• 3+ years of hands-on experience with Map-Reduce, Hive, Spark (core, SQL, and PySpark).
• Proficiency in writing and understanding complex SQL (Hive/PySpark-dataframes) for optimizing joins when processing large datasets.
• Hands-on experience with UNIX shell scripting.
• Knowledge of cloud platforms like GCP/AWS, with experience in building Microservices and scalable solutions, is preferred.
• 2+ years of experience in designing and building solutions using Kafka streams or queues.
• Familiarity with GitHub and proficiency in leveraging CI/CD pipelines.
• Experience with NoSQL databases, including HBase, Couchbase, and MongoDB.

Metasys Technologies is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identify, national origin, veteran or disability status.
•

Last updated on Oct 18, 2023