Big Data Engineer

Big Data Engineer
80 C2C
NYC, NY
6+ Months
Any VISA
F2F
 
Job Description
 
Our client is seeking a highly motivated Senior Software Engineer with Big Data to join a small, highly collaborative team and help create the next generation of data products for trading and investing.
 
Responsibilities:
 
• Develop production-grade data pipelines and ETL processes to support leading-edge analytics and big data processing.
• Curate massive amounts of data and make it accessible via state-of-the-art technologies.
• Review internal and external technological techniques, processes, and tools - to improve efficiency and better serve clients worldwide.
 
• Gather new product requirements from business stakeholders.
• Design new production workflows and architectures to support those products.
• Collaborate with Data Scientists and Software Developers to develop experiments and deploy solutions to production.
• Rapidly integrate new content sets (financial and non-financial) into big data ecosystem.
• Contribute ideas and constructive feedback to the tech and business teams.
• Stay current on technological trends.
• Take ownership of new projects and initiatives.
• Stay current on technological and analytical trends.
• Be passionate about data and big data tech.
• Maintain positive attitude.
• Motivate and coach other members of the team.
 
Example Projects:
Develop data ingestion and normalization framework that can collect and process data from hundreds of sources daily (and in real-time).
"Productionalize” the analytical processes developed by the Data Science team.
Productionalize – (verb) Performance tune models, automate data processing workflow, deploy in a highly-available and scalable way.
Develop analytical libraries and tools that allow the Data Science team to take full advantage of Data Platform.
 
Background:
 
We will consider candidates from a wide range of backgrounds, however, the many of the problems the candidate would be tasked with solving will require writing complex programs, designing systems, and analyzing data. Therefore, candidates with a computer science or engineering background are preferred.
 
Successful candidates will have:
• 5-8 Yeas professional experience in engineering or other technical role.
• Experience deploying applications in a production or mission-critical environment.
• Experience in financial services is a plus.
 
Education:
 
Bachelor's Degree from top tier university in Computer Science, Engineering, Physics, Mathematics, or similar quantitative discipline. Master's Degree a plus.
 
 
 
Skills Required:
 
• Good Programming Skills: Python, Java, Scala, R, SQL.
• Knowledge of cloud platforms and common architectures: AWS, Google.
• Strong Familiarity with Big Data technologies and architectures: Hadoop, Spark, Kafka, etc.
• Experience with containers and scalable computing platforms: Docker (ECS), Mesos, Kubernetes.
• Good verbal and written communication.
 
Preferred:
 
• Familiarity with Apache Spark: tuning and maintaining clusters, optimizing jobs, etc.
• Familiarity with analytical techniques and machine learning workflows.
• Familiarity with financial data sets and use cases.
• Strong familiarity with AWS.
• Knowledge of Lambda architectures.
• Requires conceptual and practical expertise in own discipline and basic knowledge of related disciplines.
• Is fully self-sufficient and competent in regards to output quality and quantity.
• Requires little or no supervision on a day to day basis.
• Has basic or good knowledge of best practices and how own area of expertise integrates with others.
• Understands key business drivers and is aware of the competition and the factors that differentiate them in the market.
• Provides informal guidance for colleagues with less experience.
• Is accountable for contributing with a competent standard and delivers every time.
• Plans and prioritizes work using resources in the best possible way to complete tasks on time and with high quality.
• Solves problems using existing solutions; exercises judgment based on the analysis of multiple sources of information.
• Is starting to look at 'best practice'.
• Provides creative insights and / or solutions to address client / organizational challenges.
• Impacts a range of activities within own team and other related teams; works within broad guidelines and policies.
• Works quickly and is able to assist other team members where appropriate. Works independently.
• Is able to explain difficult information; works to build consensus.
• Generates healthy debate within team, influencing team to look for a 'better approach'.
• Shares knowledge and expertise across organizational boundaries and endorses "best practices".
• Synthesizes information from various sources by breaking it down into relevant components and generating new data when needed; considers a broad range of issues when determining relevance of data.
• Integrates complex information and concepts and organizes analysis efforts into a systematic framework; verifies logic of others and challenges ideas, assumptions and conclusions.
• Evaluates current situation based on previous experience; identifies advanced implications/ conclusions from the logical analysis of a complex situation or issue; weighs options critically.
• Utilizes knowledge of the organization's business model and operations to drive projects and analyses that support the organization's mission.
• Applies extensive understanding of the policies, practices, trends and information that impact the organization and its customers to anticipate and plan for each step of a complex project.
• Demonstrates understanding of the financial impact of decisions/solutions to drive projects and analyses that support the organization's mission.
 
• Conducts a deep review of data and issues to quickly reveal the root cause of complex problems; anticipates and prepares for potential problems.
• Researches, recommends and reviews solutions to complex problems; guides the analysis of a problem all the way to a successful resolution.
• Develops interim and long-term solutions to complex problems to ensure successful project completion.
• Tests and maintains software applications and related programs and procedures creatively using a variety of software development tools following design requirements of customer.
• Ensures that system applications and tests are successfully completed and documented and all problems are resolved.