Hybrid Position (80% Remote - 20% On-site Critical Meetings as needed) Location - Brooklyn, NY The Senior Data Engineer will assist the Application Engineering team in building a robust, secure, and modern data pipelines to ingest, process, transform applications data using Informatica Intelligent Cloud Services or a comparable ETL/ELT tool employing modern data movement strategies and methodologies. The data needs to be store in an Azure Data Lake and eventually brought into a cloud-based data store such as Snowflake or a similar data store. Then, the Analytics reporting solution needs to be built either using cloud-based Google Looker or Microsoft Power BI or another similar tool.
The Senior Data Engineer will also build highly available, robust, concrete data pipelines and reporting solutions following industry best practices, adhering to internal security guidelines, modifying, and running automated CI/CD pipelines used for releasing code, and modifying and executing terraform modules for deploying infrastructure components, working collaboratively under the direction of the data engineering management and leads. The resource will be a person with integrity who is dependable and is fully focused on delivering optimal solutions with little to no maintenance and operations overhead.
RESPONSIBILITIES: - Be experienced in Data Engineering best practices, technologies, tools and processes.
- Bring s ound knowledge of Data Warehouses and Lake House concepts and practical implementation experience.
- Build a framework of repeatable solutions and playbooks enabling efficient and predictable data pipelines.
- Have hands-on development experience in the implementation of an agile, cloud centric data warehousing and reporting platform with team members of various experience level.
- Interact with clients, both technical and non-technical stakeholders.
- Handle relationships with end users. Interact regularly to gather feedback, listen to their issues and concerns, and recommend solutions.
- Meet critical deadlines and deliver in short sprints.
- Ensure successful delivery of new reports and dashboards as needed.
- Maintain and curate data documentation including Architectural Decision Records (ADR), how-to guides, data lineage and ownership using Azure DevOps or similar tool.
- Maintain query performance and tuning to ensure cost optimization.
- Participate in joint application development sessions with co-engineers and end users and be willing to brainstorm.
- Complete technical documentation and be willing to transfer knowledge as needed.
REQUIRED SKILLS: - 12+ years developing Data Pipelines / Flows using ETL/ELT tools and technologies.
- 10+ years of strong SQL fluency (query optimization, windowing functions, aggregation, etc.).
- 5+ years building complex Analytics and Reporting solutions.
- 3+ years' experience with a cloud data lake/warehouse solution (Snowflake, Redshift, GCP etc.).
- Hands on experience working with data integration tools like Informatica Intelligent Cloud Services, Informatica Power Center or SSIS or a similar tool.
- Extensive experience developing production grade, large scale data solutions.
- Experience performing conceptual, logical, and physical data modeling using data modeling tools in complex, large-scale environments.
- Experience working with Microsoft Azure cloud computing platform and services.
- Experience managing data orchestration at scale using tools such as Airflow, Prefect and Dagster.
- Experience with traditional RDMS platforms (Oracle and SQL Server).
- Experience working with version control systems (e.g., Git)
- Good understanding of CI/CD principles.
- Experience developing dashboards and reports in applications such as Oracle Analytics Server (OAS), Microsoft Power BI and Google Looker.
DESIRABLE SKILLS/EXPERIENCE: - Experience using Azure services for Security, Blob Storage, Data Lake, Databricks, Data Factory etc.
- Programming experience with Python or Java
- Experience with Azure Monitoring services
- Microsoft Certified Azure Solutions Architect Expert or a Snowpro Certification or a similar one