Job Description
The Senior Data Engineer will play a key role by building certified data pipelines and views to be consumed by other business areas and visualization layers.
The role involves hands-on data engineering and the development of end-to-end data pipelines (processes supporting data ingestion, curation, visualization, and delivery using Microsoft Azure Data & Analytics Services). A vital skill to have will also be translation of business requirements to technical solutions, working with development teams, architects, and business analysts by applying enterprise standards, architecture, and technologies.
- 6 + years of previous data engineering experience, with 3+ years of experience building out table structures to host multiple different data entry points to serve multiple customer needs.
- Build, test, and maintain data pipelines with focus in continuous integration and fault tolerant ETL processes.
- Maintain/enhance our PowerBI / SSAS datasets and C#, SSRS/SSIS Solutions.
- Advanced experience with SQL, Stored Procedures and ETL processes using SSIS, Visual Studio and other data extraction tools.
- Strong knowledge of Azure Databricks, ADF, ADLS, Synapse etc.
- Hands-on experience on designing and developing scripts for ETL processes and automation in Azure data factory, Azure Databricks and PySpark using Python, Spark, and T-SQL
- Good understanding of data modeling and ability to apply different techniques within a cloud platform.
- Knowledge of Azure networking – Subnets, private endpoints, firewalls etc.
- Knowledge of GIT/Jenkins – using pipelines for deployment