Data Engineer / Data Architect

11/2021 – offen

Azure Data Stack Consultant
Support the clients of the internal consultancy organization. Implementation of executive reporting services for a leading German manufacturing client.
Tasks:
Azure Devops Implementation of deployment pipelines for Databricks and Data Factory. Development of data pipelines in SPARK SQL and Azure Data Factory.
Technology stack: Azure Devops, Azure Blob Storage, Azure Data Factory, Azure Databricks, PowerBI

Azure Databricks, Big Data, Microsoft Azure

6/2021 – offen

Azure Big Data and DevOps Consultant (Datalake Technical Lead)
Working on a large scale Data Lake infrastructure in Azure cloud, supporting several projects as technical consultant for implementation and architecture (Devops for Big Data).
Technology stack: Azure Databricks , Azure Devops, Azure Data Lake Store Gen1 - Gen2, Azure Data Factory, SQL server , PowerBI, Logic Apps, Function Apps, ML workspace
Tasks:
Design and implementation of Azure Devops tooling for Azure Data Factory, Azure Databricks and Azure Logic Apps, CI/CD with azure pipelines through several testing stages with fully automated cloud infrastructure.
Infrastructure automation with Terraform (for azure resource manager)
Conceptualization and implementation of Azure Data Lake Store Gen2 security. Meta and master data management tooling.
SPARK development on Azure Databricks.
Azure Data lake store Gen1 to Azure Data Lake store Gen2 migration

Azure Databricks, Big Data

6/2019 – 6/2021

Working as Big Data Architect to further develop a several 100TB scale Mapr Hadoop environment.
Technology stack: On-premise, Redhat 7, Mapr 5.2 -Mapr 6.1 -- and ecosystem packages: Hive 2.3, tez 0.9, Spark 2.3, Drill 1.15., Flume, Kafka. Airflow
Tasks:
Consultant for SPARK and HIVE,, optimized several processes from dozens of hours of runtime to minutes.
Devops automation of maintenance tasks in Ansible and Puppet.
Patching and upgrade automation.
Coordination of upgrades patching and incident follow-up

Big Data