Senior Data Engineer

11/2022 – 10/2023

Part of two agile teams (18 members) providing data analytics platform for customs and border security business departments. Redesigned ETL architecture for ~60 data products processing customs and personal data. Improved performance by factor of ~100. Technical lead in decision making with architects and review of Pull-Requests and tickets of 12 developers.

Kafka, SQL, Apache Spark, Python

1/2022 – 9/2022

Part of agile team (~30 members) providing cloud data platform for industry-leading business departments. Designed and implemented architecture for ~30 ETL applications and APIs processing global IoT data on a terabyte scale. Deployment of various managed services in hybrid cloud setup. Enablement of client developers. Optimized storage to save ~4000€ per month.

Python, Azure, Databricks (Delta Lake), PySpark, SQL, NiFi, Shell, CI/CD, Terraform}}

Apache Spark, Azure Databricks, Bash (Shell), Kontinuierliche Integration (CI), Microsoft Azure, Python, SQL

6/2020 – 1/2022

Part of agile team (6 members) with goal to provide data-driven insights to end-users on globally distributed IoT data of industry-leading client. Designed and implemented architecture for ETL and machine learning applications that process data daily on a giga- and terabyte scale. Full responsibility from raw data collection over processing with machine learning to visualization. Deployment of various managed services in private network on Azure. Enablement of client developers.

Microsoft Azure, Apache Spark, Azure Databricks, Bash (Shell), Kontinuierliche Integration (CI), Python, SQL