Data Architect | Big Data Architect | DevOps Engineer | Cloud Engineer

2/2023 – 6/2023

- Design and implementation of large scale, high available, fault tolerant Apache Airflow. The implementation consists of running Airflow with different executor type both on bare metal/virtual machines and OpenShift (Kubernetes). Containers are built from source using Buildah based on ubi9-micro image. Podman and systemd script were used to run containers and services on bare metal/virtual machine nodes. The setup also covers Disaster Recovery scenarios. Helm charts were used for OpenShift deployment.
- High available, disaster recoverable and load balanced Postgres setup using PgBouncer, HAProxy, Patroni with backup and recovery via BarMan on both bare metal and Kubernetes.
- High available Redis setup using Sentinel.
- S3 compatible storage for Airflow remote logging.
- Monitoring of complete stack with Prometheus.
- Support the migration of workflows from Control-M, Automic and Tivoli WS to Airflow.

Architekturvisualisierung, Cloud (allg.), Docker, OpenShift, PostgreSQL, System Architektur

1/2023 – 3/2023

- Design, and implementation of Cassandra clusters in three different environments. The production cluster consists of multi-datacenter (primary and secondary clusters) setup to safeguard datacenter disaster scenarios. Clusters were tuned based on workload.
- Reaper was setup to perform Cassandra repair jobs.
- Prometheus was installed, dashboards were configured to show metrics collected from
Cassandra nodes and application.
- Complete setup was automated by Ansible.

Design (allg.), Red Hat Enterprise Linux (RHEL), Systems Engineering, Architektur (allg.)

10/2017 – 12/2022

- Leading cross-functional team including OPS and Dev members
- Part of core team for DWH (Oracle, Hadoop) migration to GCP
- Design and setup of GCP projects via Terraform (GCS, BigQuery, Dataproc, Dataflow, Pub/Sup, Cloud Composer, Cloud Build, etc.)
- Evaluation of Databricks platform including Delta Live Table, Job orchestration, Databricks SQL, Cluster policies, Dashboards and monitoring and alerting
- Design and implementation of PII data pseudonymization for cloud migration
- Installation, configuration and maintenance of MapR eco-system, fully automated by Ansible and Puppet (MapR core, Spark, Hive, Hue, Livy, Objectstore, NFS and monitoring packs)
- Setup and configuration on MapR Monitoring solution (OpenTSDB, Elasticsearch,CollectD, FluentD, Kibana, Grafana)
- Major upgrade and patching of MapR clusters with full automation by Ansible
- MapR cross-clusters data replication setup, cross cluster data access setup
- Containerization of MapR client/edge node by Ansible automation (OpenShift, Podman, Buildah, Ansible bender)
- Datacenter migration planning and execution which include migration of all MapR clusters to new DC
- Data restructuring to achieve better Application (Spark, Hive) performance
- Application tuning (SQL performance tuning) for Spark and Hive
- Spark streaming application development to replace Apache Flume
- Apache Airflow installation, configuration, update and maintenance on OpenShift. Application orchestration by Airflow
- JupyterHub installation and configuration including custom configuration per user
- Installation and configuration of Datahub/OpenMetadata (Data Catalog) including in-house development of data lineage generator for different data sources (Oracle, Hive,SAS)
- Installation, configuration and maintenance of Cassandra, Kafka, Postgres and Redis
- PoCs on different in-memory query engines (Apache Drill, Trino, Starburst)

Design (allg.), Container Management, Data Science, Apache Hadoop, Apache Spark, Big Data, Datawarehouse / DWH, Maschinelles Lernen, Back up / Recovery, KVM (Kernel-based Virtual Machine), Kubernetes, Architektur (allg.)