Senior Data Enginner

8/2024 – offen

Strong understanding of Coalesce’s architecture and core principles, including its metadata-driven approach and node-based interface for declarative data transformations
Proficient in working with YAML template structures and custom configurations to define node behavior, parameterization, and transformations across development environments
Hands-on experience designing and managing various Coalesce node types (Source, Transform, Stage, Snapshot, etc.), enabling reusable, modular, and version-controlled data pipelines
Familiar with Coalesce’s key features such as dynamic SQL generation, data lineage tracking, and modular design that enables scalable development and seamless integration with cloud data platforms like Snowflake
Designed and implemented data transformation workflows in DBT to model, cleanse, and optimize data for analytics in Snowflake
Developed modular and reusable DBT models to ensure efficient data transformation, version control, and documentation
Optimized Snowflake queries and tables by implementing clustering, partitioning, and warehouse scaling strategies for performance improvement
Integrated DBT with CI/CD pipelines using Git, dbt Cloud, and Snowflake to enable automated testing and deployment of data models
Monitored and troubleshooted DBT runs and Snowflake performance using query profiling, logging, and materialized views to ensure efficiency
Built parameterized Coalesce pipelines that ingest from Snowflake internal stages (e.g., @INTERNAL_STAGE/...) and standardize data into curated tables using reusable YAML metadata
Managed version control with Git: feature branches, commits and tagged releases to keep lineage and configuration changes auditable
Deployed by promoting Git branches from one Coalesce workspace (Dev) to another (Prod) via Coalesce’s Git integration for consistent code and metadata across environments

Data Engineer

9/2023 – 7/2024

Migration multiple repos into one monolithic repo
Migrating from Azure Synapse Analytics to Databricks involves transferring and adapting data, workloads, and queries to the Databricks platform while ensuring compatibility, optimizing performance, and maintaining data integrity
Implemented and maintained an efficient GitHub pipeline to streamline the software development process, ensuring seamless integration, automated testing, and continuous deployment for project enhancements.
Developing new architecture for existing process

Data Engineer

12/2019 – 9/2023

In my data modeling experience, I've adeptly designed relational database schemas, defining entities, attributes, and relationships to ensure data integrity and optimize performance. I've also implemented data modeling best practices, such as normalization and denormalization, to support efficient data manipulation and analysis within various projects
Implemented Apache Airflow by creating and configuring an instance tailored to project requirements, utilizing various operators such as BashOperator or PythonOperator. Enhanced workflow efficiency by orchestrating complex data processes and integrating with external systems
Utilized Python programming for data manipulation, analysis, and automation, enabling efficient data processing and insights generation
Designed and optimized data warehousing solutions using Redshift and Snowflake, ensuring scalable and high-performance data storage and retrieval
Working on migration from AWS Redshift to Snowflake warehouse
Utilized Snowflake's innovative features, Snowpipe and Time Travel, to enhance data processing efficiency and enable seamless temporal analysis in a dynamic and collaborative environment
Implemented transformation framework to transform raw data from different data soruces to datawarehouse using Python
Implemented data validation techniques and quality assurance processes, ensuring data accuracy, completeness, and consistency using Python
Creating and migrating transformation and validation frameworks (in house tools) from a dedicated server to serverless using AWS microservices such as ECS, ECR, Fargate, Lambda, Glue and Cloudwatch
Leveraged Terraform for infrastructure as code, automating the provisioning and management of data engineering environments and resources. Using terraform to create EC2 instance and install Neo4j on this instance
Utilized S3 for scalable and cost-effective data storage, implementing appropriate partitioning strategies for optimized data retrieval
Using AWS S3 for large-capacity, low-cost file storage in one specific geographical region, because the storage and bandwidth costs are quite low. Retrieving historical data from S3, when disaster happened. Using Python boto3 and psycopg2 libraries to pull data from S3 and to import in Redshift
Created interactive dashboards and reports using BI tools such as Domo, Tableau, and MicroStrategy, enabling stakeholders to visualize and analyze data effectively
Implemented Data Governance practices using Atlan, ensuring metadata management, data lineage tracking, and adherence to data governance policies
Leveraged data masking techniques with ALTR to protect sensitive data and ensure compliance with privacy regulations
Created APIs for data integration and consumption, enabling seamless data exchange between systems and applications
Utilized HTTP methods (GET, POST, PUT, DELETE) and status codes to handle requests and responses effectively. Implemented CRUD (Create, Read, Update, Delete) operations for various resources through REST endpoints. Conducted API testing using tools like Postman or Swagger to ensure functionality, reliability, and compliance with specifications
Using BASH (shell scripting) to create different commands to check is previous process finished, is it finished successfully, how many files have been processed, ...
Collaborated with cross-functional teams to understand business requirements, translate them into scalable data solutions, and deliver projects within timelines

Data Engineer