Senior Data Engineer
Glencore Information ServicesAug, 2024 - Present1 yr 6 months
Architected multi-cloud data platforms (Azure, AWS, GCP), engineering ingestion and transformations with Synapse/ADF, Databricks, AWS Glue, and BigQuery. Established medallion architecture with Delta Lake and Unity Catalog on ADLS2, elevating governance and reusable data products. Standardized orchestration via Apache Airflow, decoupling workloads from Synapse; automated Spark job registration using Azure CLI wrappers. Built end-to-end CI/CD pipelines with GitHub Actions and Azure DevOps; containerized local development using Docker and shipped Python libraries through JFrog (Poetry/PyPI), reducing release cycles by more than 50%. Enhanced resilience with fault-tolerant designs, Python Function Apps, and Logic Apps for alerting, improving SLA adherence to more than 50%. Advanced BI by modernizing Power BI and Tableau assets; enabled self-service insights, cutting time-to-decision by more than 50%. Authored HLD/LLD using data modeling best practices; templatized Synapse artifacts with configuration-driven parameters to boost maintainability by 100%. Integrated SharePoint via SMB to ADLS2 and standardized Delta tables with Serverless SQL distribution; partnered with CloudOps to institutionalize Airflow. Delivered proofs of concept with Microsoft Fabric and integrated GCS into the ingestion framework; designed an AWS stack (S3, Glue, Lambda, Kinesis, Athena/Redshift, IAM) to scale to 2 million records/day. Planned initiatives, refined estimates, and supported BAU, reducing incident MTTR by more than 50%.