Senior Platform Engineer
QuantiphiAug, 2024 - Present1 yr 8 months
Configured SSL certificates for Dataflow and Datastream pipelines, ensuring secure data transmission for ingestion from Oracle to Google Cloud Platform (GCP). Utilized Terraform as Infrastructure-as-Code (IaC) to automate and manage the SSL certificate configuration and Github Actions for CI/CD. Designed and built scalable platform services and self-service tools, including CI/CD and observability enhancements, primarily using Python and the relevant libraries. Streamlined and secured data transfer by implementing best practices for certificate management and pipeline configuration. Managing GCP platform operations and infrastructure as code (Terraform), overseeing CI/CD pipelines, coordinating with stakeholders for platform upgrades and root cause analysis, and evaluating global and local solutions across diverse scenarios. Configured and deployed a managed Terraform PAC pipeline using Tekton, automating infrastructure provisioning and CI/CD processes. Set up an Infrastructure pipeline for BigQuery (BQ) and Cloud Storage using Terraform, ensuring seamless and scalable data handling. Developed and deployed a data validation tool, integrating it with Cloud Run and utilizing Artifact Registry to host the application, ensuring efficient and reliable data validation. Wrote SQL queries and Python scripts to load historical data from Teradata into BigQuery, leveraging BigQuery Data Transfer Service (DTS) to automate and schedule data transfers. Conducted a thorough review of IAM security for the project, ensuring proper access controls, role assignments, and resource security across the infrastructure. Designed and implemented architecture solutions for multiple small-scale Proof of Concept (POC) projects, evaluating different GCP tools and services. Leveraged various networking tools within GCP to optimize cloud infrastructure, improve connectivity, and ensure secure communication between services.