Supreeth Gowda

Building and maintaining a modern data stack built using Fivetran, Airflow (MWAA), Snowflake, DBT, and PowerBI. Implemented configuration-driven data ingestion, data quality, error logging, and data orchestration frameworks leveraging AWS services. Designing, developing, deploying, and maintaining data ingestion frameworks for CDC, streaming and batch data using Fivetran, Amazon Data Firehose, AWS Lambda, DynamoDB, S3, and AWS Glue. Orchestrating workflows with Airflow, deploying via Terraform, and integrating PagerDuty for real-time failure alerts. Managing Snowflake with RBAC, optimizing cost/performance via dashboards for query monitoring, and enforcing guardrails on query execution. Participated in hackathons involving Amazon Bedrock/Chat-GPT for image recognition and web scraping data analysis.

Interacted with various startup teams to understand their challenges and provided consultation on building data pipelines. Brainstormed and evaluated potential product ideas, identifying opportunities to develop them into viable product companies. Collaborated with a hospital management team to address their data challenges by developing solutions using Python.

Handled, maintained, and supported ETL data processes for Redshift, transitioning to a Data Lake architecture and managing thousands of ETL jobs, ensuring data integrity and reliability. Offloaded and transformed data from Redshift to Parquet format using Apache Spark on EMR, loading it into Redshift Spectrum to implement a Data Lakehouse solution. Worked on an in-house built AWS Glue catalog sharing infrastructure to minimize data duplication and enhance data consistency, leveraging AWS Glue, Lake Formation, Resource Access Manager (RAM), and Redshift Spectrum. Managed and supported a centralized data warehouse handling 45-50 petabytes of data.

Designed and developed a comprehensive Data Lake on AWS using AWS Glue Jobs, Glue Crawler, Data Catalog, Athena, and EMR. Created PySpark code with RDDs and UDFs for complex data transformations. Implemented and managed computational workflows and data processing pipelines with Apache Airflow. Built an in-house data deletion framework for CCPA/GDPR compliance within the Data Lake environment using Athena's CTAS feature after doing a POC on Apache Hudi, Delta Lake and Snowflake.

Designed and deployed data pipelines using Python, PySpark, MySQL, AWS platforms, Ganglia, Ansible, NiFi, Docker, and GitHub. Conducted comprehensive testing of Spark processing using PySpark, Zeppelin, PyCharm, Jenkins, Perforce, and AWS platform. Utilized Python in Jupyter Notebook, Splunk, and AWS CLI for end-to-end validation of big data pipelines. Pitched, implemented, and presented five new ideas involving testing tools, Android integration, voice integration using Android and Python, and PySpark at internal hackathons.

Developed structured Java code using RapidMiner and Weka APIs to automate machine learning detector building and engineered complex features from foundational data sets. Created Java code to extract features from log files and relational databases for intelligent tutoring systems (e.g., ASSISTments ITS) and configured automated detectors to identify student behaviors such as boredom, frustration, confusion, and concentration.

Involved in requirements gathering and test plan preparation. Developed scripts using Web (HTTP/HTML), Web (Click and Script), and Ajax (Click and Script) protocols in HP LoadRunner, embedding C and Java code for error handling. Set up monitors using Sitescope to collect performance metrics on application and web servers. Conducted various performance tests, analyzed metrics and test results, and prepared reports. Performed tuning activities to identify and eliminate bottlenecks, enhancing overall performance.