Nivin Srinivas S

Constructed efficient data pipelines using PySpark, DBT, BigQuery, and Apache Airflow, resulting in improved data processing speed and reliability. Developed Enterprise-Level Data Pipeline Template Repository Architected a comprehensive template repository integrating Apache Airflow with AWS Batch and Databricks for scalable data processing, enabling standardized deployment of both batch and data-intensive workflows Implemented robust monitoring system with PagerDuty and Slack integrations for real-time alerts, reducing incident response time and improving system reliability Established development best practices by incorporating pre-commit hooks with Black and Flake8, ensuring consistent code quality and maintainability across teams Containerized the solution using Docker and automated deployment to Amazon ECR, enabling seamless CI/CD pipeline integration and standardized environment management Engineered infrastructure-as-code using Terraform to provision and manage AWS resources (ECR, Batch, IAM roles, networking), ensuring consistent and reproducible infrastructure deployment across environments Spearheaded the development and productization of a data lake solution, providing critical data access that enhanced decision-making for customers.

Developed ETL pipelines using PySpark, Airflow, and BigQuery for seamless data processing. Engineered the backend infrastructure to efficiently manage and forecast AWS EMR and Databricks cloud costs, utilizing REST APIs, Flask, MySQL, and SQLAlchemy. Established a robust CI/CD system from scratch for the Data Science team, automating code deployment with Github Actions. Implemented a comprehensive healthcheck and reporting mechanism within the Unravel data platform, enhancing telemetry and system monitoring. Created REST APIs to deliver real-time critical budget notifications via Slack and email.

Responsible for designing and implementing a Data warehouse using BigQuery. Responsible for building ETL pipelines using Apache Spark. Implemented data pipelines for transforming and ingesting user events data in JSON to BigQuery for customer journey analytics. Set up Airflow and successfully migrated data pipelines and cron jobs to Airflow for efficiently scheduling and monitoring workflows. Created Dashboards and sales reports using Google Datastudio and Retool. Automated app deployments using Bitbucket pipelines CI / CD. Automated the process of onboarding new restaurant outlets using Python, Google APIs and Slack.

Built, tested and deployed chatbots for multiple clients of Yellow Messenger. Gained experience in building web applications using Python.