Aiswarya K S

Built and maintained data pipelines using Databricks for batch and incremental processing. Scheduled and monitored workflows using Apache Airflow. Set up GitHub Actions integration with Databricks. Migrated data pipelines from SnapLogic to Databricks. Performed Databricks Serverless migration to improve scalability and reduce maintenance. Monitored production pipelines and resolved job failures. Investigated and fixed data and pipeline issues in production environments. Supported other teams in building and troubleshooting BDX workflows and collaborated with cross-functional teams to ensure reliable data delivery.

Built and optimized scalable data pipelines using Databricks and Google Cloud Platform, ensuring efficient data processing for analytics workloads. Orchestrated end-to-end workflows with Apache Airflow, automating complex dependencies and improving data delivery timelines. Integrated new business logic into existing codebases, enhancing dashboard functionality and aligning outputs with evolving stakeholder requirements. Converted SQL scripts into Spark code to improve data processing efficiency. Developed automated data pipelines using DBT and Snowflake to power and refresh Omni dashboards with up-to-date insights. Gathered business requirements from clients and converted them into actionable tickets.

Optimized Data Pipelines: Designed and developed scalable, production-ready ETL pipelines for data ingestion and transformation, ensuring seamless integration with BI applications. Automated recurring processes for data extraction, cleaning, and transformation using SQL, PySpark, and Shell scripting improving reporting timelines. Migrated Hive scripts to PySpark, enhancing performance and efficiency, and automated data copying between prod and stage Hadoop clusters. Migrated existing Cron jobs to Tidal Scheduler, ensuring reliable and timely execution of data processing tasks. Achieved Databricks Certified Data Engineer Associate certification, contributing to cloud migration projects. Analyzed job failures, implemented preventive actions, and introduced monitoring solutions to maintain stable data workflows. Built and maintained ETL pipelines using Alteryx to ingest, transform, and load data into AWS Redshift, ensuring seamless data flow and integration. Implemented robust data quality checks and optimized SQL and Python scripts for data cleaning, enhancing data accuracy and usability. Designed and implemented end-to-end BI solutions using MicroStrategy and Power BI, including advanced DAX queries and interactive dashboards for actionable insights. Developed multiple regression forecasting models and implemented machine learning models (Isolation Forest, Prophet) for anomaly detection, achieving high prediction accuracy. Created MicroStrategy Dossiers, Documents, and Power BI dashboards (e.g., Decomposition Trees), enabling detailed analysis and data-driven decision-making. Automated data ingestion from S3, Airtable, and Adobe Dashboards using APIs and Alteryx, integrating the data into Redshift for reporting. Led client discussions, provided proactive mentorship to junior team members, and ensured smooth project delivery.