Implemented business logic in Spark to write efficient jobs for indexing landing data and transforming staging data as per business use case for reporting purposes. Optimized existing Spark jobs/flows by implementing various spark optimization techniques, thus reducing run time by 40% and cost by 60%. Managed large datasets, performed data modeling, and conducted performance tuning to ensure optimal data processing. Transformed data within the pipeline for optimal processing and displayed it on user-friendly dashboard. Utilized Object-Oriented Programming principles to design and optimize an end-to-end ETL (Extract, Transform, Load) pipeline for batch data ingestion, transformation, and visualization on dashboard.

Consolidated data from four databases of different regions and implemented it from scratch in new web application. Collaborated closely with clients to define the user experience and scope of work strategy for projects.