Rajat Gupta

• Designed and developed robust data pipelines using Spark and Hive to support ETL processes, ensuring timely and accurate data flow across multiple sources and destinations.
• Managed and optimized AWS infrastructure to support large-scale data processing, implementing best practices for cost efficiency, data security, and scalability.
• Led data modeling efforts to enhance data accuracy and accessibility, transforming raw data into structured formats suitable for analytics and reporting.
• Conducted performance tuning on Spark and Hive processes to minimize execution times, improve throughput, and maintain high performance standards.
• Worked closely with American Express stakeholders to understand data requirements, align on project goals, and deliver solutions that support their business objectives.
• Provided technical guidance to a team of data engineers, conducting code reviews, mentoring team members, and fostering a collaborative development environment.
• Implemented data validation and quality checks throughout the pipeline to ensure data integrity, resolving issues proactively to maintain high data quality.

Project Description: New data-lake using AWS Environment

• Created Data Ingestion Pipelines from various RDBMS databases to AWS S3 layer using AWS DMS
• Worked on creating ETL data pipelines consisting of PySpark, Glue Jobs, Lambda, DynamoDB, Athena, S3 with parquet files
• Built new infrastructure in AWS using Terraform & CI/CD Pipeline using Bitbucket and TeamCity

Tech Stack: Hive, Spark, Python, AWS (EC2, S3, RDS, Redshift, SNS, EMR, Glue, DMS), Airflow, Terraform

Project 1: Customer Care IVR Automation

Description: Automate the current IVR process to remove dependency on Customer Care Executive

• Handled Data Ingestion via Kafka for real time processing of the data with Spark Streaming for large scale data processing
• Involved in translation of complex functional and technical requirements into detailed high and low level design
• Integrated HSM API with Kafka to provide hardware encryption along with software level 256-bit encryption to secure
• Transactions like debit/credit card details along with integrating with REST API for encryption and decryption of messages
• Played a key role in transformation by Spark scripts for data transformation from structured & semi-structured data
• Performed analysis on data by implementing various machine learning algorithms using Spark ML
• Improved algorithms by deploying best hyperparameter, deployed GridSearchCV for tuning

Tech Stack: Confluent Kafka, Spark, Python, Machine Learning, IBM RTC, Jenkins, REST API, Web Services, HSM

Project 2: Datawarehouse Migration

Description: To create Datalake which shall be the single & comprehensive source of information to improve decision making

• Setup environment on AWS EMR for development purposes
• Oversaw data extraction from Charging System nodes Oracle Exadata to HDFS using Data ingestion tool Sqoop
• Analyzed data by Spark scripts to extract meaning and value from structured data

Tech Stack: Sqoop, Hive, Spark, Python, AWS (EC2, EMR)

Project 1: Charging System Tariff Plans Regression | U Mobile, Malaysia

Description: Constant update in tariff raises persistent need to track the impact on ROI which determines end user satisfaction.

• Developed PySpark scripts to calculate traditional & ad-hoc KPIs from structured & semi-structured data
• Managed Data ingestion using Sqoop, cleaning and manipulation of data using Spark scripts

Tech Stack: Sqoop, Hive, Spark, Python

Project 2: New Datalake for LTE Network

Description: To create new Datalake which shall be the single & comprehensive source of information for critical KPIs

• Performed ETL from ENIQ Oracle database to HDFS using Sqoop and processing of Data by Hive scripts
• Tech Stack: Sqoop, Hive, Shell Scripting

Project 3: Portal Development

• Migrated old dashboard written in PHP scripts to new dashboard using Java

Tech Stack: Core Java, JSP, Servlets, MySQL