Rajat Gupta

I have 10 years of working experience, so I started my career as a Java developer for the initial 2 years. And for about the past 8 years, I'm working as a data engineer. So if I talk about my skill set, I have worked on Hive, Pyspark, And for real-time, I worked on Kafka. And other tools which I've worked on in the past few years are Databricks and Ocassion. I also worked on the DX tool. And other things like in DevOps, I am also involved in creating CICD pipelines and new infrastructure could telephone. So this is about my overall experience.

So if you need to optimize a PySpark job, which has two large datasets. So first, I will analyze and accordingly, we'll set the partitioning key so that there is no major shuffling happens. And the partitioning here, I will see if there is any data skewness or not. If the data skewness is still there after setting the required partitioning key, I will choose the sorting key then, in which I will uniformly distribute the data so that whenever I use the joins between two large datasets, my job will not get stuck, and since I'm using large datasets, I will be using a more memory-optimized cluster instead of compute-optimized. So in this way, I can optimize it. And also, some of the parameters in Spark properties while submitting, I will increase those and tune those parameters. And then accordingly, we can write the output to Snowflake.

So, basically, the process of tuning a PySpark application is like this. First, we can increase or decrease the number of Spark partitions. The default is 200. Second, if we're using a join condition and one of the conditions has small data, we can broadcast that particular data frame to a bigger data frame. The third way of tuning is, if we're using a big table join, we need to see if there's any data skewness by setting a partitioning key. If there is data skewness, we can use the sorting technique to optimize our PySpark application. Another way is to choose Spark properties according to our requirements.

I've not worked on a snowflake.

So we can monitor, whenever we face some failure conditions. If you're using AWS services, we can use the SMS service to send a notification whenever we get a failure condition, so we will get an alert on our email, and we don't need to monitor it. Just we can monitor our emails for any failure emails, and accordingly, we can rectify those errors. This is how we can monitor the ETL job failures and any delays. We can also add a delay condition in our program. If it doesn't meet the condition after this time, this will also send an alert.

So if we have to construct a serverless data processing pipeline, what we can do is, as soon as our data lands in our S3 bucket, we can put a trigger based on which Lambda function we use. We can get some preprocessing parameters and then use Lambda. If we are using AWS services, we can either trigger the Blue job, which will launch another serverless pipeline for processing. So in this way, we can design a serverless architecture using Lambda, Glue, or maybe if we need DynamoDB for metadata information. Those things are also possible.

So if you have to complete the data quality framework, so what we can do is we can create unit test cases using Python, PySpark, and then do the unit testing in our local environment only, so that whatever data is sent accordingly, we can pair test cases to check on our local environment only for any data quality issues.

So if we need to automate the backup process for a NoSQL database, one example is using DynamoDB, which is a managed version of a NoSQL database on AWS. There is an option where we can do the backup through the web itself, so it is automatically taken care of. Otherwise, we can do the manual backup also. At the end of each day, we can create a backup in a different region, so that we can access the data in case of a disaster.

So the strategy is, basically, to migrate the existing Python history cluster. We can add the multistrading capability through Python. First, we will create a generalized framework so that any ETL pipelines can be run in a multi-threaded environment. Whatever tables it will be running on, it will be running in parallel, basically. By creating a generalized framework and running in multithreading, we can create a more enhanced parallel processing capability.

So if we need to process the data in real time, what we can do is either use Lambda for some preprocessing steps or if we need to use a Kinesis, we can ingest the data using Kinesis also. And within Kinesis, we can process the data itself. So in that way, we don't need to go to the Lambda. Just all the data ingestion and processing can happen within Kinesis.

To design a high availability disaster recovery system, so we can create a multi-available tenant architecture. In which, it will be available in multiple regions so that there is no high availability option.