Ponithapunitha girish

Hi, I'm Punita, and I am a data engineer with 14 years of experience. So overall, I have 14 years of experience, and I have around 6 to 7 years of data engineering experience. So in my past projects, I have worked on various data pipelines, like onboarding data from sources, massaging and uploading transformed data to the Cloudera cloud system. That's pretty much about myself. So, I'm a multitasker and a quick learner. This is pretty much about myself.

Yeah. So in Python-based mod, ETL extract transformation load, the incremental would be based on the data which we would be getting if there is a batch processing, then we would be doing it with the help of the Hadoop MapReduce. But since it is a Python mod, so it is good to have Spark and build a Spark tech stack to be used along with Python flavors. So, we would use the Spark for our incremental data loads.

Yeah, so to ensure there's zero downtime, since we would be using Spark, which has in-memory computation logic, we would expect it to have 0 downtime during the ETL pipeline deployments. And also, we would ensure that the code gets deployed to multiple regions and will have a backup with a 3 times replication factor, so that it reduces downtime during future pipeline deployments.

For validating the correctness of any ETL process in any of the BI tools, we first determine whether the specific input location is providing structured input data. What we are getting is structured enough. So if it is not structured, we would then proceed with using the data cleansing process, which includes removing or adding delimiters, removing extra spaces, and modifying or updating the columns if necessary. That is one of the correctness approaches we would be following.

So the data integrity would be maintained within the transaction SQL database to S3 by ensuring that all the data has been uploaded properly and it is partitioned well using partition techniques and the data, which is like you know, optimized enough. This is one of the data integrity approaches, but we would follow that the volume, veracity of the data has been considered. And even in terms of security purposes, we would use principles like I'm not sure what this means, so let's remove the filler, and ACL, access control list, so that the data's integrity, privacy, and security are also maintained.

Yeah, so sparks partitioning and caching is used in a way that it can perform much better. And partitioning, in terms of partitioning, we would be considering coalitions as a better partitioning because it'll have less shuffling. And caching, in terms of caching, we would be considering persist so that you know, we can define the memory levels. So these are the ways we can make the performance of the job better.

So in this function, I just come across that it doesn't have any static kind of method, and also it doesn't return anything. So it is just a void method wherein it is saying that if the particular report type is HTML, do this. Or else if it is PDF, then generate the PDF report. But also, coming to say I see that it is enclosed with a backslash, which is actually not needed. So that is what the first glance I look into that. Yeah. Okay.

Okay, so in terms of batch data processing, if both of them are executed, since the TMap 1 is having row 1 and updating with T file output delimiter. But if you again execute T map 1, then it will get overridden with row 2 and T file output delimiter 2.