Parimal Panda

I have 19 years of experience in data architecture, building big data pipelines using Spark and Hadoop technologies. Primarily, I've played a role in technical leadership, building a team from scratch. I've also worked across SaaS platforms, payment systems, managing full-stack applications, and on Big Data technologies.

I have 19 years of experience primarily in a technical leadership role, managing medium to large-sized teams. I have worked on various domains, starting from managing end-to-end SaaS applications with full tech, full-stack technologies, managing big data platforms using Spark and Hadoop technologies. I have worked across multiple domains, including payment systems and BFSI. I have also managed projects on compliance.

You can collect all the logs generated from multiple talent jobs into a Kafka system or Kafka cluster. And, from the Kafka, you can have different consumer services which can read those logs and upload them into Elasticsearch. You can build Kibana on top of the Elasticsearch where it can view all the log metrics.

Well, if you want to, process both the batch and the streaming data, that's a case where you should go with a Lambda type architecture, which can handle both batch data processing and also streaming applications.

Python comes with the same learning library, MLIP, which can be integrated within Spark, AWS pipeline.

You can have a job which basically compares the data in RDS versus the data you have in S3. So you can have a CDC job and you can have a threshold. So, you collect the metrics between that, and you compare it with the count. In RDS, if the count is higher than the count of records in RDS, if it's higher than what you have in S3, that means the data are not in sync between RDS and S3. So you have to run a CDC job to sync up the data. And while this happens, we also need to generate logs and metrics through which you can monitor the entire system.

You can do an end check process to ensure that the file, which is available in the CSV file that is in the source system, is complete before you write data into the SD/SDFS system.

You can have you design the CICD pipeline. So, basically, what you have to do is, you can have a Jenkins pipeline on a Docker-based system. So after a change is committed to the source code depository, you can trigger the pipeline to build, run unit tests, and deploy, and create a Docker image. That Docker image can be uploaded to a Docker registry, which can then be deployed into a target system.

You can use partition pruning so that you're not scanning the entire partition. You're reading the partition where the data is and you're only interested in that.