Rushabh Doshi

So my name is Youshro Sabdousi. I have completed my graduation in 2011. After that, I prepared for the GATE examination for one year and created the GATE exam in 2012 and pursued my master's. I completed my master's in 2014. And after that, I joined the VEP Engineering College as a contract-based lecturer along with freelancing work. I did the freelancing work from 2016 to December 2019. In between, I also worked as an academician, from a contract-based lecturer in one of the engineering colleges, which is VVP Engineering College. After that, I continued my journey with freelancing. I did projects in Laravel, Windows application, Python script, and Node JS script. After that, in the COVID era, I joined Improvise as a software intern where I learned Node JS, Blue JS, Nuxt JS, Golang, Python's Scrapy framework, GitLab, GitHub, Dockerization, and Kubernetes config for the same, and CICD pipeline for the same, like build, test, and deploy config file. Then in January 2021, I joined Improvise as a full-time software engineer, where I handle and maintain various PHP-based, Laravel framework-based applications, like five applications are there. Along with one project, I also used a Python Scrapy to scrape reviews from around 30 to 35 different providers. We have used the Python Scrapy for the same, and I have deployed several spiders for the Python Scrapy one. Apart from that, I also worked on one of the tempests of a custom browsing project where I used the Fast API with Dockerization and customized Wikipedia parsing for the custom browser application. Apart from that, I also did several Kubernetes-related tasks in LiveCode and used Google Colab to scrape data from a one-year website. After that, in January 2023, I joined Log Binary as a senior backend engineer where I worked on one project table, which has the Django framework-based and has multiple databases and multithreaded along with the Selenium environment. And that project is in the Django Rest framework with Docker images and all that, where I mostly work with API integrations, our modules, and deploy modules. After that, I also worked with Alliant Healthcare, one of the Django REST API-based applications, which has a backend with the Django Rest framework and provides the organization to assign patients.

To set up a secure endpoint, I will discuss my live project, Qt Healthcare, where we have used the API gateway for that. The API gateway will authenticate the user with the JSON web token and authenticate the user with Firebase. After that, the particular app token is also there, which would be inserted by the API gateway whenever the user is authorized. And then, that API with the app password and authorization, user UUID key, and authorization password will be entered. The API with the user UID and application key or secret key would be added into the headers and sent to the Django API. So this way, we will secure the rest API. We have used the API gateway, which will authenticate the user and then add the app key and user UID. So we will also make sure which user made this API request.

So in this way, optimize the Django application to increase the volume of posts in the database. In my live project, I used a tableless approach with MongoDB, along with PostgreSQL, to reduce the read requests to the PostgreSQL. That would be the frequent read triggers to read data from PostgreSQL. To address that, we reduced the read requests by saving the data into MongoDB, and whenever necessary, we read from MongoDB. And, we also maintained the maximum connection limit and used the default connection policies provided by the Postgres library. If any connection issues arose, we closed the idle connections and called the close connection. This would close the idle connections and make a new request.

So for real-time data processing with a Python service, there are mainly two to three services. Other Lambda services, step services, and step functions are also there. We can leverage all those things. We can use CloudStack or CloudFormation to deploy that service architecture in AWS, where we can use SQS for message transferring and SMS for notifications and Lambda for whenever the data is available, we can trigger the Lambda function to perform a certain task on it. And apart from that, if for the step function that would run based on the trigger event or the workflow that we have set with a clone job. So that would be possible in that case. We can design the service architecture with the Lambda function. The Python service architecture with the Lambda function for real-time data processing, whenever there is a file uploaded to an S3 bucket, then the trigger will trigger an event to the Lambda function, and which will take that file and convert it into various different resolution formats and save it to another bucket, so that we will use that particular different resolution file for different displays.

For managing the database transaction in Django, mostly, we are dealing with this kind of thing in a multithreaded environment. So for that, we can use the atomic transaction decorator, which will help us to make sure that only one thread is active in the function while performing the database transaction. So the atomic transaction and other atomic transaction decorators would be beneficial in that case whenever we're dealing with database-related activities in a multithreaded environment.

What consideration? Okay. So for consideration, the annual event scaling the Python-based API horizontally, in that case, if we use a step function or if we use the lambda function, that will be a serverless application. A step function is also a serverless application. We can leverage it. And for that, the infrastructure scaling and all that thing, would be maintained by AWS itself. So we don't need to worry about it, but we can also have cloud analytics, which will analyze our incoming and outgoing data and incoming requests, and it will scale accordingly. In the horizontal case, it will deploy EC2 instances whenever necessary, and it will scale down the EC2 instances whenever there are no requests.

So here, we need to put the decoration. We can leverage the Python function tool LRU cache with the maximum size. Here, the size would be specified, so that would also prevent a memory leak or out of memory bound. And, also, we need to set the time out limit for the each and every cache function. So once the cache function will be saved along with the timing one, and if all the timing that contained the particular cache data, if the cache data for the particular value having some threshold time limit, then we need to calculate it again. So that in that way, we will update with the particular data. In that case, we will update it, and we will response back with the updated value instead of having the old value in the cache. There are two cases. We need to store the time stamp when the particular data would be cached into the function and whenever that data would be called again, then we need to check whether it is present in the cache one, and it would be there in the last 5 minutes or 5 seconds. It is not more than the 5 minutes and 5 seconds old, then if it is so, then we need to calculate and compute again the new value. If it is not, then we will just return the cached value so that in that way, we will add the timestamp mechanism along with the caching of the value so that we will response back with the updated value instead of giving the old value. And, also, we are not setting here the caching limit or memory limit, which we'll also call the memory out of bound error. So we need to also add the cache limit for the particular number. Like, the length should not be greater than 64, and all that. If it is there, then we need to use another cache mechanism, and we can leverage the LRU cache mechanism for the function tool for the same.

If this function would be called multiple times, then there will be the possibility that multiple processes could open the connection or the connection would be refused. So there is no other mechanism for this one to grab the data from it. And while executing this cursor execution, if we use the Django one or any other thing, then we need to use the Django query set instead of running this execute row query directly. So SQL injection, that will be also possible. So we need to avoid SQL injection if it is possible. And regarding the connection one, we need to make sure that we can use the existing open connection if it is there, and this connection as DB name, user, password, and host would be there. But if it is required any connection security mechanism like any protocol that we need to use it, then there would be the problem in that case. And if it is not connected, then data mechanism is also not there. And finally, if the connection would be closed, that would be also there. But yes.

So we can use AWS Kubernetes config, which is EKS, Elastic Kubernetes Service. It will continuously check the app status with the heartbeat and have a minimum of two to three containers, or ports, running with the same Django application. If one of the Django application instances clears, then we will immediately redeploy the new port. And we need to check the CloudWatch log to see what caused the Cheng web crash, and we need to fix it. In a zero-downtime deployment strategy, we can use a gradually increasing deployment by deploying new changes in a partitioned way, like, if there are four nodes already running with the Django application, then we deploy two new ports with the new implementation, and the other two will remain as they are. After that, the two will be deployed fully. Then we deploy the tenant, and then we close the old ones and replace them with the new ones. So that would be the proposing downtime deployment strategy that I have experienced.

So Python AWS function for that, the event trigger, that I need to identify the triggers and set the proper IAM role for that Lambda function 1. So that no more permissions than what is required would be granted to the Lambda function. And, that particular Lambda function, in the Lambda function deployment, we don't need to think about the scalability and reliability issues because all the infrastructure as a code things would be handled by AWS itself. So, the number of requests that we can make with the Lambda function 1, that would be only considered and the computation time for the Lambda function that would be considered in the financial calculation of the cost, so that it would be much more easier, much more, we can say cheaper contact, compared to we need to deploy the ECS instance for that or EC2 instance for that.

So the advantage of using a build pipeline for Django would be helpful to upload every new image of Django with the updated one. And for that, we can use cloud build to load our DML or app build appspec.yml. We can start building a cloud build pipeline with the cloud format and cloud stack 1. So, the cloud format will contain all the necessary services that need to be taken care of. And whenever any AWS application is made to the Django application from GitHub or from an AWS commit, then an automatic trigger will occur, and it will generate the Docker image. After that, the Docker image will be deployed into the node using EKS like Kubernetes services. If there are 5 nodes, it will deploy 2 nodes at a time, and the 3 nodes will remain as they are with the old deployment. Then, the 2 nodes will be running fully, and it will go for the next 2. In that way, we can maintain zero downtime and automate the build timeline using cloud stack or cloud format, like cloud build auto or cloud build pipeline. We can also use Jenkins and the AWS cloud build thing that will take care of all the continuous integration and continuous deployment.