Brahmabit Mahapatra

I am a software engineer with more than five and a half years of experience. In total, I have been working with multiple startups across multiple domains, including hospitality, health care, as well as briefly in e-commerce. I have gotten a chance to explore a lot of languages, tools, and technologies, which includes blockchain, low-code application development, BPM tools, and, of course, native application development using the Node.js stack, such as MongoDB, Angular, Node.js Express, as well as MongoDB React, Node.js, and Express. So I have built scalable solutions, which are being used by more than 10,000 users handling over tens of GBs of data every single day. And the peak usage for one of the applications that I had built was almost like 100,000 users, which translates to around 1,000 users every minute. So mostly, my experience has been on the backend side, using Node.js. I do have a fair bit of capability on Python as well. On the frontend side, I have worked on Angular briefly as well as on React. And on the DevOps side, I have been working with all three major clouds: AWS, Azure, and GCP. Primarily, I have worked a lot on GCP and Azure. I have a fair bit of understanding of Kubernetes, deploying infrastructure as a code, and I strongly believe in system designing. So I have been doing that for quite a bit now, especially high-level design and low-level designs. And I do have a fair bit of understanding of microservices, and I have been developing all my applications from scratch using the microservice architecture. Thank you.

A Node.js backend application could be potentially divided into 3 or 4 parts. The one is, let's start with the data part. So we need to create our data access objects so we could use an ORM, something like Mongoose, to connect with MongoDB. And we can create our DAOs or data access objects over there where we could have helper classes to access the databases. So, basically, all the logic for accessing databases could be potentially done under that. Then, there could be a layer on top of it, which handles most of your business logic and everything. So which would be something like the service layer, let's call it. It will contain all the business logic and other details over there. Then on top of it, we could have our handlers, which would potentially incorporate these business logic and, like, connect the interfaces of our DAOs and business logic. And finally, on top of it, we'll have our routes, which will be using handlers to create the endpoints on the API. Now, coming back to the way I'll be designing things, properly structuring things, so for MongoDB per se, I would separate out or create a singleton for MongoDB connections, which would be used globally. I use OOP's concepts, like, on a daily basis and follow the proper design patterns. So creating using ORMs or creating models for validation is something that I would do to handle a Node.js and MongoDB combination. And all my business logic could have utilities and helpers associated with their service logic or the business logic. So that is how I would design my application.

For securing sensitive data on a MongoDB application, there could be a few ways to address this. One is, of course, to use SSL certificates between Node.js and our cloud service application to prevent any kind of attack. The second is to have proper access controls on the MongoDB side to prevent unauthorized access to the app and the database. Initially, we could store this data as clear data on the MongoDB side. And on the application side, while serving the data, we could mask that data and provide it to the end user. If those things don't work, we could store encrypted data on MongoDB and decrypt it every time we want to fetch data. So that is one of the approaches that I understand and I know of. There may be a few other approaches, but this is what I know at this point.

Which Node.js tools and libraries do you think are most useful?

To integrate error tracking and monitoring tools into Node.js back in deployment on Azure, there are multiple ways to do this. Basically, if I'm using Azure log stacking or log analytics, what I need to do is either use the Azure SDKs to add these things, add the monitoring tools, or have an endpoint which could be called every time an error happens, or write a utility for it. And every time something happens, it pushes that data onto Azure. But coming back, if I'm not using an external logging service or I am using an internal one, per se, let's say I start storing my log data on MongoDB. I'll be creating a utility which adds or fends the logs to our MongoDB database every single time something happens. If that solution is not viable, I could always go with something like a file system or a file write to a file system, and then I could pick that file and process it using an ELT stack or anything. But the problem with a file system would be if we are using a distributed deployment environment, those file systems could potentially be a bottleneck.

So especially, while using or designing a distributed system, for what I would do to ensure consistency of my data is I could follow certain patterns like the saga pattern, which would help me and allow me to track everything that I'm doing, or create basically an orchestration or a choreographed feedback for the data consistency part of it. And everything would be associated with event sourcing, which would help us analyze or deeply that entire thing to prevent any kind of issues in the data consistency. But if we have to make our MongoDB asset compliant, what we could do is there is a possibility there is something called sessions or transactions, which allows you to write multiple writes to multiple collections on MongoDB and then complete that session or transaction. If anything or any failure occurs, it would roll back all the previously done operations. So that is what will help us ensure consistency of data on MongoDB. But to ensure consistency of data models is something we could use an existing ORM, or we could use some kind of validated libraries where we create the models and everything on Node.js. I don't remember any particular library for Node.js, but for example, in Python, you have something called Pydantic, which will help you create a model. Then we can integrate it with Mongo, PyMongo or something like that.

So, this particular application creates an express app. The only thing which is missing right now is that it is not exposed yet on any kind of server. However, what this part particularly does is it has created a middleware which validates our authentication token. If the authentication token is valid, it proceeds or allows you to use your endpoints. If the token authentication authorization fails, it sends back a response saying the authentication token is failing. If at all possible, right now, what happens is it basically checks if this thing is checking or not. The return value of authentication not happening is not there. But if the only thing that middleware does right now is it checks if that token is available or not. If yes, it proceeds for validation, there could be certain returns over there as well. But if that token is not there, it basically sends back a 403 response asking for an authentication token. And finally, if all those requirements are sufficient, it proceeds to the endpoint, which basically sends back "hello world." Over here, we could ensure better error handling and performance by adding certain try-catch blocks if things are failing or if it is not going in the way that we would like it to. And there could be potential problems like call stacks and other things on this API as well.

This has an entire phase, high depository where you have to get on, get my ID, and you have the DB context. This is kind of following my previous part, which is following dependency injection at this point, particularly when that repository class is being initialized. It is taking my DB context as the dependency over here. Then, the pattern being followed here is dependency injection. We also have the abstract class of the repository, which has an interface for insert, update, delete applications. Once this repository class has initialized, the details for each of those abstract classes, that is insert, update, delete functions, are added. This repository could be initialized where we pass our database helper, let's say, or database context. And then, we can call any other functions that we might use. Ideally, the DB context should be a singleton pattern. That is the only thing I think.

How do you architect the system? If we want to have millions of operations happening through MongoDB, or through our Node.js application onto MongoDB. The application should be horizontally scalable. We can't have vertically scalable limits because it has certain up limits and potentially very expensive. But what we could do is we could potentially horizontally scale this entire thing so that it allows multiple concurrent requests to come into the Node.js application. Then, similarly, we would need to scale our MongoDB horizontally, having multiple instances in that particular cluster. And, of course, we need to enable sharding so that we could potentially store a lot of data across these individual instances. Now, finally, apart from that, we need to have purge mechanisms or archiving processes in place so that we don't exceed the limit of the database at any given point in time. Or we need to have auto scaling enabled on MongoDB. And all of these need to be on an orchestration layer so that auto scaling happens easily. We don't have to manually scale it. So as the load increases on Node.js, MongoDB for that matter, it should automatically scale. That is how I would try to handle a lot of concurrent requests. That would be the starting point. Maybe the exact process might require some additional processes and other things in place, but at this point, that would be my approach.

Environment configuration could potentially store things like database URI, then username, passwords, any other third party endpoints, any other variables that you might like to be configurable. And this could be potentially set for each of the individual environments that an application is running for, let's say production environment, pre-production, or staging, and finally production. So that is how I think environment configuration would help manage different stages of a Node JS application life cycle, but I don't exactly know how this works.

Yeah. One of the major things that I did was to implement caching. So I created a pass-through cache for our applications, especially while the transactions were being written to our MongoDB database. So, it put it basically, it basically wrote your data to our Redis DB. And then, from that data, we created events to write it to our MongoDB application to our MongoDB database. So what it did was it potentially reduced huge latency on our application. So it basically took down the response time for our API, one of the APIs, from around 900 to 50 milliseconds to 1.25 seconds, or something like that, from that point to around 300 milliseconds. 200 to 300 milliseconds for that particular API, which had a lot of things being written to the database. That particular API. So that is what I did, which enhanced the application profile. And apart from that, I took people at a very high level, for application and performance. Try to use multi-threading over here. So, like, adding worker threads as well and also running it as multiple processes on multiple cores. So both of these approaches could potentially enhance the application performance quite a bit.