Avinash Barde

I wanted to join Gather AI because it's one of the growing companies. I'm currently in my current organization, doing 30% back-end and 70% DevOps work. I wanted to be fully in the DevOps field, so this role and all the skills it requires match my skills. I'm looking for a particular role that aligns with my skill set, and I think this one fits that description. I wanted to join Gather AI.

database indexing works there are different ways to say database indexing is a technique and indexing is a technique used to optimize query performance by reducing the amount of data scanned during data retrieval. indexes work similarly to an indexed book, allowing the database to quickly locate a particular row without scanning the entire data set. So, how it works is first we need to create an index. When we create an index on a column, the database stores the values in that column into a sequence of a data structure. Without an index, the database performs a full table scan, but with an index we can directly point out that particular index and return that particular row. This speeds up the query, making a full table scan unnecessary. Index searching uses binary search or a binary tree or hash-based lookup, which is much faster in most cases. There are many different types of indexes, like single-column indexes or multi-column indexes, also known as composite indexes, and unique indexes.

So for keeping a system secure, we need to apply authentication and authorization over it. So that only authorized users can access that particular system. Also, inside a system, based on authorization, users can access particular components. This is one way. Another way is to provide a gateway or load balancer, or a gateway, in between, so that it will be one layer between a system and external access. We can also deploy a particular system into HTTPS so that communication between client and server will be encrypted. We can use some encryption algorithms. If that particular system is a containerized system, then we can scan all the containers with the help of some DevSecOps tools, like Tri-V and BlackDock, and then only deploy that particular container so that it will be a vulnerability-free system. Vulnerability scanning can be applied, and we can integrate all the different tools into our existing CICD so that our system will be secure.

What are the challenges you faced when trying to autoscale Kubernetes clusters, VMs, or EC2 instances horizontally? So, there are different factors or challenges I faced when adding scalability to virtual machines, particularly when scaling horizontally, which involves adding more VMs to the cluster. In that case, I faced multiple challenges, including networking and load balancing challenges, ensuring proper network operation and reliable load balancing across all the VMs. This is one challenge I faced. Also, in the case of distributed data, we faced challenges in managing data consistency between multiple VMs, especially when there's a stateful application involved. In that case, we faced this challenge. We also faced a challenge in the orchestration of the VMs, which involves dynamically adding or removing VMs from the cluster, requiring automation or coordination. Another challenge we faced is the scaling bottleneck. Horizontal scaling might not eliminate all bottlenecks, particularly in system components that cannot be scaled horizontally. Scaling bottleneck or performance and latency issues can be introduced when distributing the workload across multiple VMs, sometimes resulting in overhead and latency. Additionally, managing efficient autoscaling and scaling can be tricky.

Managing DB changes that developers want to deploy to production is a critical aspect of RDevOps. So we want to follow some CI, CD, or we can say this is typically managed through some change management or continuous integration and continuous development pipelines along with other best practices also involved. So we need to establish a change management process, for that we can use a change request process via Jira or ServiceNow, then approval workflow, and then use a version control system, such as Git. Then we will test changes in a non-production environment, and then implement a CI, CD pipeline. First, we will test particular changes in a non-production environment. Then we will implement CI, CD for production. Implementing CI, CD will involve multiple things: the CI part will automatically build and test the application whenever changes are committed. The CD part will automatically deploy changes to a staging environment and then to production according to your request. For deployment, we can use different strategies like BlueGreen, Canary. We are mostly using the BlueGreen strategy. In BlueGreen, there will be two identical environments. For example, one is the live environment, and the other is blue or green. All the changes will happen to the green environment first, and then all the traffic from green will be moved to blue. Our blue environment will become the latest environment, and after that, the green environment will be updated.

When choosing between hosting your application in Kubernetes and VMs, I consider several parameters. First, I need to determine if the application is a microservice architecture or a monolith. If it's a microservice architecture, I'll divide it into different small functional functionalities or services. If it fits, I'll deploy it on Kubernetes. Otherwise, I'll deploy it on a VM. For example, if the application has only one functional code or one functional service, a VM would be a suitable choice.

We need a route table in the networking context. It's used to determine where to forward traffic based on the destination IP address. In a networking context, if I say like the analogy of a route table is like a map that helps routers decide where to send traffic. We need a route table in contexts like load balancing, where we use it to distribute traffic across multiple nodes. For instance, a load balancer like NGINX will use the route table to forward traffic to specific nodes based on the routing information in the table. This helps in efficient traffic redirection.

So, here in this example, if you see VPC security group IDs, over there they have hard-coded that particular VPC security group ID, which is potentially a security risk. We can't hard-code this type of thing. Instead of that, we can store that particular ID inside a VAR file or provide it through HACCP or PORT or through any cloud-native secret store like parameter store. Likewise, we can provide that particular value instead of hard-coding it. This is what I identified. Any other issues, I've found this one.

Deploying a multi-tier application using Terraform in both AWS and Azure environments, ensuring high availability, involves several steps. First, we need to architect the multi-tier application, which consists of a front-end, back-end, and database tier. For the front-end and back-end, we can use load balancers to handle user traffic, and for the back-end, we can expose APIs that are only accessible to the back-end, while the front-end can only access those APIs. For the database tier, we can use cloud network database services such as Amazon RDS or Azure Database, which can be replicated across availability zones for high availability. To achieve high availability, we can deploy the front-end and back-end layers across multiple availability zones. Using Terraform, we can manage the infrastructure as code, and it supports both AWS and Azure providers. We can create separate modules for both environments, which are reusable components that can be managed through Terraform workspaces. This allows us to handle multiple environments through Terraform workspaces. Additionally, we can deploy the back-end tier as a microservice or API, and in the case of Azure, we need to set up a VPC. Finally, we can execute Terraform commands on a particular workspace to deploy the environment. In Terraform, we can create separate modules for AWS and Azure, which can be managed through workspaces. This approach allows us to handle multiple environments, including AWS and Azure, and ensure high availability in both environments.

To deploy an Azure-based application with high availability and minimal latency, we need to design an architecture that can distribute application components across multiple Azure regions and availability zones. This ensures the application can handle failure and maintain performance under high load or adverse conditions. To achieve this, we can use Azure Traffic Manager for global routing and load balancing. For regional redundancy, we can deploy application components across regions and zones. Additionally, autoscaling can be applied on Azure VMs or Azure App Service to handle varying loads. For database replication, we can create multiple replicas to ensure high availability. We can also use Azure Redis caching to reduce database load and improve performance. To manage high traffic, we can use Azure Front Door for global traffic distribution and load balancing. The compute layer will consist of App Service and VMs. Monitoring and alerting will be handled by Azure Monitor and Azure Application Insights, which provide metrics such as response time, latency, availability, error rate, and CPU and memory usage. These metrics can be used to create alerting or reporting across the globe. We require a global traffic distributor, which is available in Azure. This architecture ensures high availability and minimal latency across the globe.

Okay, now, so GDPR, okay. So while deploying any application, we need to follow some standard compliances or standard compliance we use, like we can't reveal any information which is sensitive or organization-related information, or any information that will potentially expose a client or something. That kind of information we can't expose outside the organization. Also, some of the secrets, some of the tokens, all those we will need to use either inside as an environment variable or store that particular secret inside a cloud-native secret store or in case of Kubernetes, store it as a Kubernetes object called a secret. We can store it over there. Also, not revealing any client's information to the outside, that's one point.

I have experience with setting up a distributed email interface on a platform like AWS SageMaker. Now, setting up a distributed machine learning interface on a platform like Enhance, leveraging deploying on to SageMaker, Enhance leverages cloud infrastructure to train large models or our database across multiple instances. So, I will explain the step-by-step breakdown of the process based on my experience with AWS SageMaker. So, first, we will discuss what AWS SageMaker provides, which is a managed platform to build, train, and deploy machine learning models. When it comes to distributed learning, SageMaker enables us to train models, scale horizontally, or use built-in algorithms. There are some built-in algorithms like those in TensorFlow and PyTorch. So, let's discuss the steps to set up a distributed machine learning interface on AWS SageMaker. First, we need to create a SageMaker environment. There is a concept called a Notebook. First, we need to create that Notebook, which has computation logic to run a particular model or execute a particular model. Then, we need to prepare data for distributed training and choose a particular distributed training approach. We need to set a job, which will decide which algorithm to use and all. This is how we can optimize it for distributed training.