Dhananjay Pandit

Hi, my name is Dananjay Pandit. I have been a senior software engineer, working at Axon for about five and a half years now. My background includes undergraduate studies from the University of Mumbai, and a master's in software engineering from Arizona State University. I've been working with Axon for about five and a half years now. I started off as a software engineer, and I've been promoted to a senior software engineer while working at Axon. A little bit about my current role is that I'm on the records management team, which helps build a cloud-based reporting offer for law enforcement officers. I am currently looking for a new job since I'm moving from the United States back to Mumbai, India to be closer to family. I'm looking for new job opportunities that are either based in and around Mumbai or completely remote. That's about it. My technology stack is as follows: I have used Golang extensively at Axon for the last five and a half years. All of my undergraduate and graduate studies used Python as my primary language, so I'm very familiar with that. I have recently finished the Azure Cloud course on Udemy. We use Azure heavily here at Axon, so I knew about some of the features with some hands-on experience. However, doing the course has helped me get a lot more insights into some of the specifics, including the internals and workings of Microsoft Azure. That's a brief introduction about myself.

So which bill concurrency patterns would be most beneficial for handling multiple video streams simultaneously? I don't have too much experience with video streaming specifically. Our company's application is cloud-based and does not involve video streaming. However, I do have experience with two concepts, goal line, which I feel would be pretty relevant for this answer. First, I think the use of gRPC as our API with Go would be beneficial when paired with Go concurrency patterns. It would be nice to have an API that streams responses, including video streams, back to our clients. In terms of which Go concurrency patterns we would use, I feel like channels would be a great answer to that. We would be able to do error handling very easily using channels. We could also defer error handling in case nothing comes up, and the channel can let us know if it finds anything concerning. So I feel like that would be a really interesting pattern to use for video streaming. GRPC streams combined with channels in Golang would be really powerful, and I feel like it would be able to do the job. Any issues that I see, not necessarily. I cannot think of anything right now, but that could also be because I don't have super close knowledge of the workings of video streaming specifically. But I think that would be my answer. GRPC streaming APIs coupled with Golang channels.

AWS Step Functions in the orchestration of media processing workflows would be beneficial for orchestrating workflows where you want to execute a series of tasks in a specific order. I think that's the main question. And whenever we want to do any sort of orchestration within our application where first you want to do this, then you want to do that, then you want to do that, whatever you have a workflow, so to speak, that you need to build into your application, that is when the step functions are really going to be beneficial. Likewise with function apps, where you want something to be transient. You don't want it to live for a very long time. It is a step to a larger means. So you would want it to do that step and then just go away. So I think that is where I would use AWS Lambda, for example. But I think the step functions give us an additional benefit on top of it that you can orchestrate your workflows as part of it. So you can say, first do this, then do that. If something goes wrong, you can have fallbacks in place so that you don't completely fail, so you can gracefully handle failures as part of a step function as well. So I would use it for orchestration mainly around things that need to be transient, however, still need to be tied to a singular workflow. Cron jobs or something like that is something that immediately comes to mind. So, if you upload, for example, a new media content to our servers, step functions could be a great way to ingest those. So step one would be to support the whole uploading of the entire file. Step two can be where you extract the metadata out of those files. And then step three can be like elastic search ingestion for those metadata for that video. So, yeah, that's how you can orchestrate those things using AWS Step Functions.

Oh, in Kubernetes, how would you ensure zero downtime deployments for a live video streaming service? Zero downtime is definitely a challenge. I would say we could achieve that by asking follow-up questions to clarify the requirements. What does it mean to have a zero downtime deployment? Are you just talking about deployment specifically? Because in that case, that is a much focused answer. I'm going to assume that is what is being asked since the question specifically mentions 0 downtime deployments. There are a couple of ways to do things. One way to do it would be to gracefully terminate the connections and the pods that are going away as part of the deployment, and then spin up new pods in its place after one of them has gracefully shut down. By graceful shutdown, I mean the connections being handled by the pods that are going away should be shut down gracefully. Whatever requests that are in the midst of being served need to be completed before it shuts down, and then new pods will be spun up in its place, and those would be responsible for handling the traffic. However, in this instance, it's really hard to maintain zero downtime because we never know if one of the pods is running a long-running process. Like if a pod is having a WebSocket connection, if it's a really long-lived connection to the client, we cannot just wait for the pod to finish its connection because it's going to be a while before it does that. So I think the other approach that I really like is blue-green deployments or canary deployments through Kubernetes, where we could spin up the entire pods cluster with the new version as a new deployment, and not have it serve any traffic. The old deployment would still be responsible for handling 100% of the traffic. And slowly, we can start cutting over a small percentage of traffic over to the new deployment. So let's say 10% of the traffic gets cut over to the new deployment. Then if everything goes well, then we can slowly start scaling up. This is, of course, under the assumption that the deployment we're making, the new deployment, doesn't immediately need to go out to all customers because this is a relatively slow-moving process. So you would go to 10, then you would maybe go 25, 50, 75, and 100. And so we would have to have checks and balances in place at every one of those checkpoints to ensure that things are live and the new deployment is serving traffic without any major errors.

To ensure data consistency during stateful application upgrades in Kubernetes, one method is to use persistent storage for data. This can be achieved by storing data onto the Kubernetes pods itself, either through a ConfigMap or a VolumeMount. However, a more reliable approach is to use a Persistent Volume (PV) or a StatefulSet, which allows for data to be stored independently of the pod's lifecycle. This way, during an upgrade, the new pod will have access to the same data as the previous one, ensuring data consistency across upgrades. The application itself can then reference the stored data as needed, without needing to know the details of the storage mechanism.

What Python data structures and algorithms would be most efficient for processing live video stream metadata? Processing live video stream metadata. so it's a live feed. So it's continuously coming in. and I think the first one that comes to mind would be a queue. right? So you would have something that comes in. it's 1st in, 1st out. comes if it comes in first, it's the first one to get processed, and we want to sort of maintain that order. So we would want to use a queue in that sense to make sure that we are streaming, and processing the stream metadata in the right order. So queues would be a great example for the Python data structure, for processing live video stream metadata. In terms of algorithms, anything that can leverage the queue data structure would be a good algorithm for processing video stream metadata. again, I don't have professional experience with live video streaming. so I don't know, specifically which algorithm would be beneficial.

Looking at this Kubernetes deployment, yaml, this configuration is causing intermittent issues with the application's availability. Identify why this might be problematic and what could be done to mitigate the issue. App version, apps v one. Deployment, video backend. Replicas 2. Video backend. Video backend. So, immediately, I'm looking at the limits and the requests. I'm seeing that the limit is the exact same for both memory and CPU as the request. So, whenever there's a spike in traffic coming to this particular video backend application, it's unable to scale up because we've limited the memory and CPU to its initial request of 256 megabytes and 0.5 of the CPU cores. So, it doesn't have any room to scale up and request additional resources. Unless we have something in place like a horizontal pod autoscaler, it's not going to be able to scale up its resources or its replica count because we're just saying there are 2 replicas for this particular application. I would recommend in this instance to have a higher memory limit, depending on how much memory and CPU we're planning to consume, I would say at least we should go up to, like, 512, double the memory, 512 megabytes, and double the CPU as well, in terms of 1 instead of 0.5. So, at least there is some buffer between the memory and the CPU request and the memory and CPU limits. And that should help the issues we have been having with intermittent issues with this application's availability.

We can use Golang's built-in testing package to create a load test for a video streaming API. So, our test would have to act as an end-to-end test with spawning of multiple different client connections to a video streaming API. And we can do that through concurrency control in Go. So we would spin up x number of new clients as part of the testing package and then try to connect all of those clients at the exact same time, do the video streaming API, and see if it is able to gracefully handle all of the connections, look at the latency and all of those numbers as part of those clients that we have respond, and see if it's able to gracefully error handle and all of those types of things as part of the testing package. We can assert that it doesn't error out. We can assert that the latency is within acceptable metrics. And so, concurrently spawn multiple different client connections to the video streaming API as part of the load test and see if our API gracefully handles the situation or if it dies completely.

What Terraform modules would be instrumental in scaling a media streaming platform on AWS? I don't have hands-on experience with Terraform, so I'm not sure what Terraform modules would be instrumental. So I'm going to skip this answer.

I know a little bit about Terraform. We do use it in my current company. However, Terraform is more used by our site reliability engineers, as opposed to our actual developers. So that's why I don't have hands-on experience using Terraform. However, the benefit of using Terraform is basically to manage our infrastructure. I understand Terraform as infrastructure as code. So you can write infrastructure similarly to how you can write applications as code in Kubernetes, where you can define the services, the config maps, and the secrets, and then have that be deployed to Kubernetes in a declarative fashion. You do the same thing with Terraform and your infrastructure, where you can stand up servers or request additional resources through Terraform. The benefit of doing that is it can go through a code review pipeline, so people can approve or deny changes as part of Terraform, so you're not willy-nilly killing additional servers or creating new servers and stuff like that. It can go through an approval workflow. It can be version-controlled, so you can put it into Git, which also means that you can have CICD as part of Terraform, and then have that go and scale up servers or scale down servers or the infrastructure as required. So those are the advantages that I see. Terraform would be at a layer lower than Kubernetes, where we're talking about the actual infrastructure for Terraform, and Kubernetes would be managing the services that get deployed on the infrastructure set up by Terraform.