Manish Choudhary

Hi. This is Manish. Uh, I have been working in the IT industry as a Python developer for the past 6, uh, 5 years 6 months. So far, I have worked on data science, machine learning, and currently working on back end development. So in my recent project, I'm working on API development, maintenance, and doing some data transformation, creating some installations. So that's a brief

So as we all know that, uh, with this statement, we, uh, usually use to operate or open file operations. Like that, we can use database interactions as well. Uh, whenever we connect to a database, then with the statement automatically close the connections after the operation of the database or any commit or any execution of the query.

So Python ETL, uh, usually, uh, we can create, uh, APIs on top of, uh, like, Spark. Spark, uh, will be very helpful and useful to deal with the large datasets. Apart from Pandas library, Python can be used to transform the data in chunks, and, uh, Dash is also nowadays popular. So these libraries are, uh, can be used. Apart from that, we can use caching mechanisms of Python, And, uh, we can do multiprocessing or multithreading as well to deal with multiple or, uh, big datasets. So that's

So, like, uh, in a skill, uh, we can write objects directly or, like, we can do data selections and these deselections, and we can use those objects whenever we need it. So SQL can be very helpful, uh, to use objects directly from the databases.

So AWS Lambda will be very helpful, uh, as this is serverless, uh, service. So we don't need, uh, any servers for that. We just need to run our scripts on, uh, schedule basis, and it is very much helpful to run the pipelines

So to when comes, uh, operation on the databases or connecting with databases, so, usually, we have integrate error, which we can use in exception handling. Uh, sorry. Not integrate. Integrity errors or some formatting issues with the query or, uh, some data type errors. So that's we can handle

A simplified Python code log in to send a page of my constrained database lambda function is below. Appears to be an oversight that could lead to errors or unexpected behavior. There appears to be an oversight there appears to be an oversight that could lead to a lesser unexpected So here we can, uh, like, we are not using any exception handling. So in line number 4 after line number 4 in for loop, whenever we invoke the client, Lambda Lambda client, or both of the client, we would say, Uh, there we can add exception handling so that we are sure that we are able to connect through, uh, Lambda client to the Lambda.

This is a security snippet, which is meant to select all the calls from sales data where the revenue is higher than the previous month's revenue. So it looks like we are directly using, like, and, uh, revenue in where condition, and we're adding our result. Instead of doing this, we can use, uh, self join, and we can get the desired output.

So the best way to debug Python applications, uh, over the SQL transactions or transformations, we can add, uh, as much as exception handlings, and we can do the proper logging of those, uh, exceptions, like integrate error or anything else or, uh, if you're not able to connect to database. So they're the best way is to use exception handling.

Not sure about

Can you discuss an approach to manage your state effectively in DX application working with stream date based Python backend. Not sure again