Jagannath Das

Hello. My name is, and I am a data analyst with 2 plus years of experience working in group technology and system solutions, and building into machine learning models in collaboration with senior data scientists. And, as a data analyst, I also build dashboards in Tableau and work in machine learning, mainly. So, I have completed my B.Tech in 2019, August, and secured a 67.8 c g p a in my finals in my semesters.

Can we discuss the Python technique you have used for feature selection in your dataset and the impact it had on multiple things. The technique, some Python technique I have used for feature selection is the p-value. We use p-value to see how efficient this feature is. If the p-value of a feature is greater than 0.05, then we reject that feature for consideration in model building. And we do recursive feature elimination. By removing the least important feature, this way, we use BIF, checking BIF of the model. This impacts significantly on the model and another, some models do not need to do this p-value selection. Some models have to do this p-value selection, like some models like logistic regression, and linear regression. Those type of models require p-value selection. If the p-value is less than 0.05, then we'll omit the features. Random forest and XGBoost, those models don't require this p-value selection.

So the Python based machine learning model, to deploy the machine learning model, first, we start from the business objective of the business. We talk with the client. We understand the objective of the business. Then we decide how to approach the project with the senior data scientist, project manager, and team manager. So after that, we prepare data. We get the data from the data engineer or client server, and then we do feature engineering. Then we clean the data. After feature engineering, we do exploratory data analysis to determine which features impact the model. Then after doing EDA, we do feature engineering, such as encoding categorical and discrete variables, and scaling continuous variables. For categorical variables, we use one-hot encoding or get_dummies. For discrete variables, we also use one-hot encoding. For continuous variables, we use min-max scaling or standard scaling. After processing the data, we train the model. Then we evaluate the model and test it. After testing, we build the model on top of a web framework, such as Flask or FastAPI. We test the model in a local server. Before deployment, we save the model in a.joblib file or.pkl file. Then we containerize the model with Docker. After containerizing the model, we deploy it on an AWS EC2 instance. We use continuous integration and deployment with Jenkins to automate the deployment process.

To mitigate overfitting in a super risk revenue model, we can take an approach that passes to overcome the issue. One option is to ask for more data from the client or data engineer if the data is limited. After obtaining additional data, training the model may reduce some overfitting. We also have Lasso and Ridge for supervised learning problems, which are known as L1 and L2 regularizations. Lasso penalizes large coefficients and sets smaller coefficients to 0, minimizing their effect. Ridge, on the other hand, reduces the coefficient value or the weights of the features, but does not set them to 0. Additionally, we can use pruning in decision trees, where reducing the number of split nodes can prevent overfitting. In ensemble methods, such as random forest and boosting, we can also use integration to mitigate overfitting.

We use Python's multiprocessing library, telling libraries, like, using GIL, Global Interpreter Lock, using threading library is more suitable. The threading library is more suitable for input/output context for model. We typically prefer a multi-purpose thing, but we use threading. So for so and multi processing is suitable for CPU-bound tasks like model training. It creates a separate process, each with its own Python and memory space, efficiently bypassing the GIL and threading those. And that's more appropriate for input/output tasks. Due to the GIL, threading may not provide significant speedup for CPU-bound tasks.

Can we discuss your machine learning project where you had to optimize data movement to memory and computational budget constraints. Yes. So, one of my recent projects, which I was working on, was the fraud detection in service claims. The way to optimize code is to identify where we have huge amounts of data. Previously, we were working on huge amounts of data, but after that, we thought of another issue, that the data is also imbalanced, so we thought of making it balanced and taking a sample from the data. It's almost 6.3 million data points. So, we used stratified sampling from the data, which balances the data and maintains the ratio of the data in a classification problem. This balances both classes instead of a 4:1 ratio. By doing this, we optimized the model, and it doesn't require much time to load the data. Previously, we were working on all 6.3 million data points, then we took only 10% of the data by sampling using a stratified sampling technique from the scikit-learn library. In this static sampling, we also tried to do it using pandas and the groupby method. We tried to do the stratified sampling within a group and using a lambda function. However, we found that doing stratified sampling using pandas took much more time. So, then we shifted to the scikit-learn library for stratified sampling, which reduced the memory usage and improved the computational efficiency before building the model process, which we do in EDA or feature engineering, and maintaining the model's performance. Despite the optimization, we were able to implement robust model predictive performance, meeting the project requirements also.

In the following Python function, design table that machine learning security, locate and explain the bug that could cause incorrect result. So calculate a glitch increase, breach levels, I'll look at and explain them what that could cause incorrect result. The original, code snippet lacks the intention, which is necessary in Python. So there is an intention error. So the logic itself is correct. I found I found the logic. It's correct. So the so the current accuracy equals to calculate, accuracy equals to okay. Calculate accuracy, like, prediction comma level and print accuracy. Used to here used to double instead of and w equals to if a trade equals to equals to correct this input and friction. Accuracy equals to So I don't find any bug in the code. I think the code is, correcting it since.

Given this Python produce for the loading machine model, discuss what could go wrong and how it is. So in the load, in the last line of loaded mod loaded underscore model equals to load underscore model. it has some error like, inverted commiser, beyond the black backslash in beginning itself. So and the backslash is also, after the model dot pkl, which is not correct, I think, which is not correct. So, the it is supposed to write that load underscore model, bracket stat, inverted comma stat, and then we keep the path there over, of the model that we get. and then we, then we close the inverted commas. Without using that last backslash. We have to also omit that last backslash on that model that we can.

Distributed training of a machine learning model involves several steps and considerations, such as multi-worker mirrored strategy. We choose a framework like TensorFlow or PyTorch, then we set up the environment, creating a virtual environment in that system, in that coding platform. We initialize the model. Yes. Training with TensorFlow is straightforward by providing a simple API for initializing and operating. Then, we do potential challenges, like data handling, efficient distributing data across multiple nodes of GPU can be complex. We can also use communication like synchronizing ingredients and model words across devices can introduce latency. These are communication overhead and hardware availability, like this is also one type of challenge that hardware availability. We have to work within the hardware that is available to us. So, to answer that, we use hardware optimization for the hardware used, we can do containerization, like Docker. And we do check for fault tolerance. So, we implement the checkpoint of a safe model. We do scalability by monitoring the performance and efficiency of the training model, training process, optimize the workload distribution. We use, we also see that complexity in debugging. So, we use logging and monitoring tools, like Craftsmen. We do for monitoring we use Craftsmen. So, like this, we do distributed training of many models.

So, we can use a Python generator for memory efficiency, then improved performance, then simplified and data pipelines. I can do lazy eval generator is used to basically for lazy evaluation, like generators. So handling infinite data streams, and enhance readability and maintainability, and parallelism and concurrency, using we can also do pipeline integration with this Python generators.

In Python, we use a library like NumPy to streamline complex calculations in data pre-processing. By using NumPy itself, which is a scientific library for calculations, we can efficiently handle data centers and data into the data centers and to the data centers. NumPy is very much efficient while reshaping data and performing edit calculations. Python is also very efficient while handling missing data. Additionally, Python gives some aggregation functions like group by and to use for aggregation, and sorting. We have some mathematical operations in NumPy that are efficient in various areas. We use pandas for data loading, data exploration, and then after data cleaning, data transformation, grouping. As I said, grouping and aggregations, and merging and joining columns for often data.