Suman khatua

Building Medallion Architecture in MS-Fabric. Built metadata driven medallion architecture with 3 layers (Bronze, Silver, Gold). Perform CDC in Gold Layer and doing complex transformation in Gold and Silver Layer. Understanding the business with Client to help in building the reports, data mismatch in their old system. Building automated test cases for the pipelines built. Monitoring the pipeline with very less manual effort and seamlessly fixing if something fails. Migration of old data from multiple data sources such as: NAV, Salesforce, SFTP, Blueridge etc.

Building End-to-end Pipelines. Refined and enriched insights of trends through the development of full pipelines using Global Transactional Data. Additionally make dashboards upon the enriched data using Python, SQL, Shell-script. Working with different source connections like FTP, Shared-Drive, AWS etc to get the data and import into required table for further transformations. Worked on data building tool (DBT) to create macros for different use case of data test building, data transformations. Constructed full pipelines in a 4 layer structure (staging, intermediate1, intermediate2 and core), running, testing, debugging the same. Implemented multiple test cases to large set of data such as not-null, unique, foreign key relationship, range, accepted values etc. Created more than 30 python functions to validate different types of data inconsistency. Validated most of the source data with 99.9% validations meeting use case requirements to build machine learning models. Databricks Migration: Migrating the on-premises pipelines into databricks using different layers (Bronze, Silver and Gold). Orchestrating the pipelines to get optimized storage and compute resources used. Using Different utilities of databricks like versioning, external tables, internal tables, views etc. to enhance the process of pipelining data.

Building AWS Lambda and Step Functions. Development and enhancement of the infrastructure required for extraction, optimal transformation, and loading of data from a wide variety of data source applications using AWS data services, Spark SQL, Apache Airflow. Build scripts in Scala for generating data flow and templates for the data extraction interface and load the metadata details in a MongoDB. Building multiple AWS Lambda functions to convert the generated data in MongoDB to responsible for triggering AWS Step function to execute extraction job and ingesting data into AWS Data lake.

Building ETL Pipelines. Built ETL pipeline with Azure Data factory to load data from different on-prem servers to raw layer of azure Data Lake. Then cleansing data with Azure Databricks to move the data to cleanse layer for ADRM Data modelling. Applied transformations like scd1 and scd2 on the cleansed data and breaking source tables into different entities in 3NF form as per ADRM model to make the data available in stage layer. Developed daily pipelines to migrate millions of data insights from the stage layer ADRM data to Azure Cosmos DB using spark-cosmos connector. Proper incremental logic, transformation based on the modelling, optimization and partitioning done in spark to move the data to Cosmos DB efficiently. This helped our client to reduce cost by 40%. Implemented near real time batch pipeline in Azure Data Factory and Spark which will load one transactional table incrementally to CosmosDB after doing proper cleansing and transformations.