Soumith Beldha

Implemented an end-to-end pipeline in the Databricks Lakehouse to help businesses derive key KPIs. Developed a scalable, reliable, and easily maintainable data ingestion pipeline that loads data from Azure ADLS and Azure MI SQL instances, accommodating both historical and incremental loads. Designed a data model with various fact, dimension, and reference tables and built the pipeline using Databricks medallion architecture. Utilized Databricks Workflows for scheduling and managing the pipeline. Created a pipeline that ingests data from MongoDB and MySQL, transforming the data to generate final analytical views, including a streaming use-case. Developed several dashboards based on these views using the Databricks Dashboards API. Scheduled ingestion and transformation jobs using Databricks Workflows and integrated them with an API to trigger the workflows.

Developed robust end-to-end ETL pipelines on AWS capable of backfilling, reloading, and are scalable, maintainable, and dynamic enough to be reused by multiple source systems. Developed a centralized Data lake in S3 that handles data from different sources in different formats, cleans and converts data into a columnar format for downstream processing. After the Data ingestion process, we have a metadata-driven governance & Data Quality layer where transient EMRs with Spark help process the data lake, joins various normalized tables to form denormalized domains. From this GDQ layer, data is moved to an individual data warehouse in Redshift capable of processing data in parallel from multiple sources using spectrum and stored procedures and warehouse data. The complete project flow is orchestrated with end-to-end pipelines using Apache airflow.

Migrated Finance Datamart from Azure synapse to AWS Redshift. Created stored procedures that are dynamic enough to perform type-2 load processing for multiple domains and source systems concurrently. Built controls in place for reconciliation between each hop to ensure data integrity. Sorted serializable isolation violations when tens of pipeline instances were running concurrently using transaction controls and table locks without causing deadlock issues. Reduced processing time of stored procedures from one hour to 10 minutes by optimizing partition sizing and using appropriate distribution and sort keys.

Developing and Modifying the Sybase objects like Stored Procedures, Tables, Views, Indexes, schema, etc. in Development Environment. Preparing the test cases and performing Unit testing on the code developed and the regression testing of the process flow in Development Environment. Also performed the Migration of Unix from Aurora environment to Aquilon. Maintaining the Unix Environments of all the Non-Prod Environments like NFS paths, AFS paths, Unix User accessibility, Host performance, Data Purging, etc. Supporting the Production Maintenance team in case of deployments or Production issues. Interacting with the Clients for knowing and understanding the new developments and business requirements.