Ambuj Kumar

Worked on a realtime streaming and batch lambda architecture pipeline for ingesting blockchain events and populating KPIs/dashboard in DeltaLake. Created batch/streaming analytics jobs for the lambda architecture using Airflow managed periodic PySpark jobs, writing to DeltaLake. Modeled data warehouse for KPI tracking on Snowflake (OLAP) and Databricks Delta. Created and managed DBT models for extensive data quality enforcement on DBT cloud. Modeled and created updating pipelines to a Neo4j knowledge graph for end user data relationship management product. Used GitActions, Docker, Kubernetes, Terraform for CI/CD operations. Ensured GDPR and CCPA compliant data platform.

Created streaming pipelines to ingest sensor data and process them in real time to populate dashboards and the warehouse. Created pipelines for sensor data published into Kinesis (and S3 for failsafe reprocessing) ingested by a Databricks job, written into Azure Delta tables and Clickhouse (GCP earlier). Worked on Looker and SQL Analytics dashboards for Clickhouse/GCP. Data quality testing and improvement via periodic comparison jobs. Built pipelines as part of a SOLID principled ML codebase including ad hoc time bound backruns, API CDC jobs for metadata entities and the MLlib production optimized code, in Python (including Pandas API). Designed and integrated entities of the product using Databricks Delta (parquet DeltaLake) and Clickhouse. Used Terraform and Github Actions for DevOps/Infra/CI CD. Ensured GDPR, CCPA, and HIPAA compliant data platform.

Worked on multiple API source ingestion, dump schema creation and entity modelling using Cosmos and Scala Azure Functions. Worked on global multi region sources and associated rule based implementation of Spark Azure Databricks notebooks driven ETL region specific pipelines. Integrated entities in the property domain using Azure Cosmos Graph and Azure Databricks Notebooks, followed by Scala web-service APIs deployed on Azure HDinsights for quick search. Worked on streaming data application element of the pipeline, detecting refreshes. Designed individual table based schema handling, ingestion and implementation of a data warehouse for KPI tracking and its respective components for a full fledged reporting data warehouse. Created Spark jobs for handling daily data from Mongo, MySQL, Postgres and folder dumps to update the data warehouses, using Airflow scheduling. Managed scaled ingestion from public competitor APIs for tracking relevant parameters in analytics warehouse on Redshift. Worked on complex custom reporting Spark logic driving insightful marketing strategy. Benchmarked the real-time elements of the solution with Kafka Streams.

Created Spark batch jobs based on derivation from incoming data-model via a productionised ML model with associated business logic. Implemented Flask APIs layer and simulator for the application. Tested end-to-end pipeline and DevOps of associated individual component log monitoring. Overall design and development of the lambda architecture: MQTT based, Kafka, Spark pipeline for data ingestion and alert detection. Developed cloud agnostic framework. Created Scala Flink complex event processing and detection pipeline from incoming data-model with business logic. Worked on APIs layer implementation in Akka and a simulator for data. Tested end-to-end pipeline and DevOps of associated individual component log monitoring on AWS. Created data format based overall design and development of a MQTT based, Kafka, Flink, RDBMS and Cassandra pipeline for data ingestion and event/milestone detection.

GE Healthcare Device Monitoring Product: Deployment and maintenance of the Azure cloud based cluster (DevOps), along with pipeline design and data handling constraint using a Data Virtualization tool. Implemented detection algorithms of different respiration and lung parameters, and accumulation algorithms for case-end aggregation requirements. Data modeling for Cassandra for real-time data storage and case-end data aggregation. Data modeling for data-warehousing and UI based consumption. Company Log Data Analytics: Involved in PIG scripting and the HIVE database, to staging layer for processing before loading into final Hadoop table. Worked on OOZIE workflows for executing Java, Pig and Hive actions based on decision nodes, scheduled Oozie Workflow and Coordinator Jobs.