Sachin Chaudhary

Contributed as a key developer in the design and development of a custom Data Lineage Framework, enabling comprehensive lineage tracking for data ingestion and ETL pipelines. Integrated the open-source Spline framework to capture metadata from Apache Spark jobs running on AWS EMR, AWS EMR Serverless, and AWS Glue. Developed custom parsing logic to extract and transform captured lineage data, enabling Neo4j for graphical visualization of job dependencies using APIs. Engineered a configuration-driven parsing framework from scratch, enabling the capture and graphical representation of lineage for non-Spline-supported jobs. Collaborated with cross-functional teams to integrate lineage capabilities into broader data pipeline ecosystems and align with governance and compliance frameworks.

Designed and developed a scalable PySpark-based Data Ingestion Framework leveraging Apache Spark, AWS S3, AWS EMR and Hadoop, capable of ingesting terabyte-scale datasets efficiently. Built and integrated pre-ingestion data validation checks using PySpark DataFrames to ensure high data quality and consistency. Improved ingestion performance by optimizing Spark job execution, partitioning strategy, and I/O operations, resulting in a 30% reduction in processing time. Integrated the framework with orchestration tools like Apache Airflow and metadata/catalog management via AWS Glue, enabling seamless pipeline scheduling and governance. Ensured adherence to data governance, scalability, and security best practices in a distributed data processing environment.