Mr. Krishna Chaitanya

Integrating Heterogeneous ETL Pipelines: Towards Unified Data Processing Across Cloud and Legacy Systems

Abstract :

As organizations increasingly maintain a complex mix of cloud-native and legacy systems, seamless integration of ETL processes across heterogeneous platforms has emerged as a critical technical bottleneck in modern data architecture. This presentation addresses the fundamental challenge of heterogeneous ETL integration by proposing an innovative abstraction layer that effectively decouples data transformation logic from execution environments, enabling organizations to achieve true platform independence in their data processing workflows. The research introduces a comprehensive metadata-driven ETL integration architecture specifically designed to synchronize both batch and streaming dataflows while maintaining strict data quality standards and regulatory compliance requirements. The proposed solution leverages widely-adopted open-source tools including Apache Airflow for workflow orchestration, Talend for data integration, and cloud-native APIs for seamless connectivity across diverse environments. Through rigorous experimental validation, the framework demonstrates significant improvements in data consistency, job orchestration efficiency, and comprehensive pipeline monitoring capabilities.

Profile:

Krishna C Batchu is a Senior Software Engineer with over 19 years of experience driving technological transformation across diverse industries including Financial Services, Airlines, Healthcare, Banking, and Energy. Currently at Fidelity Investments, he specializes in Azure DevOps, Artificial Intelligence (Copilot, Generative AI), and Test Code Automation, with proven expertise in optimizing Software Development Life Cycles (SDLC/STLC).

Throughout his career, Krishna has led critical projects at major organizations including Southwest Airlines, Navy Federal Credit Union, UT Southwestern, US Bank, and Chase Bank. His technical expertise spans programming languages (SQL, Java, Python), ETL tools (Informatica, SSIS, Talend), cloud services (Azure Data Factory, Databricks, Snowflake), and database management systems.

Krishna holds a Bachelor's degree in Mechanical Engineering and a Master's degree in Energy Systems from J N T University. He is Microsoft Certified in Business Intelligence Solutions and has received multiple recognition awards for his contributions to data management and software engineering excellence.

His core strengths include aligning technology with business strategy, fostering innovation, and delivering scalable, high-performance solutions that drive efficiency and competitive advantage across complex enterprise environments.