Mr. Deepak Chanda

Intelligent Automation in Data Engineering: Harnessing AI for scalable, Real-Time, and Ethical pipelines

Abstract:

The rise of artificial intelligence (AI) is redefining how data engineering operates, enabling the creation of smarter, faster, and more adaptive data systems. This presentation explores the transformative role of AI in modern data engineering — from automating ingestion and real-time processing to enhancing data quality, orchestration, observability, and governance. We examine how AI-driven workflows not only boost operational efficiency and cost optimization but also introduce intelligent decision-making into pipeline architecture. The session will address the ethical dimensions of AI in data engineering, including bias mitigation and compliance, while also forecasting the future of autonomous pipelines and the evolving skillsets required for data engineers. Attendees will gain strategic insights and practical considerations for building scalable, AI-enabled data infrastructures ready for tomorrow's challenges.