Mr. Dinesh eswararaj
Intelligent and Sustainable Data Ecosystems through AI-Oriented Data Engineering and Cloud Architecture
Abstract:
Modern organizations are under intense pressure to ingest, process, and govern data at scale while simultaneously enabling analytics, AI, and regulatory compliance. Traditional data engineering pipelines built with manual ETL scripts, rigid workflows, and ad-hoc data quality checks struggle to keep pace with this demand and often lead to fragile, opaque, and expensive data platforms.
This talk explores how AI-enabled data engineering and cloud architectures can transform these pipelines into intelligent and sustainable data ecosystems. We discuss practical patterns for AI-assisted ingestion (schema inference, automated mapping, and metadata generation), AI-driven data quality (pattern learning, anomaly detection, semantic validation), and their realization in lakehouse-style architectures on modern cloud platforms. Drawing on real-world experience from large-scale migration and modernization projects, the session will highlight design principles, reference architectures, and governance models that balance automation with human oversight.