Mr. Dinesh Eswararaj

AI-Powered Data Quality and Governance: A Hybrid Framework for Scalable and Sustainable Cloud Architectures

Abstract:

Ensuring high-quality, trustworthy data is a persistent challenge in modern cloud architectures. This keynote explores an AI-powered hybrid data quality and governance framework implemented within Microsoft Fabric’s unified Lakehouse ecosystem. The framework integrates PySpark-based validation logic with a large language model–driven semantic reasoning to detect anomalies, resolve inconsistencies, and enhance compliance automatically. By embedding metadata-driven automation, self-healing pipelines, and adaptive quality scoring, it achieves a scalable and sustainable approach to managing enterprise data. The session will present practical insights from real-world implementations, performance benchmarks, and cost-efficiency outcomes, demonstrating how AI can significantly reduce manual intervention while improving data reliability and sustainability across the Bronze–Silver–Gold layers of Fabric architecture.

Profile:

Dinesh Eswararaj is a Lead Data Engineer / Data Architect specializing in Azure, Microsoft Fabric, and Databricks. He has led large-scale data-platform modernizations in automotive and enterprise settings, designing medallion lake houses, metadata-driven ingestion frameworks, and cost-optimized pipelines across ADLS, Delta Lake, and Unity Catalog. His work includes a configurable Data Service Automation Framework that accelerated vendor-feed onboarding and delivered significant cost savings. Dinesh is an IEEE Senior Member and frequent reviewer/industry judge. He has published multiple articles on cloud data engineering, data quality, and AI-assisted validation, and regularly mentors teams on CI/CD, governance, and performance tuning for lakehouse architectures.