Mr. Prema K Veerapaneni

Streaming-Driven Finance: The Rise of Real-Time Data Integration

Abstract:

Financial institutions are moving from overnight batches to always on, event driven architectures where insights arrive as fast as the market moves. This talk maps the shift to real time: from legacy ETL to streaming pipelines that combine CDC from core systems with market feeds, payments events, and clickstreams. We’ll compare design choices Kafka/Kinesis + Flink/Spark Structured Streaming, micro batch vs. true streaming, lakehouse sinks, and idempotent, exactly once delivery. Practical patterns cover schema evolution, late/out of order data, stateful enrichment, and feature generation for ML. Governance and resilience are first class: data quality in flight, lineage, secrets management, and controls for PCI/PII and regulatory audit. We’ll quantify business impact via case studies fraud detection under 200 ms, intraday liquidity views, and personalized offers and show how to manage costs with FinOps, storage compaction, and autoscaling. Attendees leave with a reference architecture, an implementation checklist, and metrics (SLA/SLO) to guide the transition from pilot to production so real time becomes not just faster, but safer, cheaper, and verifiably compliant.

Key topics include:

The Real-Time Mandate, Streaming Reference Architecture CDC Stream: From Tables to Events Late Data, Watermarks, and Windows In-Flight Data Quality & Governance FinOps for Streaming Pipelines Case Studies: Fraud, Liquidity, Offers

This session is ideal for data engineers, architects, and financial technology professionals looking to modernize their data infrastructure and drive smarter, faster decisions through real time data systems.

Profile:

Prema Kumar Veerapaneni is a senior data engineering leader with 18+ years of experience in enterprise data integration, cloud-native ETL, and real-time analytics. He designs and operates governed data platforms on Informatica IICS, AWS Glue, Databricks on AWS, and Snowflake, spanning batch and streaming workloads. Prema focuses on reliability, security, and measurable outcomes embedding lineage, data quality, and cost controls into production pipelines. He is the first author of “Intelligent Evolution of AWS Glue for Scalable Serverless ETL,” accepted at ICCA 2025, and serves as a reviewer for IEEE conferences. His recognition includes the 2024 International Innovative Engineer Award for contributions to unified data ecosystems and AI-assisted integration. Prema mentors teams on schema design, observability, and governance, and aligns data capabilities with business strategy to turn fast-moving data into trusted, actionable insight. He holds an MCA from the University of Madras.