Siddhartha Parimi

The Evolution and Impact of Open Table Formats in Modern Data Architecture

Abstract:

The enterprise data landscape has fundamentally shifted from traditional two-tier architectures where data lakes stored raw information while warehouses handled analytics to unified lakehouse platforms. This transformation centers on three dominant open table formats: Apache Iceberg, Delta Lake, and Apache Hudi, each offering distinct architectural approaches to managing structured data at scale on cloud object storage. 
Apache Iceberg employs a metadata-centric, three-tier hierarchical architecture with immutable metadata files, manifest lists, and manifest files containing detailed statistics including record counts, column-level minimum and maximum values, null counts, and partition information. Its optimistic concurrency control with snapshot isolation enables instant access to historical snapshots without transaction log scanning, while hidden partitioning allows partition specification evolution without data rewrites. The format's storage-agnostic design supports multiple cloud platforms with mature connectors across Spark, Flink, Trino, Presto, and Hive. 
Delta Lake utilizes an append-only transaction log architecture with sequentially numbered JSON files establishing total operation ordering. Periodic checkpoint consolidation prevents unbounded log growth while maintaining efficient query planning. The format demonstrates deep Apache Spark integration with Structured Streaming support for exactly-once semantics, and recent innovations including deletion vectors for efficient updates without full file rewrites, liquid clustering for automated data organization, and UniForm features enabling cross-engine compatibility. 
Apache Hudi differentiates through dual table types: Copy-on-Write tables rewrite entire files for optimal read performance, while Merge-on-Read tables append changes to log files achieving lower write latency with deferred compaction. The format's indexing capabilities using Bloom filters, HBase, and simple indexes enable efficient upsert operations, while the timeline service provides comprehensive audit trails supporting incremental processing and change data capture patterns. 
Understanding these architectural differences, operational complexity variations, and ecosystem maturity levels enables organizations to construct optimized lakehouse platforms aligned with specific workload requirements across bulk loading, streaming ingestion, concurrent writes, query performance, and modification-heavy scenarios. 

Profile:

Siddhartha Parimi is a seasoned Technical Product Manager with over 11 years of experience in data architectures, analytics, and large-scale data management. As a Product Manager at Dell Technologies, he drives enterprise-wide data strategies and modern Lakehouse architectures, delivering multi-million dollar programs with substantial cost savings and streamlined governance. 
Previously at Deloitte Consulting, Siddhartha led transformative data projects, including an innovative Accounting Hub that standardized finance data and governance within a year, significantly enhancing reporting accuracy and operational efficiency. At Swanktek Inc., he designed a groundbreaking business intelligence solution for Montgomery County's Department of Health and Human Services, consolidating 30-40 disparate programs into a unified platform that reduced fraud and improved data integrity. 
His career demonstrates consistent success in transitioning proof-of-concepts into enterprise-scale solutions. With a Master's degree in Engineering from Wright State University, Siddhartha combines deep technical expertise in cloud architecture and modern data platforms with strong business acumen. His ability to deliver scalable, cost-efficient solutions while balancing technical requirements with strategic objectives has established him as a leader in enterprise data modernization initiatives.