Apache Iceberg: The Open Table Standard Transforming Data Lakes

Partition Evolution — Unique to Iceberg

This is the feature that sets Iceberg apart. With Delta Lake and Hudi you have to rewrite the entire table when you change partitioning. With Iceberg you can change the partitioning strategy without moving a single byte of data.

-- Start with partitioning by day
CREATE TABLE orders (
  order_id BIGINT,
  customer_id BIGINT,
  amount DECIMAL(10,2),
  order_date TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(order_date));

-- Later grow to hourly (without rewriting the table!)
ALTER TABLE orders
ADD PARTITION FIELD hours(order_date);

-- Iceberg writes new data in the new partition structure,
-- old data stays unchanged. Query engines understand both.

Time Travel & Rollback

Query historical data or recover accidentally deleted records:

-- Query yesterday's table (Spark SQL)
SELECT * FROM orders
TIMESTAMP AS OF '2026-03-20 09:00:00';

-- Query a specific snapshot
SELECT * FROM orders VERSION AS OF 4521;

-- Roll back to a previous snapshot
CALL catalog.system.rollback_to_snapshot('db.orders', 4521);

-- View snapshot history
SELECT * FROM orders.snapshots;

Schema Evolution Without Downtime

Add, rename or drop columns — safely and without rewriting data:

-- Add a column (takes effect immediately, no rewrite)
ALTER TABLE orders ADD COLUMN discount DECIMAL(5,2);

-- Rename a column
ALTER TABLE orders RENAME COLUMN amount TO order_amount;

-- Reorder a column
ALTER TABLE orders ALTER COLUMN discount AFTER order_amount;

-- Drop a column (data remains, but is no longer read)
ALTER TABLE orders DROP COLUMN legacy_flag;

Row-level Deletes & Updates (Copy-on-Write vs Merge-on-Read)

Iceberg supports two strategies for updates and deletes:

-- Copy-on-Write (CoW): rewrites files on every update
-- → Faster reads, slower writes
-- Good for: batch updates, reporting tables

-- Merge-on-Read (MoR): stores delete/update markers
-- → Faster writes, reads require a merge
-- Good for: frequent CDC / streaming updates

-- MERGE INTO (upsert) — works in Spark, Trino, Flink
MERGE INTO orders t
USING updates s ON t.order_id = s.order_id
WHEN MATCHED AND s.status = 'cancelled' THEN DELETE
WHEN MATCHED THEN UPDATE SET t.amount = s.amount, t.status = s.status
WHEN NOT MATCHED THEN INSERT *;

Test locally with Spark + Iceberg

# pip install pyspark
# Download the Iceberg Spark runtime JAR

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("IcebergDemo") \
    .config("spark.jars.packages",
            "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.0") \
    .config("spark.sql.extensions",
            "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.local.type", "hadoop") \
    .config("spark.sql.catalog.local.warehouse", "/tmp/iceberg-warehouse") \
    .getOrCreate()

# Create a table
spark.sql("""
  CREATE TABLE local.db.orders (
    order_id BIGINT,
    customer_id BIGINT,
    amount DECIMAL(10,2),
    order_date DATE
  )
  USING iceberg
  PARTITIONED BY (months(order_date))
""")

# Insert data
spark.sql("""
  INSERT INTO local.db.orders VALUES
    (1, 101, 99.95, DATE '2026-03-01'),
    (2, 102, 249.00, DATE '2026-03-15'),
    (3, 101, 59.99, DATE '2026-03-20')
""")

# Query with time travel
spark.sql("SELECT * FROM local.db.orders.snapshots").show()
spark.sql("SELECT * FROM local.db.orders").show()

Iceberg on AWS: Glue + Athena

-- In AWS Glue Data Catalog / Athena:

-- Create a table as Iceberg
CREATE TABLE orders (
  order_id BIGINT,
  customer_id BIGINT,
  amount DOUBLE,
  order_date DATE
)
LOCATION 's3://my-datalake/orders/'
TBLPROPERTIES (
  'table_type' = 'ICEBERG',
  'format' = 'parquet',
  'write_compression' = 'snappy'
);

-- Insert data (no S3 copy needed!)
INSERT INTO orders VALUES (1, 101, 99.95, DATE '2026-03-21');

-- UPDATE (Athena engine v3 + Iceberg)
UPDATE orders
SET amount = 89.95
WHERE order_id = 1;

-- DELETE
DELETE FROM orders WHERE order_id = 1;

-- Time travel
SELECT * FROM orders
FOR SYSTEM_TIME AS OF TIMESTAMP '2026-03-20 12:00:00';

Migrating from Parquet to Iceberg (in-place)

-- Migrate an existing Parquet table to Iceberg
-- WITHOUT copying data (only create metadata)

-- In Spark:
from pyspark.sql import SparkSession

# Migrate an existing Parquet table
spark.sql("""
  CALL catalog.system.migrate(
    'db.existing_parquet_table'
  )
""")

-- Or via snapshot (safer: the original stays intact)
spark.sql("""
  CALL catalog.system.snapshot(
    source_table => 'parquet.db.orders',
    table => 'iceberg.db.orders'
  )
""")

-- Validate the migration
spark.sql("SELECT COUNT(*) FROM iceberg.db.orders").show()
spark.sql("SELECT * FROM iceberg.db.orders.snapshots").show()

Table Maintenance: Compaction & Cleanup

-- Compact small files together (compaction)
CALL catalog.system.rewrite_data_files(
  table => 'db.orders',
  strategy => 'sort',
  sort_order => 'order_date ASC'
);

-- Remove expired snapshots (data cleanup)
CALL catalog.system.expire_snapshots(
  table => 'db.orders',
  older_than => TIMESTAMP '2026-03-01 00:00:00',
  retain_last => 5
);

-- Remove unused metadata and orphan files
CALL catalog.system.remove_orphan_files(
  table => 'db.orders',
  older_than => TIMESTAMP '2026-03-01 00:00:00'
);

-- View table statistics
SELECT * FROM db.orders.files LIMIT 20;
SELECT * FROM db.orders.partitions;