Watch this on You Tube https://www.youtube.com/watch?v=Yo0x4u6jc4M

🔗 If you’d like to learn Azure Databricks step by step, check out the full playlist here:
👉 https://www.youtube.com/playlist?list=PLNj2XeCNjFeosTuxZLjfYvnW4H1hsPH07

🌟 Introduction

In today’s data-driven world, organizations need a scalable, open, and flexible way to manage data across storage and compute platforms. Azure Databricks and Apache Iceberg together offer exactly that!

In this blog, we’ll explore how to connect Azure Databricks to Azure PostgreSQL and create an Apache Iceberg table using a simple, step-by-step approach. This approach helps you modernize your data lake and unlock new possibilities for analytics and machine learning. 🚀

💡 What is Apache Iceberg?

Apache Iceberg is an open table format designed for large-scale, analytic datasets stored in data lakes. It brings data warehouse-like reliability to the data lake by supporting:

• ✅ ACID transactions
• ✅ Schema evolution
• ✅ Partition evolution
• ✅ Time travel queries
• ✅ Hidden partitioning

With Iceberg, you can build a true lakehouse architecture that combines the performance of a warehouse with the flexibility of a data lake.

🧩 Why Connect Azure Databricks with Azure PostgreSQL?

Azure PostgreSQL often stores transactional or operational data. But for large-scale analytics, it’s better to replicate or move that data to Iceberg tables in Azure Databricks. This gives you:

• ⚡ Faster query performance
• 🧠 Seamless integration with Spark and ML workloads
• 🧱 Data versioning and audit support
• ☁️ Scalable, cost-efficient storage

---

⚙️ Prerequisites

Before we begin, ensure you have:

1. ✅ Access to an Azure Databricks Workspace
2. ✅ A running Azure PostgreSQL Flexible Server
3. ✅ Correct JDBC connection details (hostname, port, username, password)
4. ✅ A Databricks cluster with Iceberg support enabled

---

🪜 Step-by-Step: Creating an Iceberg Table from Azure PostgreSQL

Let’s go hands-on and build it! 👇

---

🔹 Step 1: Define Connection Details

In your Databricks notebook, start by specifying the PostgreSQL connection details.

jdbcHostname = "jbpos-sql-vnet.postgres.database.azure.com"
jdbcPort = 5432
jdbcDatabase = "postgres"

jdbcUrl = f"jdbc:postgresql://{jdbcHostname}:{jdbcPort}/{jdbcDatabase}"

connectionProperties = {
  "user": "jvivek2k1",
  "password": "xxxxxxxxx", 
  "driver": "org.postgresql.Driver"
}

Here:

• jdbcHostname → your PostgreSQL server name
• jdbcDatabase → the database you want to connect to
• user and password → your login credentials
• driver → PostgreSQL JDBC driver class

🔹 Step 2: Read Data from Azure PostgreSQL

Now, let’s pull data from your public.customer table in PostgreSQL into a Spark DataFrame.

df = spark.read.jdbc(
    url=jdbcUrl,
    table='public.customer',
    properties=connectionProperties
)

✅ This reads all rows and columns from your PostgreSQL table into Spark.
You can verify the data with:

display(df)

🔹 Step 3: Write Data to an Iceberg Table

Once the data is in Databricks, we can save it as an Iceberg Table in the Unity Catalog or Hive Metastore.

df.write.format("iceberg") \
  .mode("overwrite") \
  .saveAsTable("finance.default.postgres_customer_iceberg")

🔹 Step 4: Validate the Iceberg Table

After writing, you can run SQL queries in Databricks SQL or the notebook itself to validate the table:

SELECT * FROM finance.default.postgres_customer_iceberg;

🌍 Benefits of Using Iceberg Tables in Azure Databricks

1️⃣ High Performance Queries — Iceberg handles large datasets efficiently with advanced partition pruning and metadata optimization.

2️⃣ Schema Evolution — Add or modify columns without rewriting entire datasets.

3️⃣ Data Time Travel — Query data as it existed at any previous point in time.

4️⃣ Open Source & Interoperable — Works with multiple engines (Spark, Trino, Flink, Snowflake, etc.).

5️⃣ Cost-Effective Storage — Store data in open formats on low-cost cloud storage.

---

🏗️ Real-World Use Cases

• Building a Data Lakehouse from operational systems
• Creating auditable, version-controlled datasets
• Simplifying ETL pipelines by standardizing on Iceberg tables
• Enabling ML workloads with consistent and reliable data layers

---

🧠 Pro Tips

💬 Use Azure Key Vault integration to securely store your PostgreSQL credentials instead of embedding them in code.
⚙️ Use Incremental Loads instead of full overwrite for production pipelines.
📊 Consider using partition columns for large tables to improve query performance.

---

🎯 Summary

In this blog, we:
✅ Connected Azure Databricks to Azure PostgreSQL
✅ Loaded data from a PostgreSQL table into Databricks
✅ Created an Apache Iceberg table for modern data analytics
✅ Validated the data through SQL queries

By combining Azure Databricks + Apache Iceberg + Azure PostgreSQL, you’re enabling a modern, open, and scalable data lakehouse architecture that’s built for performance and flexibility. 💪

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.