Enterprise Databricks Architecture

Introduction

Watch this as a video on our you tube channel JBSWiki.

Many enterprises are rapidly adopting Azure Databricks for building machine learning models and serving real-time predictions. However, when strict network security measures are in place—like disabling public network access on Databricks workspaces—it can become incredibly challenging to integrate those models into tools like Power BI.

In this blog, we’ll explore how to securely call a Databricks Model Serving endpoint from Power BI under the scenario where:

The Databricks workspace has Allow Public Network Access = Disabled
Any direct call from Power Automate to Databricks fails with a 403 Unauthorized network access error

We’ll overcome this limitation using Logic Apps running inside a Virtual Network (VNet) and acting as a secure bridge between Power BI and Databricks.

Let’s dive in! 🔍

The Challenge: Network Restrictions and 403 Errors

By default, services like Power Automate send traffic over the public internet. If your Databricks workspace is configured with Allow Public Network Access disabled, any direct HTTP request to its REST APIs from Power Automate will fail.

The result is a 403 Unauthorized network access to workspace error.

This happens because Databricks:

Blocks all public network traffic
Only allows communication from services or VNets that are directly peered or integrated

In highly secure enterprise environments, keeping Databricks private is essential. But it poses a problem:

How can Power BI users trigger predictions from Databricks ML models if public access is disabled?

The Solution: Introducing Logic Apps as a Secure Proxy

Instead of connecting Power Automate directly to Databricks, we introduce Logic Apps running inside an Azure VNet.

Logic Apps can:

✅ Connect to Databricks Model Serving endpoints privately through peered VNets or private endpoints
✅ Expose an HTTP endpoint that Power Automate can call publicly
✅ Act as a secure proxy, handling all authentication and network routing

This architecture ensures:

Network security compliance
Seamless integration between Power BI and Databricks
Avoidance of 403 errors

Let’s walk through the full solution step by step. 🚀

Solution Architecture

Here’s how the integration flows:

User clicks a button in Power BI ➡ triggers Power Automate.
Power Automate ➡ sends an HTTP POST request to Logic Apps.
Logic Apps ➡ securely calls the Databricks Model Serving endpoint within the VNet.
Databricks Model Serving ➡ returns prediction results to Logic Apps.
Logic Apps ➡ sends the response back to Power Automate.
Power Automate ➡ updates Power BI visuals or datasets with prediction results.

This ensures Databricks never exposes its endpoints publicly, yet Power BI can still retrieve real-time predictions.

Step 1 — Create Databricks Model Serving Endpoint

First, make sure you’ve deployed your machine learning model to a Databricks Model Serving endpoint.

For this blog, let’s assume you’ve published an endpoint like:

https://adb-1311343844234579.11.azuredatabricks.net/serving-endpoints/HDFC_High_price_prediction/invocations

This endpoint:

Requires authentication via a Databricks PAT (Personal Access Token) or Azure AD token.
Accepts JSON requests.
Returns prediction results in JSON format.

Remember, because public network access is disabled, only resources inside your VNet—or peered VNets—can reach this endpoint.

Step 2 — Create Logic Apps in VNet

Next, deploy a Logic App Standard into a VNet.

Benefits:

Can communicate privately with Databricks.
Supports secure inbound and outbound traffic.
Scales to enterprise workloads.

Create an HTTP Trigger

Configure Logic Apps to start on an HTTP request.

Request Body JSON Schema for our scenario looks like this:

{
  "type": "object",
  "properties": {
    "inputs": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "Date": {
            "type": "string"
          },
          "OPEN": {
            "type": "integer"
          },
          "HIGH": {
            "type": "integer"
          },
          "LOW": {
            "type": "integer"
          },
          "CLOSE": {
            "type": "integer"
          }
        },
        "required": [
          "Date",
          "OPEN",
          "HIGH",
          "LOW",
          "CLOSE"
        ]
      }
    }
  }
}

his defines the expected JSON payload your Logic App will receive from Power Automate.

Step 3 — Logic Apps: Call Databricks Model Serving

Inside Logic Apps, add an HTTP action to call your Databricks endpoint:

Method: POST
URI:

https://adb-1311343844234579.11.azuredatabricks.net/serving-endpoints/HDFC_High_price_prediction/invocations

Headers:

Authorization: Bearer dapi****************
Content-Type: application/json

Body:

{
  "inputs": [
    {
      "Date": "2024-07-03",
      "OPEN": 2300,
      "HIGH": 2400,
      "LOW": 2298,
      "CLOSE": 2350
    }
  ]
}

Logic Apps will securely send this payload over private networking to Databricks and wait for the response.

Step 4 — Deploy Power Automate Flow

Now, let’s connect Power Automate to Logic Apps.

Your Power Automate flow will:

Trigger from Power BI (e.g. a button click).
Call the Logic Apps HTTP endpoint.
Receive the ML prediction results.
Optionally, update Power BI visuals or datasets.

Power Automate HTTP Request

Configure your HTTP action:

Method: POST
URI: The URL from your Logic App’s HTTP trigger (Step 1).
Headers:

Content-Type: application/json

Body:

{
  "inputs": [
    {
      "Date": "2024-07-03",
      "OPEN": 2300,
      "HIGH": 2400,
      "LOW": 2298,
      "CLOSE": 2350
    }
  ]
}

Why Not Call Databricks Directly From Power Automate?

A natural question is: why can’t we skip Logic Apps and call Databricks directly from Power Automate?

Here’s why:

Power Automate sends HTTP requests from public endpoints.
Databricks rejects all public traffic if public access is disabled.
There’s no way for Power Automate to reach Databricks privately.

Logic Apps in a VNet acts as a secure intermediary:

Power Automate → Logic Apps → Databricks
Databricks → Logic Apps → Power Automate

This architecture bridges private and public networks securely.

Benefits of This Architecture

Implementing this solution provides:

✅ Enterprise Security

Complies with strict network isolation policies.
Prevents exposing Databricks to the internet.

✅ Seamless User Experience

Power BI users get real-time predictions without knowing about the backend complexity.

✅ Scalable Architecture

Logic Apps can handle thousands of requests.
Easy to maintain and extend for other models or services.

✅ Governance and Monitoring

Centralized logging in Logic Apps.
Easy to integrate with Azure Monitor for alerting.

Use Case: Predicting Stock Prices

Imagine you have a machine learning model predicting HDFC high prices.

Power BI user clicks a “Predict” button.
Power Automate triggers a flow.
Flow sends stock price inputs to Logic Apps.
Logic Apps calls Databricks Model Serving.
Databricks returns the predicted high price.
Power BI visual updates dynamically with the prediction!

All of this happens securely, without exposing Databricks to the public internet. 🔒

Conclusion

Integrating Power BI with Databricks Model Serving under strict network security constraints can seem daunting.

But with the help of Logic Apps deployed inside a VNet, you can:

Securely bridge public and private networks
Enable real-time ML predictions in Power BI
Maintain enterprise-level security and compliance

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

JBs Wiki

Tag Archives: Enterprise Databricks Architecture

Integrating Power BI with Databricks Model Serving in Secure Networks Using Logic Apps and Power Automate