Securely Scheduling Index Maintenance on Azure SQL Database Using Automation Account & Managed Identity

Index maintenance is a critical task for optimizing database performance, ensuring queries run efficiently, and keeping Azure SQL Database in top shape. Traditionally, scheduling index maintenance required storing credentials in scripts, posing security risks. However, in this blog, we’ll walk through a secure way to automate index maintenance using Azure Automation Account, Managed Identity, and PowerShell, ensuring no explicit credentials are stored. πŸ”’

Check out the video in you tube for a hand on experience.

βœ… Why Automate Index Maintenance?

  • Enhances query performance by reducing fragmentation.
  • Ensures index optimization without manual intervention.
  • Uses Managed Identity for secure authentication, avoiding stored credentials.
  • Fully automated scheduling with Azure Automation Account.

πŸ—οΈ Prerequisites

Before we begin, ensure you have the following:

  • An Azure SQL Database instance.
  • An Azure Automation Account.
  • The Ola Hallengren IndexOptimize stored procedure installed on your database.
  • PowerShell Az module installed in the Automation Account.

🎯 Step 1: Create a Managed Identity for the Automation Account

To allow your Automation Account to securely authenticate against Azure SQL Database, assign it a System-Assigned Managed Identity:

  1. Navigate to Azure Portal β†’ Automation Accounts.
  2. Select your Automation Account.
  3. Under Identity, enable System Assigned Managed Identity.
  4. Copy the Object ID; we’ll need it in the next step.

πŸ”‘ Step 2: Grant Database Access to the Managed Identity

Now, we need to create a login and database user in Azure SQL Database for the Managed Identity.

Run the following T-SQL script in your Azure SQL Database:

CREATE USER jbdbreindex FROM EXTERNAL PROVIDER;
ALTER ROLE db_owner ADD MEMBER jbdbreindex;

This grants db_owner permissions to the Managed Identity, allowing it to execute maintenance tasks.

πŸ–₯️ Step 3: Create an Azure Automation Runbook

Next, let’s create a PowerShell Runbook that connects to Azure SQL Database using the Managed Identity and executes the IndexOptimize stored procedure.

  1. In Azure Portal, navigate to Automation Accounts β†’ Runbooks.
  2. Click Create a Runbook.
  3. Provide a name (e.g., IndexMaintenanceRunbook), select PowerShell as the Runbook type.
  4. Paste the following PowerShell script:
Write-Output "Index Maintenance started"

# Instantiate the connection to the SQL Database
$sqlConnection = new-object System.Data.SqlClient.SqlConnection

# Connect using Managed Identity
Connect-AzAccount -Identity

# Get a Token
$token = (Get-AzAccessToken -ResourceUrl https://database.windows.net ).Token

# Initialize Connection String
$sqlConnection.ConnectionString = "Data Source=jbsserver.database.windows.net;Initial Catalog=jbdb;Connect Timeout=60"

# Set the Token for authentication
$sqlConnection.AccessToken = $token

# Open the connection
$sqlConnection.Open()
Write-Output "Azure SQL database connection opened"

# Define the SQL Command
$sqlCommand = new-object System.Data.SqlClient.SqlCommand
$sqlCommand.CommandTimeout = 0
$sqlCommand.Connection = $sqlConnection

Write-Output "Issuing command to run stored procedure"

# Execute IndexOptimize stored procedure
$sqlCommand.CommandText= "EXECUTE dbo.IndexOptimize
@Databases = 'jbdb',
@FragmentationLow = NULL,
@FragmentationMedium = 'INDEX_REBUILD_ONLINE,INDEX_REBUILD_OFFLINE',
@FragmentationHigh = 'INDEX_REBUILD_ONLINE,INDEX_REBUILD_OFFLINE',
@FragmentationLevel1 = 10,
@FragmentationLevel2 = 30,
@LogToTable='Y'"

$result = $sqlCommand.ExecuteNonQuery()
Write-Output "Stored procedure execution completed"

# Close connection
$sqlConnection.Close()
Write-Output "Run completed"
  1. Click Save and Publish the Runbook.

⏰ Step 4: Schedule the Runbook

Now, let’s schedule the Runbook to run automatically.

  1. Go to Automation Account β†’ Runbooks.
  2. Select your Runbook (IndexMaintenanceRunbook).
  3. Click Schedules β†’ Add a Schedule.
  4. Set the recurrence (e.g., Daily at Midnight).
  5. Link the schedule to the Runbook.

πŸ“Š Step 5: Monitor Runbook Execution

To verify successful execution:

  1. Go to Automation Account β†’ Runbooks β†’ Job History.
  2. Check if the Runbook ran successfully.
  3. Use SELECT * FROM dbo.CommandLog in SQL to view maintenance logs.

πŸŽ‰ Conclusion

Congratulations! 🎊 You have successfully scheduled Index Maintenance on Azure SQL Database securely using Azure Automation, PowerShell, and Managed Identity. This approach ensures:

  • πŸ”’ No explicit credentials are stored in scripts.
  • πŸ“ˆ Automated performance tuning for SQL Database.
  • πŸ› οΈ Seamless execution with minimal manual intervention.

Resolving ADF Pipeline Errors: “Client Address Not Authorized” by Whitelisting IP Addresses for Azure Key Vault

When working with Azure Data Factory (ADF) pipelines, secure communication between various Azure services is crucial, especially when accessing sensitive data stored in resources like Azure Key Vault. Recently, while executing an ADF pipeline, I encountered the following error message:

Operation on target Web URLToken TenantId failed: {“error”:{“code”:”Forbidden”,”message”:”Client address is not authorized and caller is not a trusted service.\r\nClient address: xx.xx.xx.xxx\r\nCaller: appid=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx;oid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx;providers/Microsoft.DataFactory/factories/datafactoryjb;abc_er_sde=/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/jb_ADF_RG/providers/Microsoft.DataFactory/factories/datafactoryjb\r\nVault: jbkeyvault;location=northeurope”,”innererror”:{“code”:”ForbiddenByFirewall”}}}

This error indicates that the client address attempting to access the Azure Key Vault was not authorized, preventing the pipeline from retrieving necessary secrets stored in the vault. The root cause of this issue was that the Azure Key Vault had strict network security settings that only allowed authorized IP addresses to access it. The ADF instance running in the North Europe region was not on the list of authorized addresses, causing the connection to be blocked.

In this blog post, I will walk you through how I resolved the issue by whitelisting the required IP addresses for Data Factory and ensuring secure communication between Azure Data Factory and Azure Key Vault.

Understanding the Error: Client Address Not Authorized

The error message received during the pipeline execution highlights two key aspects:

  • Client Address Not Authorized: The IP address attempting to connect to the Key Vault (ADF service in this case) was not whitelisted, which led to the request being blocked by the Key Vault’s firewall.
  • Forbidden by Firewall: The error code ForbiddenByFirewall suggests that the Key Vault is configured with firewall rules that only allow specific IP addresses to access it.

By default, Azure Key Vault can restrict access to its secrets by using firewall settings to specify which IP ranges or services are allowed to connect. This provides an extra layer of security, but in this case, it became an obstacle because the IP addresses associated with the ADF instance were not permitted.

Steps to Resolve the Issue

To resolve this issue, I followed these steps to add the necessary IP addresses to the Azure Key Vault’s firewall settings:

1. Identify the Region of the Azure Data Factory

The first step was to identify the region in which my Azure Data Factory instance was running. In this case, the ADF instance was running in the North Europe region, as seen from the error message and confirmed in the Azure portal.

2. Retrieve the IP Address Range for ADF in North Europe

Azure services like Data Factory operate from a specific set of IP ranges depending on the region they are deployed in. Microsoft publishes the IP address ranges for each service and region, which are updated periodically. You can download the full list of IP ranges for different Azure services and regions from this Microsoft link.

In this case, the required IP addresses for Data Factory in the North Europe region were as follows:

{
  "name": "DataFactory.NorthEurope",
  "id": "DataFactory.NorthEurope",
  "properties": {
    "changeNumber": 7,
    "region": "northeurope",
    "regionId": 17,
    "platform": "Azure",
    "systemService": "DataFactory",
    "addressPrefixes": [
      "4.207.242.72/29",
      "13.69.230.96/28",
      "13.74.108.224/28",
      "20.38.80.192/26",
      "20.38.82.0/23",
      "20.50.68.56/29",
      "20.223.64.60/32",
      "20.223.65.144/29",
      "48.209.130.96/28",
      "48.209.130.112/29",
      "52.138.229.32/28"
    ]
  }
}

You can get the details from Download Azure IP Ranges and Service Tags – Public Cloud from Official Microsoft Download Center.

These IP addresses need to be whitelisted in the Azure Key Vault to allow ADF to access the secrets.

3. Whitelist the IP Addresses in Azure Key Vault

To whitelist the necessary IP ranges, follow these steps:

  1. Navigate to the Azure Key Vault in the Azure portal.
  2. Select Networking from the left-side menu.
  3. Under Firewalls and virtual networks, ensure that you have selected to allow access from selected networks.
  4. Add the necessary IP address ranges to the firewall rules:
    • “4.207.242.72/29”
    • “13.69.230.96/28”
    • “13.74.108.224/28”
    • “20.38.80.192/26”
    • “20.38.82.0/23”
    • “20.50.68.56/29”
    • “20.223.64.60/32”
    • “20.223.65.144/29”
    • “48.209.130.96/28”
    • “48.209.130.112/29”
    • “52.138.229.32/28”
  5. Once the IP addresses are added, save the changes to apply the updated firewall rules.

4. Validate the Pipeline Run

After whitelisting the required IP addresses in the Key Vault, I re-ran the Azure Data Factory pipeline. This time, the pipeline successfully connected to the Key Vault and retrieved the required secrets, and the entire workflow executed without any issues.


Conclusion

The error "Client address is not authorized" occurred because the Azure Key Vault was configured with strict firewall rules, preventing Azure Data Factory from accessing it. By identifying the region of the ADF instance and retrieving the correct IP ranges for that region, I was able to whitelist the necessary addresses and resolve the issue.

This process highlights the importance of maintaining secure communication between Azure services, and how careful configuration of firewall rules can help both secure and enable workflows. Always ensure that the IP ranges for the services you are using are updated and whitelisted in sensitive resources like Azure Key Vault to prevent any disruptions in your pipelines.


Additional Resources:

Regards;
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided β€œAS IS” with no warranties, and confers no rights.