Tag Archives: SQL Server Performance

Standard

Posted by

Posted on

February 5, 2025

Posted under

Azure SQL Database

Comments

Leave a comment

Securely Scheduling Index Maintenance on Azure SQL Database Using Automation Account & Managed Identity

Index maintenance is a critical task for optimizing database performance, ensuring queries run efficiently, and keeping Azure SQL Database in top shape. Traditionally, scheduling index maintenance required storing credentials in scripts, posing security risks. However, in this blog, we’ll walk through a secure way to automate index maintenance using Azure Automation Account, Managed Identity, and PowerShell, ensuring no explicit credentials are stored. 🔒

Check out the video in you tube for a hand on experience.

✅ Why Automate Index Maintenance?

  • Enhances query performance by reducing fragmentation.
  • Ensures index optimization without manual intervention.
  • Uses Managed Identity for secure authentication, avoiding stored credentials.
  • Fully automated scheduling with Azure Automation Account.

🏗️ Prerequisites

Before we begin, ensure you have the following:

  • An Azure SQL Database instance.
  • An Azure Automation Account.
  • The Ola Hallengren IndexOptimize stored procedure installed on your database.
  • PowerShell Az module installed in the Automation Account.

🎯 Step 1: Create a Managed Identity for the Automation Account

To allow your Automation Account to securely authenticate against Azure SQL Database, assign it a System-Assigned Managed Identity:

  1. Navigate to Azure Portal → Automation Accounts.
  2. Select your Automation Account.
  3. Under Identity, enable System Assigned Managed Identity.
  4. Copy the Object ID; we’ll need it in the next step.

🔑 Step 2: Grant Database Access to the Managed Identity

Now, we need to create a login and database user in Azure SQL Database for the Managed Identity.

Run the following T-SQL script in your Azure SQL Database:

CREATE USER jbdbreindex FROM EXTERNAL PROVIDER;
ALTER ROLE db_owner ADD MEMBER jbdbreindex;

This grants db_owner permissions to the Managed Identity, allowing it to execute maintenance tasks.

🖥️ Step 3: Create an Azure Automation Runbook

Next, let’s create a PowerShell Runbook that connects to Azure SQL Database using the Managed Identity and executes the IndexOptimize stored procedure.

  1. In Azure Portal, navigate to Automation Accounts → Runbooks.
  2. Click Create a Runbook.
  3. Provide a name (e.g., IndexMaintenanceRunbook), select PowerShell as the Runbook type.
  4. Paste the following PowerShell script:
Write-Output "Index Maintenance started"

# Instantiate the connection to the SQL Database
$sqlConnection = new-object System.Data.SqlClient.SqlConnection

# Connect using Managed Identity
Connect-AzAccount -Identity

# Get a Token
$token = (Get-AzAccessToken -ResourceUrl https://database.windows.net ).Token

# Initialize Connection String
$sqlConnection.ConnectionString = "Data Source=jbsserver.database.windows.net;Initial Catalog=jbdb;Connect Timeout=60"

# Set the Token for authentication
$sqlConnection.AccessToken = $token

# Open the connection
$sqlConnection.Open()
Write-Output "Azure SQL database connection opened"

# Define the SQL Command
$sqlCommand = new-object System.Data.SqlClient.SqlCommand
$sqlCommand.CommandTimeout = 0
$sqlCommand.Connection = $sqlConnection

Write-Output "Issuing command to run stored procedure"

# Execute IndexOptimize stored procedure
$sqlCommand.CommandText= "EXECUTE dbo.IndexOptimize
@Databases = 'jbdb',
@FragmentationLow = NULL,
@FragmentationMedium = 'INDEX_REBUILD_ONLINE,INDEX_REBUILD_OFFLINE',
@FragmentationHigh = 'INDEX_REBUILD_ONLINE,INDEX_REBUILD_OFFLINE',
@FragmentationLevel1 = 10,
@FragmentationLevel2 = 30,
@LogToTable='Y'"

$result = $sqlCommand.ExecuteNonQuery()
Write-Output "Stored procedure execution completed"

# Close connection
$sqlConnection.Close()
Write-Output "Run completed"
  1. Click Save and Publish the Runbook.

⏰ Step 4: Schedule the Runbook

Now, let’s schedule the Runbook to run automatically.

  1. Go to Automation Account → Runbooks.
  2. Select your Runbook (IndexMaintenanceRunbook).
  3. Click Schedules → Add a Schedule.
  4. Set the recurrence (e.g., Daily at Midnight).
  5. Link the schedule to the Runbook.

📊 Step 5: Monitor Runbook Execution

To verify successful execution:

  1. Go to Automation Account → Runbooks → Job History.
  2. Check if the Runbook ran successfully.
  3. Use SELECT * FROM dbo.CommandLog in SQL to view maintenance logs.

🎉 Conclusion

Congratulations! 🎊 You have successfully scheduled Index Maintenance on Azure SQL Database securely using Azure Automation, PowerShell, and Managed Identity. This approach ensures:

  • 🔒 No explicit credentials are stored in scripts.
  • 📈 Automated performance tuning for SQL Database.
  • 🛠️ Seamless execution with minimal manual intervention.

Standard

Posted by

Posted on

November 12, 2024

Posted under

AlwaysON

Comments

Leave a comment

AlwaysON – Script to sync SQL Server Agent Jobs from Primary Replica to Secondary Replica in an Always On Availability Group

Environment

Blog29_1

-> Create a Job called “SQL Server Agent Job Synchronization” on all the Database Servers as part of your Alwayson Availability group. In my Environment, the Job will be created on Database Server JBSERVER1, JBSERVER2 and JBSERVER3. The Job “SQL Server Agent Job Synchronization” will have the below script executed as part of it.

-- Script to sync SQL Server Agent Jobs from Primary Replica to Secondary Replica in an Always On Availability Group
-- Dont forgot to change the listener name below
SET NOCOUNT ON;

DECLARE @primary_replica NVARCHAR(128),
        @local_replica NVARCHAR(128),
        @job_name NVARCHAR(128),
        @job_id UNIQUEIDENTIFIER,
        @tsql NVARCHAR(MAX),
        @sql NVARCHAR(MAX);

				

-- Get the primary replica name
SELECT @Primary_Replica = primary_replica
FROM sys.dm_hadr_availability_group_states a INNER JOIN sys.availability_group_listeners b
ON a.group_id=b.group_id where b.dns_name='DISL' ---Change the LISTENER NAME here

-- Get the current replica name (where this script is running)
SELECT @local_replica = @@SERVERNAME;

-- If this server is the primary replica, no need to sync jobs
IF @local_replica = @primary_replica
BEGIN
    PRINT 'This server is the primary replica. No job sync required.';
    RETURN;
END


-- Create a table to store jobs from the primary replica
IF OBJECT_ID('tempdb..#primary_jobs') IS NOT NULL
    DROP TABLE #primary_jobs;

CREATE TABLE #primary_jobs (
    job_id UNIQUEIDENTIFIER,
    job_name NVARCHAR(128)
);

-- Insert jobs from primary replica into the temp table
SET @sql = 'INSERT INTO #primary_jobs (job_id, job_name)
            SELECT job_id, name FROM [' + @primary_replica + '].msdb.dbo.sysjobs';

EXEC sp_executesql @sql;

-- Loop through jobs on primary replica and compare with local (secondary) replica
DECLARE job_cursor CURSOR FOR
SELECT job_id, job_name
FROM #primary_jobs;

OPEN job_cursor;
FETCH NEXT FROM job_cursor INTO @job_id, @job_name;

WHILE @@FETCH_STATUS = 0
BEGIN
    -- Check if the job exists on the local (secondary) replica
    IF NOT EXISTS (SELECT 1 FROM msdb.dbo.sysjobs WHERE name = @job_name)
    BEGIN
        PRINT 'Job missing on secondary replica: ' + @job_name;

        -- Script job creation from the primary replica
        DECLARE @job_creation_script NVARCHAR(MAX) = '';
        DECLARE @step_creation_script NVARCHAR(MAX) = '';
        DECLARE @schedule_creation_script NVARCHAR(MAX) = '';

        -- Step 1: Script the job creation
        SET @job_creation_script = 'EXEC msdb.dbo.sp_add_job @job_name = ''' + @job_name + ''', @enabled = 1, @description = ''' + @job_name + ''';';
        
        -- Step 2: Script the job steps from the primary replica
        DECLARE @step_id INT,
                @step_name NVARCHAR(128),
                @subsystem NVARCHAR(128),
                @command NVARCHAR(MAX),
                @on_success_action INT,
                @on_fail_action INT;
				

						set @sql=N''
				set @sql =         'SELECT step_id, step_name, subsystem, command, on_success_action, on_fail_action  INTO ##Primary_Job_jbs_wiki_details
        FROM [' + @primary_replica + '].msdb.dbo.sysjobsteps 
        WHERE job_id = '''+convert(nvarchar(max),@job_id)+''';'
		EXECUTE master.sys.sp_executesql @sql;

        DECLARE step_cursor CURSOR FOR 
        SELECT step_id, step_name, subsystem, command, on_success_action, on_fail_action 
        FROM ##Primary_Job_jbs_wiki_details;

        OPEN step_cursor;
        FETCH NEXT FROM step_cursor INTO @step_id, @step_name, @subsystem, @command, @on_success_action, @on_fail_action;

        WHILE @@FETCH_STATUS = 0
        BEGIN
		
            SET @step_creation_script = @step_creation_script + 'EXEC msdb.dbo.sp_add_jobstep 
                    @job_name = ''' + @job_name + ''', 
                    @step_name = ''' + @step_name + ''', 
                    @subsystem = ''' + @subsystem + ''', 
                    @command = ''' + REPLACE(@command, '''', '''''') + ''', 
                    @on_success_action = ' + CAST(@on_success_action AS NVARCHAR(10)) + ',
                    @on_fail_action = ' + CAST(@on_fail_action AS NVARCHAR(10)) + ';';
                    
            FETCH NEXT FROM step_cursor INTO @step_id, @step_name, @subsystem, @command, @on_success_action, @on_fail_action;
        END
		drop table ##Primary_Job_jbs_wiki_details
        CLOSE step_cursor;
        DEALLOCATE step_cursor;

        -- Step 3: Script the job schedule from the primary replica
        DECLARE @schedule_name NVARCHAR(128),
                @enabled INT,
                @freq_type INT,
                @freq_interval INT,
                @freq_subday_type INT,
                @freq_subday_interval INT,
                @freq_relative_interval INT,
                @freq_recurrence_factor INT,
                @active_start_date INT,
                @active_start_time INT;

				set @sql = N''
		set @sql = 'SELECT s.name, s.enabled, s.freq_type, s.freq_interval, s.freq_subday_type, s.freq_subday_interval, 
               s.freq_relative_interval, s.freq_recurrence_factor, s.active_start_date, s.active_start_time INTO ##Primary_Job_jbs_wiki_details1
        FROM [' + @primary_replica + '].msdb.dbo.sysschedules AS s
        INNER JOIN [' + @primary_replica + '].msdb.dbo.sysjobschedules AS js ON s.schedule_id = js.schedule_id
        WHERE js.job_id = '''+convert(nvarchar(max),@job_id)+''';'
		EXECUTE master.sys.sp_executesql @sql;

        DECLARE schedule_cursor CURSOR DYNAMIC FOR 
        SELECT s.name, s.enabled, s.freq_type, s.freq_interval, s.freq_subday_type, s.freq_subday_interval, 
               s.freq_relative_interval, s.freq_recurrence_factor, s.active_start_date, s.active_start_time 
        FROM ##Primary_Job_jbs_wiki_details1 s;

        OPEN schedule_cursor;
        FETCH NEXT FROM schedule_cursor INTO @schedule_name, @enabled, @freq_type, @freq_interval, @freq_subday_type, 
                                              @freq_subday_interval, @freq_relative_interval, @freq_recurrence_factor, 
                                              @active_start_date, @active_start_time;

        WHILE @@FETCH_STATUS = 0
        BEGIN
			SET @schedule_creation_script = @schedule_creation_script + 'EXEC msdb.dbo.sp_add_jobschedule 
                    @job_name = ''' + @job_name + ''', 
                    @name = ''' + @schedule_name + ''', 
                    @enabled = ' + CAST(@enabled AS NVARCHAR(10)) + ', 
                    @freq_type = ' + CAST(@freq_type AS NVARCHAR(10)) + ', 
                    @freq_interval = ' + CAST(@freq_interval AS NVARCHAR(10)) + ', 
                    @freq_subday_type = ' + CAST(@freq_subday_type AS NVARCHAR(10)) + ', 
                    @freq_subday_interval = ' + CAST(@freq_subday_interval AS NVARCHAR(10)) + ', 
                    @freq_relative_interval = ' + CAST(@freq_relative_interval AS NVARCHAR(10)) + ', 
                    @freq_recurrence_factor = ' + CAST(@freq_recurrence_factor AS NVARCHAR(10)) + ', 
                    @active_start_date = ' + CAST(@active_start_date AS NVARCHAR(10)) + ', 
                    @active_start_time = ' + CAST(@active_start_time AS NVARCHAR(10)) + ';';

            FETCH NEXT FROM schedule_cursor INTO @schedule_name, @enabled, @freq_type, @freq_interval, @freq_subday_type, 
                                                  @freq_subday_interval, @freq_relative_interval, @freq_recurrence_factor, 
                                                  @active_start_date, @active_start_time;
        END
		DROP TABLE ##Primary_Job_jbs_wiki_details1
        CLOSE schedule_cursor;
        DEALLOCATE schedule_cursor;

        -- Combine all scripts and execute to create the job on the secondary replica
        SET @tsql = @job_creation_script + @step_creation_script + @schedule_creation_script;

        EXEC sp_executesql @tsql;
        
        PRINT 'Job created on secondary replica: ' + @job_name;
    END

    FETCH NEXT FROM job_cursor INTO @job_id, @job_name;
END

CLOSE job_cursor;
DEALLOCATE job_cursor;

-- Cleanup
DROP TABLE #primary_jobs;


PRINT 'Job sync completed.';

-> Create a Linked Server to query the primary Replica. In my Environment, Linked servers JBSERVER2 and JBSERVER3 will be created on JBSERVER1. Linked servers JBSERVER1 and JBSERVER3 will be created on JBSERVER2. Linked servers JBSERVER1 and JBSERVER2 will be created on JBSERVER3.

-> The job will gracefully exit with a message “Script cannot run on primary Replica” if the job executes on Primary Replica. If the Job executes on the Secondary replica, It queries the list of SQL Server Agent Jobs on the primary replica and will create the jobs that are missing on the Secondary Replicas.

-> This solution just adds the missing jobs on the Secondary Replicas, but will not Drop Jobs on the Secondary Replica that are not present on the Primary.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

Standard

Posted by

Posted on

September 5, 2024

Posted under

SQL Server 2022

Comments

1 Comment

SQL Server 2022: Exploring the DATE_BUCKET Function

🕒SQL Server 2022 introduces several new and exciting features, and one of the standout additions is the DATE_BUCKET function. This function allows you to group dates into fixed intervals, making it easier to analyze time-based data. In this blog, we’ll dive into how DATE_BUCKET works, using the JBDB database for our demonstrations. We’ll also explore a business use case to showcase the function’s practical applications.🕒

Business Use Case: Analyzing Customer Orders 📊

Imagine a retail company, “Retail Insights,” that wants to analyze customer order data to understand purchasing patterns over time. Specifically, the company wants to group orders into weekly intervals to identify trends and peak periods. Using the DATE_BUCKET function, we can efficiently bucketize order dates into weekly intervals and perform various analyses.

Setting Up the JBDB Database

First, let’s set up our sample database and table. We’ll create a database named JBDB and a table Orders to store our order data.

-- Create JBDB Database
CREATE DATABASE JBDB;
GO

-- Use JBDB Database
USE JBDB;
GO

-- Create Orders Table
CREATE TABLE Orders (
    OrderID INT PRIMARY KEY IDENTITY(1,1),
    CustomerID INT,
    OrderDate DATETIME,
    TotalAmount DECIMAL(10, 2)
);
GO

Inserting Sample Data 📦

Next, we’ll insert some sample data into the Orders table to simulate a few months of order history.

-- Insert Sample Data into Orders Table
INSERT INTO Orders (CustomerID, OrderDate, TotalAmount)
VALUES
(1, '2022-01-05', 250.00),
(2, '2022-01-12', 300.50),
(1, '2022-01-19', 450.00),
(3, '2022-01-25', 500.75),
(4, '2022-02-01', 320.00),
(5, '2022-02-08', 275.00),
(2, '2022-02-15', 150.25),
(3, '2022-02-22', 600.00),
(4, '2022-03-01', 350.00),
(5, '2022-03-08', 425.75);
GO

Using the DATE_BUCKET Function 🗓️

The DATE_BUCKET function simplifies the process of grouping dates into fixed intervals. Let’s see how it works by bucketing our orders into weekly intervals.

-- Group Orders into Weekly Intervals Using DATE_BUCKET
SELECT 
    CustomerID,
    OrderDate,
    TotalAmount,
    DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderWeek
FROM Orders
ORDER BY OrderWeek;
GO

In the above query:

  • WEEK specifies the interval size.
  • 1 is the number of weeks per bucket.
  • OrderDate is the column containing the dates to be bucketed.
  • CAST('2022-01-01' AS datetime) is the reference date from which the intervals are calculated, cast to the datetime type to match OrderDate.

Analyzing Sales Trends 📈

Now that we have our orders grouped into weekly intervals, we can analyze sales trends, such as total sales per week.

-- Calculate Total Sales Per Week
SELECT 
    DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderWeek,
    SUM(TotalAmount) AS TotalSales
FROM Orders
GROUP BY DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime))
ORDER BY OrderWeek;
GO

This query helps “Retail Insights” identify peak sales periods and trends over time. For example, they might find that certain weeks have consistently higher sales, prompting them to investigate further.

Grouping by Month

SELECT 
    CustomerID,
    OrderDate,
    TotalAmount,
    DATE_BUCKET(MONTH, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderMonth
FROM Orders
ORDER BY OrderMonth;
GO

Analyzing Orders Per Customer

SELECT 
    CustomerID,
    COUNT(OrderID) AS NumberOfOrders,
    SUM(TotalAmount) AS TotalSpent,
    DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderWeek
FROM Orders
GROUP BY CustomerID, DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime))
ORDER BY OrderWeek;
GO

Counting Orders in Each Weekly Interval

This query counts the number of orders placed in each weekly interval.

-- Count Orders in Each Weekly Interval Using DATE_BUCKET
SELECT 
    DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderWeek,
    COUNT(OrderID) AS NumberOfOrders
FROM Orders
GROUP BY DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime))
ORDER BY OrderWeek;
GO

Average Order Value per Week

Calculate the average value of orders in each weekly interval.

-- Calculate Average Order Value Per Week
SELECT 
    DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderWeek,
    AVG(TotalAmount) AS AverageOrderValue
FROM Orders
GROUP BY DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime))
ORDER BY OrderWeek;
GO

Monthly Sales Analysis

Analyze total sales on a monthly basis.

-- Analyze Monthly Sales Using DATE_BUCKET
SELECT 
    DATE_BUCKET(MONTH, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderMonth,
    SUM(TotalAmount) AS MonthlySales
FROM Orders
GROUP BY DATE_BUCKET(MONTH, 1, OrderDate, CAST('2022-01-01' AS datetime))
ORDER BY OrderMonth;
GO

Identifying Peak Ordering Days

Identify the days with the highest total sales using daily buckets.

-- Identify Peak Ordering Days
SELECT 
    DATE_BUCKET(DAY, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderDay,
    SUM(TotalAmount) AS TotalSales
FROM Orders
GROUP BY DATE_BUCKET(DAY, 1, OrderDate, CAST('2022-01-01' AS datetime))
ORDER BY TotalSales DESC;
GO

Customer Order Frequency Analysis

Determine the frequency of orders for each customer on a weekly basis.

-- Customer Order Frequency Analysis Using DATE_BUCKET
SELECT 
    CustomerID,
    DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderWeek,
    COUNT(OrderID) AS OrdersPerWeek
FROM Orders
GROUP BY CustomerID, DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime))
ORDER BY CustomerID, OrderWeek;
GO

Weekly Revenue Growth Rate

Calculate the weekly growth rate in sales revenue.

-- Calculate Weekly Revenue Growth Rate
WITH WeeklySales AS (
    SELECT 
        DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderWeek,
        SUM(TotalAmount) AS WeeklySales
    FROM Orders
    GROUP BY DATE_BUCKET(WEEK, 1, OrderDate, CAST('2022-01-01' AS datetime))
)
SELECT 
    OrderWeek,
    WeeklySales,
    LAG(WeeklySales) OVER (ORDER BY OrderWeek) AS PreviousWeekSales,
    (WeeklySales - LAG(WeeklySales) OVER (ORDER BY OrderWeek)) / LAG(WeeklySales) OVER (ORDER BY OrderWeek) * 100 AS GrowthRate
FROM WeeklySales
ORDER BY OrderWeek;
GO

Orders Distribution Across Quarters

Analyze the distribution of orders across different quarters.

-- Distribution of Orders Across Quarters
SELECT 
    DATE_BUCKET(QUARTER, 1, OrderDate, CAST('2022-01-01' AS datetime)) AS OrderQuarter,
    COUNT(OrderID) AS NumberOfOrders
FROM Orders
GROUP BY DATE_BUCKET(QUARTER, 1, OrderDate, CAST('2022-01-01' AS datetime))
ORDER BY OrderQuarter;
GO

Business Insights 💡

Using the DATE_BUCKET function, “Retail Insights” can gain valuable insights into customer purchasing patterns:

  1. Identify Peak Periods: By analyzing weekly sales data, the company can pinpoint peak periods and prepare for increased demand.
  2. Marketing Strategies: Understanding customer behavior patterns helps in tailoring marketing strategies, such as promotions during slower periods.
  3. Inventory Management: Forecasting demand based on historical data enables better inventory planning and reduces stockouts or overstock situations.

Conclusion 🎉

The DATE_BUCKET function in SQL Server 2022 is a powerful tool for time-based data analysis. It simplifies the process of grouping dates into intervals, making it easier to extract meaningful insights from your data. Whether you’re analyzing sales trends, customer behavior, or other time-sensitive information, DATE_BUCKET can help streamline your workflow and improve decision-making.

Feel free to try these examples in your own environment and explore the potential of DATE_BUCKET in your data analysis tasks! Happy querying! 🚀

For more tutorials and tips on SQL Server, including performance tuning and database management, be sure to check out our JBSWiki YouTube channel.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.