AlwaysON – Unable to access the ‘JB1’ database because no online secondary replicas are enabled for read-only access. (Microsoft SQL Server, Error: 982)

Environment

Blog29_1

-> The Application makes a connection to the Database JB1 with ApplicationIntent=ReadOnly and receives below message,

Unable to access the ‘JB1’ database because no online secondary replicas are enabled for read-only access. Check the availability group configuration to verify that at least one secondary replica is configured for read-only access. Wait for an enabled replica to come online, and retry your read-only operation.
Changed database context to ‘JB1’. (Microsoft SQL Server, Error: 982)

-> The message indicates that there are no active Read-Only partners. When checked further it is clear that Server JBSERVER2 and JBSERVER3 were down.

-> We were advised that the Servers JBSERVER2 and JBSERVER3 will not be up for next 6 Hours. Now the Application team doesn’t want to change their connection string by dropping ApplicationIntent=ReadOnly and wanted us to make sure that the primary accepts Read connections.

-> The Alwayson Availability group setting is as below,

Blog35_1.PNG

-> We changed the setting “Connections in Primary Role” from “Allow read/write connecions” to “Allow all connections” as below,

Blog35_2.PNG

-> The Application started connecting to the Primary Replica even with ApplicationIntent=ReadOnly.

-> This now brings up the question, why can’t we have the setting “Connections in Primary Role” set to “Allow all connections” instead of “Allow read/write connecions”. “Allow all connections”  can be an issue as Connections where the Application Intent is set to ReadOnly are not disallowed on the Primary Replica anymore. With this setting in place the read workloads may execute on both Primary and the secondary, If server JBSERVER2 which is on the same data centre as the PRIMARY goes down and we have intermittent network issues between DataCenters which makes JBSERVER3 to be offline Intermittently for JBSERVER1. So the Read workload might run on both JBSERVER1 and JBSERVER3.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

Advertisements

AlwaysON – How to disable Read-Only routing for an Availability Group

-> We have configured Read-Only routing list for an availability group. Now the requirement changes and the Read-Only routing needs to be disabled. If you have a scenario like this in your hand, then this post is for you!

-> Please refer article “https://jbswiki.com/2017/09/06/configuring-read-only-routing-list-for-alwayson-availability-group/” to create read-Only routing list for you Availability group.

Environment

Blog29_1

-> Looking at SSMS,

Blog_34_1.PNG

-> Checking the Read-Routing List that is setup currently on the Primary Replica using below query,

SELECT ag.name as “Availability Group”, ar.replica_server_name as “When Primary Replica Is”,
rl.routing_priority as “Routing Priority”, ar2.replica_server_name as “RO Routed To”,
ar.secondary_role_allow_connections_desc, ar2.read_only_routing_url
FROM sys.availability_read_only_routing_lists rl
inner join sys.availability_replicas ar on rl.replica_id = ar.replica_id
inner join sys.availability_replicas ar2 on rl.read_only_replica_id = ar2.replica_id
inner join sys.availability_groups ag on ar.group_id = ag.group_id
ORDER BY ag.name, ar.replica_server_name, rl.routing_priority

Blog_34_2.PNG

-> Trying a connection with Switch -KReadonly to see if it Read-Routing list works and connects to JBSERVER3.

Blog_34_3.PNG

-> We will Disable the Read-Routing list using below query,

ALTER AVAILABILITY GROUP [AG1]
MODIFY REPLICA ON
N’JBSERVER1′ WITH
(PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=NONE));
ALTER AVAILABILITY GROUP [AG1]
MODIFY REPLICA ON
N’JBSERVER2′ WITH
(PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=NONE));
ALTER AVAILABILITY GROUP [AG1]
MODIFY REPLICA ON
N’JBSERVER3′ WITH
(PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=NONE));

-> Checking the Read-Routing details on the Primary Replica after it is removed,

SELECT ag.name as “Availability Group”, ar.replica_server_name as “When Primary Replica Is”,
rl.routing_priority as “Routing Priority”, ar2.replica_server_name as “RO Routed To”,
ar.secondary_role_allow_connections_desc, ar2.read_only_routing_url
FROM sys.availability_read_only_routing_lists rl
inner join sys.availability_replicas ar on rl.replica_id = ar.replica_id
inner join sys.availability_replicas ar2 on rl.read_only_replica_id = ar2.replica_id
inner join sys.availability_groups ag on ar.group_id = ag.group_id
ORDER BY ag.name, ar.replica_server_name, rl.routing_priority

Blog_34_4.PNG

-> It is evident from the above screenshot that the Read-Routing listi ndeed is removed. Lets check what happens if we connect the listener using -KReadonly switch,

Blog_34_5.PNG

-> It connects to the primary Replica. this proves that read-only routing is indeed disabled.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

Creating Linked Server to a Multi-Subnet Availability Group Listener with ReadOnly routing

-> A multi-subnet environment is defined when the OS cluster used for AlwaysOn has server nodes that are located in multiple, different subnets. By default, the behavior of the SQL client libraries is to try all IP addresses returned by the DNS lookup – one after another (serially) until the all of the IP addresses have been exhausted and either a connection is made, or a connection timeout threshold has been reached.

-> Microsoft added a new connection string parameter that can be added to change the connection behavior. This new parameter, MultiSubnetFailover, should be used and set to “TRUE.” When set to TRUE, the connection attempt behavior changes. It will no longer attempt all of the IP addresses serially, but in parallel.

-> The parameter MultiSubnetFailover cannot be used in the Linked Server as part of the provider string. We will have to create a Data Source and then use the Data Source in the Linked Server.

Blog33_1.PNG

Blog33_2.PNG

-> I am using Windows Aunthentication to connect the Listener. If you want to use SQL Server Authentication, select as appropriate.

Blog33_3.PNG

-> Change “Change the default database to:”  as indicated in the below screenshot from database Master to a database that is part of Alwayson Availability group that is part of the listener. Make sure the Application Intent is READONLY and Multi-subnet failover is checked.

Blog33_4

Blog33_5.PNG

Blog33_6.PNG

Blog33_7.PNG

-> Once the Data Source is created and tested. Move onto creating the Linked server.

Blog33_8.PNG

-> Select “Be made using this security context” and provide the SQL Server Authentication Credential if required. I am using Windows authentication, so using “Be made using the login’s current securoty context”.

Blog33_9.PNG

-> Once the Linked server is created. Execute the below query and check if the output returned is your readable secondary and not pointing to the primary.

select * from openquery([JBSERVER1],’select @@servername’)

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

AlwaysON – Working through Crashed Synchronous Secondary Replica

Environment

Blog29_1

-> JBSERVER2 Crashes. The Server is expected to be online after 2 days.

Blog32_2.PNG

-> Checking the SQL Server Management Studio and the Cluster Administrator,

Blog32_3

Blog32_4

-> JBSERVER3 which is an Asynchronous replica, will be changed to a Synchronous replica.

-> Removing the Crashed Replica from Availability group,

Blog32_5.PNG

Blog32_6

Blog32_7.PNG

-> Remove the Node JBSERVER2 from Cluster Administrator,

Blog32_8.PNG

Blog32_9

Blog32_10

-> Server JBSERVER2 comes online after the Rebuild. Lets Add it back to the Availability group.

-> Adding JBSERVER2 to the Failover Cluster,

Blog32_11.PNG

Blog32_12.PNG

Blog32_13.PNG

-> Adding JBSERVER2 as a Replica,

Blog32_14.PNG

Blog32_15.PNG

Blog32_16.PNG

Blog32_17.PNG

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

AlwaysON – Login Synchronization across Replicas

Environment

Blog29_1

-> Create a Job called “Login Synchronization” on all the Database Servers as part of your Alwayson Availability group. In my Environment, the Job will be created on Database Server JBSERVER1, JBSERVER2 and JBSERVER3. The Job “Login Synchronization” will have the below script executed as part of it.

set nocount on
create table #Sync_Logins (Script varchar(max))
Declare @sql nvarchar(max)
Declare @Primary_Replica varchar(20)
SELECT @Primary_Replica = primary_replica
FROM sys.dm_hadr_availability_group_states a INNER JOIN sys.availability_group_listeners b
ON a.group_id=b.group_id where b.dns_name=’PPN-VDBSL’
IF (@Primary_Replica= @@servername) BEGIN;
Print N’Script cannot run on primary Replica’;
drop table #Sync_Logins
RETURN;
END;
SET @sql=N”;
set @sql = ‘SELECT ”If not Exists (select loginname from master.dbo.syslogins where name = ””” +name +”””’+’) BEGIN CREATE LOGIN ” + QUOTENAME(name) + ” WITH PASSWORD=”
+ sys.fn_varbintohexstr(password_hash) + ” HASHED, SID=”
+ sys.fn_varbintohexstr(sid) + ”, ”
+ ”DEFAULT_DATABASE=”+ QUOTENAME(COALESCE(default_database_name, ”master”))
+ ”, DEFAULT_LANGUAGE=” + QUOTENAME(COALESCE(default_language_name,
”us_english”))
+ ”, CHECK_EXPIRATION=” + CASE is_expiration_checked WHEN 1 THEN ”ON” ELSE
”OFF” END
+ ”, CHECK_POLICY=” + CASE is_policy_checked WHEN 1 THEN ”ON” ELSE ”OFF” END + ” END”
FROM [‘+@Primary_Replica+’].master.sys.sql_logins
WHERE name<>”sa”
UNION ALL
–Windows logins:
SELECT ”If not Exists (select loginname from master.dbo.syslogins where name = ”””+ name +”””’+’) BEGIN CREATE LOGIN ” + QUOTENAME(name) + ” FROM WINDOWS WITH ”
+ ”DEFAULT_DATABASE=”+ QUOTENAME(COALESCE(default_database_name, ”master”))
+ ”, DEFAULT_LANGUAGE=” + QUOTENAME(COALESCE(default_language_name,
”us_english”))+ ” END”
FROM [‘+@Primary_Replica+’].master.sys.server_principals
WHERE type IN (”U”,”G”)
AND name NOT LIKE ”%\SQLServer2005MSSQLUser$%$%”
AND name NOT LIKE ”%\SQLServer2005SQLAgentUser$%$%”
AND name NOT LIKE ”%\SQLServer2005MSFTEUser$%$%”
AND name NOT IN (”BUILTIN\Administrators”, ”NT AUTHORITY\SYSTEM”);’
insert into #Sync_Logins
execute sp_executesql @sql
SET @sql=N”;
SET @sql = ‘SELECT ”EXEC sp_addsrvrolemember ” + QUOTENAME(L.name) + ”, ” +
QUOTENAME(R.name)
FROM [‘+@Primary_Replica+’].master.sys.server_principals L JOIN [‘+@Primary_Replica+’].master.sys.server_role_members RM
ON L.principal_id=RM.member_principal_id
JOIN [‘+@Primary_Replica+’].master.sys.server_principals R
ON RM.role_principal_id=R.principal_id
WHERE L.type IN (”U”,”G”,”S”)
AND L.name NOT LIKE ”%\SQLServer2005MSSQLUser$%$%”
AND L.name NOT LIKE ”%\SQLServer2005SQLAgentUser$%$%”
AND L.name NOT LIKE ”%\SQLServer2005MSFTEUser$%$%”
AND L.name NOT IN (”BUILTIN\Administrators”, ”NT AUTHORITY\SYSTEM”, ”sa”);’
insert into #Sync_Logins
execute sp_executesql @sql
SET @sql=N”;
SELECT @sql=@sql+’ ‘+[Script] FROM #Sync_Logins;
EXECUTE master.sys.sp_executesql @sql;
drop table #Sync_Logins

-> Create a Linked Server to query the primary Replica. In my Environment, Linked servers JBSERVER2 and JBSERVER3 will be created on JBSERVER1. Linked servers JBSERVER1 and JBSERVER3 will be created on JBSERVER2. Linked servers JBSERVER1 and JBSERVER2 will be created on JBSERVER3.

-> The job will gracefully exit with a message “Script cannot run on primary Replica” if the job executes on Primary Replica. If the Job executes on the Secondary replica, It queries the list of Logins on the primary replica and will create the logins that are missing on the Secondary Replicas.

-> This solution just adds the missing Logins on the Secondary Replicas, but will not Drop logins on the Secondary Replica that are not present on the Primary.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

Difference between Readable Secondary setting Yes and Read Intent only

-> Readable Secondary setting in AlwaysON is discussed in article “https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/configure-read-only-access-on-an-availability-replica-sql-server” is as below,

No
No user connections are allowed to secondary databases of this replica. They are not available for read access. This is the default setting.

Read-intent only
Only read-only connections are allowed to secondary databases of this replica. The secondary database(s) are all available for read access.

Yes
All connections are allowed to secondary databases of this replica, but only for read access. The secondary database(s) are all available for read access.

Environment

Blog29_1.PNG

-> Creating the Read-Only routing list using below query,

ALTER AVAILABILITY GROUP AG1
MODIFY REPLICA ON
N’JBSERVER1′ WITH
(SECONDARY_ROLE (ALLOW_CONNECTIONS = READ_ONLY));
ALTER AVAILABILITY GROUP AG1
MODIFY REPLICA ON
N’JBSERVER1′ WITH
(SECONDARY_ROLE (READ_ONLY_ROUTING_URL = N’TCP://JBSERVER1.JBS.COM:1433′));
ALTER AVAILABILITY GROUP AG1
MODIFY REPLICA ON
N’JBSERVER2′ WITH
(SECONDARY_ROLE (ALLOW_CONNECTIONS = READ_ONLY));
ALTER AVAILABILITY GROUP AG1
MODIFY REPLICA ON
N’JBSERVER2′ WITH
(SECONDARY_ROLE (READ_ONLY_ROUTING_URL = N’TCP://JBSERVER2.JBS.COM:1433′));
ALTER AVAILABILITY GROUP AG1
MODIFY REPLICA ON
N’JBSERVER3′ WITH
(SECONDARY_ROLE (ALLOW_CONNECTIONS = READ_ONLY));
ALTER AVAILABILITY GROUP AG1
MODIFY REPLICA ON
N’JBSERVER3′ WITH
(SECONDARY_ROLE (READ_ONLY_ROUTING_URL = N’TCP://JBSERVER3.JBS.COM:1433′));
ALTER AVAILABILITY GROUP AG1
MODIFY REPLICA ON
N’JBSERVER1′ WITH
(PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=(‘JBSERVER3′,’JBSERVER2’)));
ALTER AVAILABILITY GROUP AG1
MODIFY REPLICA ON
N’JBSERVER2′ WITH
(PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=(‘JBSERVER3′,’JBSERVER1’)));
ALTER AVAILABILITY GROUP AG1
MODIFY REPLICA ON
N’JBSERVER3′ WITH
(PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=(‘JBSERVER2′,’JBSERVER1’)));

-> The Readable Secondary is set to “Read-intent only”,

Blog29_2.PNG

-> Connecting to the secondary Replica JBSERVER3 on database JB1 directly and trying a select query,

Blog29_3

-> It is pretty clear that we are not able to make a connection to database JB1 on Server JBSERVER3 directly. Let us try a connection using ApplicationIntent=ReadOnly and it connects fine,

Blog29_4Blog29_5Blog29_6

-> This advises us that we cannot read the readable secondaries without a ApplicationIntent = ReadOnly.

-> The Readable Secondary is set to “Yes”,

Blog29_7.PNG

-> Connecting to the secondary Replica JBSERVER3 on database JB1 directly and trying a select query. It works fine without any issues,

Blog29_8.PNG

-> In short, Readable Secondary = “Read Intent only” requires ApplicationIntent=ReadOnly to execute a select query on the secondary. Readable Secondary = “Yes” allows us to execute a select query on the seocndary directly.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

Reporting Services using AlwaysON Availability Group

Environment

Blog28_1.PNG

->SQL Server Reporting Services present in JBSERVER4 is configured using Database Server JBSERVER1 initially.

-> SQL Server Reporting Services databases are later placed on the Availability Group JBS_SSRSAG.

-> Database [ReportServerTempDB] recovery model will be changed to Full from Simple.

-> Adding the [ReportServer] and [ReportServerTempDB] database to the Availability group JB_SSRSAG.

Blog27_3

Blog28_2.PNG

Blog28_3.PNG

Blog27_6

Blog27_7

Blog28_4.PNG

Blog27_9.PNG

Blog28_5.PNG

-> Checking the new Availability group in SQL Server management Studio,

Blog28_6.PNG

-> Reconfiguring the SQL Server Reporting Services on JBSERVER4 to use the Availability group listener,

Blog28_7.PNG

Blog28_8.PNG

Blog28_9.PNG

Blog28_10.PNG

Blog28_11.PNG

-> Executing a Sample report that uses Availability group Listener JBS_APP that has read-only routing configured.

-> Please refer article “Configuring Read Only Routing List for AlwaysON Availability Group” to get more details on Availability group JBS_AG and its Listener JBS_APP.

-> The data source on the report doesn’t use Application Intent,

Blog28_12.PNG

-> Executing the report and checking the profiler trace. The queries are executed on the Primary JBSERVER1.

Blog28_13.PNG

Blog28_14.PNG

-> The data source on the report uses the Application Intent to ReadOnly,

Blog28_15.PNG

-> Executing the report and checking the profiler trace. The queries are executed on the Secondary JBSERVER3.

Blog28_13

Blog28_16

SQL Server Reporting Services Subscription Jobs

-> Execute the below query only once on database server JBSERVER1, JBSERVER2 and JBSERVER3. The table stores the details whether that particular replica is a primary or secondary replica.

create table AG_role (Role int)
insert into Ag_Role select sys.fn_hadr_is_primary_replica(N’ReportServer’)

-> Save the below script as a powershell script file on Server JBSERVER1, JBSERVER2 and JBSERVER3. Please make sure that $AGReplica is changed appropriately in the script on each of the server before saving.

$AGReplica = “JBSERVER1”;
$AG_ROLE_Previous = Invoke-Sqlcmd “select * from AG_role;” -ServerInstance $AGReplica;
$AG_ROLE_Current = Invoke-Sqlcmd “select sys.fn_hadr_is_primary_replica(N’ReportServer’) as Role;” -ServerInstance $AGReplica;
If ($AG_ROLE_Previous.Role -eq $AG_ROLE_Current.Role)
{
“No Failover occurred”
}
else
{
“Failover occurred”
#Delete the SQL Server agent jobs related to Subscription

Invoke-Sqlcmd “DECLARE @TSQL NVARCHAR(MAX) = N”;
SELECT
@TSQL = @TSQL + ‘EXEC msdb.dbo.sp_delete_job @job_name = N”’ + j.name + ”” + CHAR(13)
FROM msdb.dbo.sysjobs AS j
JOIN msdb.dbo.syscategories AS c
ON j.category_id = c.category_id
WHERE c.name = N’Report Server’;
EXEC sp_executesql @TSQL;” -ServerInstance $AGReplica;

If ($AG_ROLE_Current.Role -eq 1)
{
“Failover Occurred”
$SSRSServers = Invoke-Sqlcmd “If (SELECT sys.fn_hadr_is_primary_replica(N’ReportServer’)) = 1 SELECT MachineName FROM [ReportServer].[dbo].[Keys] WHERE MachineName IS NOT NULL” -ServerInstance $AGReplica;
Foreach ($SSRS in $SSRSServers)
{
Get-Service -Name “ReportServer” -ComputerName $SSRS.MachineName | Restart-Service;
}
}

}

Invoke-Sqlcmd “delete from AG_Role;” -ServerInstance $AGReplica;
Invoke-Sqlcmd “insert into AG_Role SELECT sys.fn_hadr_is_primary_replica(N’ReportServer’);” -ServerInstance $AGReplica;

->On the Secondary replica, the script basically deletes all the jobs related to SSRS subscription in the SQL Server Agent when it detects a failover.

->On the primary replica, the script deletes all the jobs related to SSRS subscription in the SQL Server Agent and restarts the SQL Server Reporting Services on JBSERVER4 if it detects a failover. This ensures that the subscription jobs are recreated on the primary replica.

-> Create a SQL Server agent job to call the Powershell script saved in the previous step under a job step with type PowerShell. The job should run every 1 minute and also when the SQL Server Agent starts.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.