-> Application was failing with below error,

<sub>TITLE: Connect to Database Engine
Cannot connect to JBSAG1.
ADDITIONAL INFORMATION:

Unable to access availability database ‘JBSAG12’ because the database replica is not in the PRIMARY or SECONDARY role. Connections to an availability database is permitted only when the database replica is in the PRIMARY or SECONDARY role. Try the operation again later. (Microsoft SQL Server, Error: 983)
For help, click: https://docs.microsoft.com/sql/relational-databases/errors-events/mssqlserver-983-database-engine-error</sub>

---

Microsoft ODBC Driver 17 for SQL Server : Unable to access availability database ‘JBSAG12’ because the database replica is not in the PRIMARY or SECONDARY role. Connections to an availability database is permitted only when the database replica is in the PRIMARY or SECONDARY role. Try the operation again later.

---

-> I checked the database and Always ON availability group status. Database was “Not Synchronizing” and Availability group was resolving,

SQL Server Errorlog

JBSAG1

_{2022-06-13 06:42:24.070 spid58 Always On: The local replica of availability group ‘JBAG12’ is preparing to transition to the resolving role. This is an informational message only. No user action is required.
2022-06-13 06:42:24.070 Server The availability group ‘JBAG12’ is being asked to stop the lease renewal because the availability group is going offline. This is an informational message only. No user action is required.
2022-06-13 06:42:24.070 spid58 The state of the local availability replica in availability group ‘JBAG12’ has changed from ‘PRIMARY_NORMAL’ to ‘RESOLVING_NORMAL’. The state changed because the availability group is going offline. The replica is going offline because the associated availability group has been deleted, or the user has taken the associated availability group offline in Windows Server Failover Clustering (WSFC) management console, or the availability group is failing over to another SQL Server instance. For more information, see the SQL Server error log or cluster log. If this is a Windows Server Failover Clustering (WSFC) availability group, you can also see the WSFC management console.
2022-06-13 06:42:24.070 spid78s Always On Availability Groups connection with secondary database terminated for primary database ‘JBSAG12’ on the availability replica ‘JBSAG2’ with Replica ID: {0e5b3832-3dda-41cb-9730-0f119639b535}. This is an informational message only. No user action is required.
2022-06-13 06:42:24.080 spid42s The availability group database “JBSAG12” is changing roles from “PRIMARY” to “RESOLVING” because the mirroring session or availability group failed over due to role synchronization. This is an informational message only. No user action is required.
2022-06-13 06:42:24.090 spid42s State information for database ‘JBSAG12’ – Hardened Lsn: ‘(37:1184:1)’ Commit LSN: ‘(37:1144:5)’ Commit Time: ‘Jun 12 2022 4:48AM’
2022-06-13 06:42:24.090 spid42s Microsoft Distributed Transaction Coordinator (MS DTC) resource manager [JBSAG12] has been released. This is an informational message only. No user action is required.
2022-06-13 06:44:55.330 Logon Error: 983, Severity: 14, State: 1.
2022-06-13 06:44:55.330 Logon Unable to access availability database ‘JBSAG12’ because the database replica is not in the PRIMARY or SECONDARY role. Connections to an availability database is permitted only when the database replica is in the PRIMARY or SECONDARY role. Try the operation again later.
2022-06-13 06:55:06.730 Logon Error: 983, Severity: 14, State: 1.
2022-06-13 06:55:06.730 Logon Unable to access availability database ‘JBSAG12’ because the database replica is not in the PRIMARY or SECONDARY role. Connections to an availability database is permitted only when the database replica is in the PRIMARY or SECONDARY role. Try the operation again later.}

JBSAG2

_{2022-06-13 06:36:11.050 spid25s The recovery LSN (37:1176:1) was identified for the database with ID 6. This is an informational message only. No user action is required.
2022-06-13 06:42:23.960 spid29s Always On: The local replica of availability group ‘JBAG12’ is going offline because the corresponding resource in the Windows Server Failover Clustering (WSFC) cluster is no longer online. This is an informational message only. No user action is required.
2022-06-13 06:42:23.960 spid29s The state of the local availability replica in availability group ‘JBAG12’ has changed from ‘SECONDARY_NORMAL’ to ‘RESOLVING_NORMAL’. The state changed because the availability group state has changed in Windows Server Failover Clustering (WSFC). For more information, see the SQL Server error log or cluster log. If this is a Windows Server Failover Clustering (WSFC) availability group, you can also see the WSFC management console.
2022-06-13 06:42:23.960 spid41s Always On Availability Groups connection with primary database terminated for secondary database ‘JBSAG12’ on the availability replica ‘JBSAG1’ with Replica ID: {cf81c5d4-0dbf-4573-ab9d-018423fb92b2}. This is an informational message only. No user action is required.
2022-06-13 06:42:23.960 spid47s The availability group database “JBSAG12” is changing roles from “SECONDARY” to “RESOLVING” because the mirroring session or availability group failed over due to role synchronization. This is an informational message only. No user action is required.
2022-06-13 06:42:23.960 spid47s State information for database ‘JBSAG12’ – Hardened Lsn: ‘(37:1184:1)’ Commit LSN: ‘(37:1144:5)’ Commit Time: ‘Jun 12 2022 4:48AM’}

-> In my case, there was a network issue wherein the nodes were removed from cluster due to quorum loss and then came back online automatically. But the Always On Cluster role went down as shown in below screenshot,

-> I was sure that Cluster was online and also that the nodes are all available. I right clicked on Role “JBAG12” and clicked on “Start Role”. it came online without issues,

-> Availability group came online and everything was working fine,

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

-> I was trying to add a database onto existing Always On Availability group. The database that I wanted to add was grayed out,

-> I was able to see the status for that database as “Does not meet prerequisites”. I got below message when I clicked on it,

TITLE: Microsoft SQL Server Management Studio

Auto close is enabled. Only databases with auto close disabled can be added to an availability group. To turn off auto close, set the Auto Close database property to False.

This database lacks a full database backup. Before you can add this database to an availability group, you must perform a full database backup.

-> I checked the database properties and was able to see that “Auto close” option was enabled for the database.

-> Auto close option can cause performance degradation on frequently accessed databases because of the increased overhead of opening and closing the database after each connection. AUTO_CLOSE also flushes the procedure cache after each connection.

-> Due to above reason we decided to disable Auto close option.

-> You can use below query if you dont want the GUI method. I just scripted below query from GUI “Script” option.

USE [master]
GO
ALTER DATABASE [JBSDB] SET AUTO_CLOSE OFF WITH NO_WAIT
GO

-> Once the database JBSDB auto close option was disabled, I was able to add the database onto Availability group without any issues,

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

-> I tried creating an Always On Listener on an existing 4 node Always on Availability Group with below details,

-> It failed with below error,

<sub><sup>TITLE: Microsoft SQL Server Management Studio
Create failed for Availability Group Listener ‘JBApplication’. (Microsoft.SqlServer.Smo)
For help, click: https://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=16.100.46521.71+(SMO-master-A)&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Create+AvailabilityGroupListener&LinkId=20476

ADDITIONAL INFORMATION:
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)

The Windows Server Failover Clustering (WSFC) resource control API returned error code 5057. If this is a WSFC availability group, the WSFC service may not be running or may not be accessible in its current state, or the specified arguments are invalid. Otherwise, contact your primary support provider. For information about this error code, see “System Error Codes” in the Windows Development documentation.
The attempt to create the network name and IP address for the listener failed. If this is a WSFC availability group, the WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate the network name and IP address with the network administrator. Otherwise, contact your primary support provider.
The attempt to create the network name and IP address for the listener failed. If this is a WSFC availability group, the WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate the network name and IP address with the network administrator. Otherwise, contact your primary support provider. (Microsoft SQL Server, Error: 41009)

For help, click: https://docs.microsoft.com/sql/relational-databases/errors-events/mssqlserver-41009-database-engine-error</sup></sub>

-> As per the error message, I checked what is Error code 5057 in “System Error Codes” by executing below command in Command prompt.

NET HELPMSG 5057

-> Error code 5057 refers to “The cluster IP address is already in use.”

-> This means that the IP address is already taken by some other host name.

-> Checking cluster.log for more details. Please use this article to generate cluster.log.

-> Below entries found in cluster.log which is same as what we got from Helpmsg,

_{^{000015f4.00000544::2022/05/30-11:58:35.110 ERR [RES] IP Address : IpaValidatePrivateResProperties: IP address 172.20.1.20 was detected on the network.
000015f4.00000544::2022/05/30-11:58:35.110 WARN [RHS] Resource JBAG1234_172.20.1.20 called SetResourceStatus, but OnlineOrOfflineCall call is NULL: Offline, 1.
000015f4.00000544::2022/05/30-11:58:35.110 WARN [RHS] returning ResourceExitStateTerminate.
000015f4.00000544::2022/05/30-11:58:35.110 WARN [RHS] Error 5057 from ResourceControl 20971654 for resource JBAG1234_172.20.1.20.}}

-> Lets find out the hostname that is using this IP Address by executing below command,

ping -a 172.20.1.20

-> I see that hostname JBSAPP.JBSWIKI.com is utilizing this IPAddress. I advised the same to my platforms team and got a new IP Address. I was able to create the listener after getting the new IP Address.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.