-> Application team advised that their application is not able to connect to the database that is part of Always on availability group.
-> Always on Availability group was in resolving state as shown below,
-> Always on Availability group roles were in pending state on the Cluster administrator,
-> Below were the messages found on event viewer,
Event ID: 41144 The local availability replica of availability group ‘JBSAG’ is in a failed state. The replica failed to read or update the persisted configuration data (SQL Server error: 41005). To recover from this failure, either restart the local Windows Server Failover Clustering (WSFC) service or restart the local instance of SQL Server.
Event ID: 1205 The Cluster service failed to bring clustered role ‘JBSAG’ completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered role. Event ID: 1069 Cluster resource ‘JBSAG’ of type ‘SQL Server Availability Group’ in clustered role ‘JBSAG’ failed. Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.
Event ID: 7043 The Cluster Service service did not shut down properly after receiving a preshutdown control.
-> Below errors were there on SQL Server error log,
Error: 41022, Severity: 16, State: 0. Failed to create a Windows Server Failover Clustering (WSFC) notification port with notification filter 778567686 and notification key 3 (Error code 5073). If this is a WSFC availability group, the WSFC service may not be running or may not be accessible in its current state, or the specified arguments are invalid. Otherwise, contact your primary support provider. For information about this error code, see “System Error Codes” in the Windows Development documentation. Always On: The availability replica manager is going offline because the local Windows Server Failover Clustering (WSFC) node has lost quorum. This is an informational message only. No user action is required. Always On: The local replica of availability group ‘JBSAG’ is stopping. This is an informational message only. No user action is required.
Error: 41066, Severity: 16, State: 0. Cannot bring the Windows Server Failover Clustering (WSFC) resource (ID ‘ee50bbc1-93ab-4f25-85e5-a7d245555183’) online (Error code 126). If this is a WSFC availability group, the WSFC service may not be running or may not be accessible in its current state, or the WSFC resource may not be in a state that could accept the request. Otherwise, contact your primary support provider. For information about this error code, see “System Error Codes” in the Windows Development documentation.
Error: 41160, Severity: 16, State: 0. Failed to designate the local availability replica of availability group ‘JBSAG’ as the primary replica. The operation encountered SQL Server error 41066 and has been terminated. Check the preceding error and the SQL Server error log for more details about the error and corrective actions. Error: 41017, Severity: 16, State: 1. Failed to add a node to the possible owner list of a Windows Server Failover Clustering (WSFC) resource (Error code 5908). If this is a WSFC availability group, the WSFC service may not be running or may not be accessible in its current state, or the specified cluster resource or node handle is invalid. Otherwise, contact your primary support provider.
-> Cluster.log did not have much details other than the AG group failing.
-> All messages or errors were all generic. We could not get much clue.
-> We executed Process monitor and found that DBA team have renamed C:\windows\system32\hadrres.dll to hadrres_old.dll due to a patching error on both JBSAG1 and JBSAG2. They had to rename it as patching was failing with an error that hadrres.dll is used by another process. DBA team forgot to rename it back to hadrres.dll and this caused the issues. We renamed the file back to hadrres.dll and that solved the issue.
Watch this video to grasp a real-time understanding of this matter.
Thank You, Vivek Janakiraman
Disclaimer: The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.
In this blog post, we will discuss an issue encountered with an Always On Availability Group where a database remained stuck in the Initializing / In Recovery state on the secondary replica after a failover. We will explore the steps taken to resolve the problem and provide insights into the underlying causes.
Issue
During a failover from the primary to the secondary replica in an Always On Availability Group, one particular database entered the Initializing / In Recovery state on the secondary replica. Despite restarting SQL services and attempting to suspend data movement, the database remained stuck in this state.
Resolution
Logging off users and restarting SQL services: Initially, all users were logged off from the secondary replica, and the SQL services on the replica were restarted. However, these steps did not resolve the issue.
Suspending data movement: The next approach involved suspending the data movement of the problematic database from the SQL Server Management Studio on the secondary replica. However, this operation resulted in a “lock request time out period exceeded” error.
Identifying the blocking process: To investigate further, a query was executed to identify any blocking processes using the following query: SELECT * FROM sysprocesses WHERE blocked > 0 It was discovered that the DB Startup thread for the specific database was blocking the SPID attempting to suspend data movement.
Checking recovery status: The SQL Server error log was examined to determine if the database’s recovery had completed. The log indicated that recovery for the database had finished in 43 seconds, with no user action required. Recovery completed for database JB_DB (database ID 7) in 43 second(s) (analysis 64 ms, redo 324 ms, undo 0 ms.) This is an informational message only. No user action is required.
Analyzing DB Startup thread wait type: Monitoring the DB Startup thread revealed that it was in a wait state with the HADR_RECOVERY_WAIT_FOR_UNDO type. This wait type occurs when the secondary database is waiting for the reverting and initializing phases to synchronize with the primary database after failover.
Monitoring undo progress: The “Log remaining for undo” performance monitor counter was used to track the amount of log in kilobytes remaining for the undo phase. Surprisingly, the counter showed 0, indicating that there was nothing to undo, and the recovery completed message in the SQL Server error log confirmed that the undo took 0 milliseconds.
Failing over to another synchronous secondary replica: To further troubleshoot the issue, a failover was performed from the primary replica to another synchronous secondary replica. This time, all databases returned to a synchronized state without any issues.
Summary
The issue of a database stuck in the Initializing / In Recovery state after an Always On Availability Group failover was resolved by failing over to another synchronous secondary replica. Although the root cause and exact timing of the issue remain unknown, this blog post aims to provide guidance and assistance to others who may encounter similar challenges within the community.
Thank You, Vivek Janakiraman
Disclaimer: The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.
SQL Server 2022 has arrived with a myriad of exciting performance enhancements that are set to revolutionize database management. In this article, we will explore the various aspects of these enhancements, their significance in the realm of database management, and the benefits they offer to businesses and organizations. From query optimizer improvements to in-memory enhancements, storage enhancements, security upgrades, scalability improvements, and integration with cloud technologies, we will dive into the details of each feature and understand how they contribute to elevating the performance of SQL Server.
Overview of SQL Server 2022
To begin our journey, let’s take a moment to understand SQL Server 2022. It is the latest iteration of Microsoft’s flagship relational database management system, known for its robustness, reliability, and scalability. SQL Server 2022 builds upon the strengths of its predecessors and introduces an array of performance-enhancing features that address common challenges faced in database management.
Importance of Performance Enhancements in Database Management
Performance is a crucial aspect of any database management system. Slow queries, inefficient data retrieval, and inadequate scalability can hinder the overall productivity and efficiency of an organization. Recognizing this, SQL Server 2022 has placed a strong emphasis on performance enhancements, aiming to deliver lightning-fast query execution, improved optimization, and enhanced data processing capabilities. These enhancements enable businesses to extract insights from their data more efficiently, leading to better decision-making and ultimately, a competitive edge.
Understanding SQL Server Performance
Before delving into the specifics of SQL Server 2022’s performance enhancements, it is important to familiarize ourselves with the key metrics used to evaluate database performance. Metrics such as response time, throughput, and resource utilization provide valuable insights into the efficiency and effectiveness of the database system. Additionally, we will explore common performance challenges encountered in SQL Server, including suboptimal query plans, index fragmentation, and contention issues, which the enhancements in SQL Server 2022 aim to address.
Introducing SQL Server 2022 Performance Enhancements
Now let’s take a closer look at the performance enhancements introduced in SQL Server 2022.
Overview of New Features and Enhancements
SQL Server 2022 introduces an impressive array of new features and enhancements that have been designed to tackle the performance challenges faced by organizations. These advancements span various aspects of the database engine, including the query optimizer, in-memory processing, storage operations, security, scalability, and integration with cloud technologies. By comprehensively addressing these areas, SQL Server 2022 aims to optimize performance and provide a seamless and efficient user experience.
Benefits of Upgrading to SQL Server 2022
Upgrading to SQL Server 2022 brings numerous benefits for businesses. Enhanced query optimization ensures improved query plans and faster execution, resulting in reduced response times and increased overall database performance. In-memory enhancements provide accelerated data access and manipulation, allowing for real-time analytics and quicker decision-making. Storage enhancements, such as resumable index operations and intelligent indexing, minimize downtime and optimize space utilization. Furthermore, the security enhancements bolster data protection and compliance, safeguarding sensitive information from unauthorized access.
Query Optimizer Enhancements
The query optimizer is at the heart of SQL Server’s performance. In SQL Server 2022, the query optimizer has received significant enhancements to improve cardinality estimation, adapt to changing workload patterns through adaptive query processing, and employ intelligent query processing techniques. These advancements result in more accurate query plans, optimal resource allocation, and overall improved query performance.
Improved Cardinality Estimation
Cardinality estimation plays a critical role in determining the optimal query plan. SQL Server 2022 introduces improvements in cardinality estimation algorithms, enabling the query optimizer to make better-informed decisions about data distribution and selectivity. This results in more accurate cardinality estimates, which ultimately lead to improved query plans and better query performance.
Adaptive Query Processing
Workload patterns can vary significantly over time, making it challenging to create a single static query plan that remains optimal under all circumstances. Adaptive query processing, a new feature in SQL Server 2022, allows the database engine to adapt dynamically to changing conditions. It enables the optimizer to adjust query plans based on actual runtime statistics, ensuring optimal performance even in the face of varying data and query patterns.
Intelligent Query Processing
Intelligent query processing takes performance optimization to the next level by employing intelligent algorithms and techniques. Features like batch mode on rowstore, table variable deferred compilation, and memory grant feedback leverage machine learning and intelligent algorithms to optimize query execution and resource allocation. These advancements result in faster query processing, reduced CPU and memory usage, and overall improved performance.
In-Memory Enhancements
In-memory processing has gained significant popularity in recent years due to its ability to deliver lightning-fast data access. SQL Server 2022 further enhances in-memory capabilities with features like accelerated database recovery and memory-optimized Tempdb metadata.
Accelerated Database Recovery
Database recovery after a failure or crash can be a time-consuming process. SQL Server 2022 introduces accelerated database recovery, which drastically reduces the time required for recovery by utilizing a novel log-based recovery mechanism. This enhancement ensures minimal downtime and faster database availability, leading to improved overall system performance and resilience.
Memory-Optimized Tempdb Metadata
Tempdb is a crucial component of SQL Server, used for temporary storage and sorting operations. In SQL Server 2022, memory-optimized Tempdb metadata enables faster metadata operations by leveraging memory-optimized technology. This enhancement improves concurrency, reduces contention, and speeds up Tempdb-related operations, leading to enhanced performance and scalability.
Storage Enhancements
Efficient storage operations are vital for optimal database performance. SQL Server 2022 introduces several storage enhancements that address common challenges, such as resumable index operations, intelligent indexing, and accelerated database recovery for big data clusters.
Resumable Index Operations
Creating or rebuilding indexes on large tables can be a time-consuming process, often leading to significant downtime. With resumable index operations in SQL Server 2022, administrators can pause and resume index operations, allowing for greater flexibility and minimizing the impact on database availability. This enhancement enables organizations to perform index maintenance tasks efficiently without disrupting ongoing operations.
Intelligent Indexing
Creating and maintaining indexes is a critical aspect of database optimization. SQL Server 2022 introduces intelligent indexing, which automates the index management process by analyzing query patterns and recommending optimal indexes. This feature simplifies index management, improves query performance, and reduces manual effort required for index maintenance.
Accelerated Database Recovery for Big Data Clusters
Big data clusters require robust storage and recovery mechanisms to handle large volumes of data. SQL Server 2022 provides accelerated database recovery specifically tailored for big data clusters, significantly reducing the time required for database recovery operations. This enhancement ensures faster data availability and improves the performance of big data workloads.
Security Enhancements
Data security is of paramount importance in today’s digital landscape. SQL Server 2022 introduces security enhancements that further fortify the protection of sensitive data, including features like Always Encrypted with Secure Enclaves and dynamic data masking improvements.
Always Encrypted with Secure Enclaves
Always Encrypted is a feature introduced in previous versions of SQL Server that allows sensitive data to be encrypted at rest and in transit. In SQL Server 2022, Always Encrypted with Secure Enclaves takes data security to a whole new level. By leveraging secure enclaves, SQL Server can perform computations on encrypted data without exposing the plaintext to the outside world. This feature provides an additional layer of security and ensures that sensitive data remains protected throughout its lifecycle.
Dynamic Data Masking Improvements
Dynamic data masking is a feature that allows organizations to mask sensitive data in real-time, ensuring that only authorized individuals can view the actual values. SQL Server 2022 introduces improvements to dynamic data masking, providing more granular control over data masking rules and enhancing the flexibility and effectiveness of data protection mechanisms.
Enhanced Availability and Scalability
High availability and scalability are crucial for mission-critical applications that demand continuous uptime and the ability to handle growing workloads. SQL Server 2022 introduces enhancements in these areas to meet the evolving needs of modern businesses.
High Availability Enhancements
SQL Server 2022 introduces several high availability enhancements, including improved failover capabilities, enhanced resiliency, and simplified management. Features like accelerated database recovery, always-on availability groups, and enhanced automatic failover mechanisms ensure continuous availability of critical databases, minimizing downtime and improving overall system reliability.
Scalability Improvements
As data volumes and user concurrency increase, scalability becomes a key consideration for any database system. SQL Server 2022 incorporates scalability improvements that enable organizations to scale their databases vertically and horizontally. These enhancements, such as improved parallelism, increased resource utilization, and optimized query processing, ensure that SQL Server can handle growing workloads efficiently and deliver consistent performance.
Sharding Improvements
Sharding is a technique used to horizontally partition large databases across multiple servers, enabling distributed data storage and parallel processing. SQL Server 2022 introduces improvements in sharding capabilities, providing enhanced support for sharded databases, simplified management, and improved query performance. These enhancements enable organizations to leverage the power of distributed computing and achieve seamless scalability for their applications.
Integration with Cloud Technologies
With the increasing adoption of cloud computing, SQL Server 2022 offers enhanced integration with cloud technologies to provide organizations with the flexibility and scalability required for modern data platforms.
Hybrid Cloud Capabilities
SQL Server 2022 enables seamless integration between on-premises and cloud environments through its hybrid cloud capabilities. Organizations can leverage features like Azure Arc-enabled data services to manage and operate their SQL Server instances from a centralized location. This integration allows for a unified management experience and the ability to extend on-premises SQL Server deployments to the cloud, providing greater agility and scalability.
SQL Server 2022 and Azure Integration
SQL Server 2022 integrates tightly with Microsoft Azure, unlocking a wide range of cloud-based services and capabilities. Organizations can leverage Azure SQL Database and Azure Synapse Analytics to offload certain workloads, take advantage of cloud-based analytics and machine learning, and benefit from built-in automation and scalability features. This integration empowers organizations to leverage the power of the cloud while maintaining the familiarity and capabilities of SQL Server.
Performance Tuning Best Practices
Optimizing the performance of a SQL Server database requires a combination of best practices, tools, and techniques. SQL Server 2022 introduces various performance tuning features and improvements, which, when combined with established best practices, can deliver outstanding performance.
Analyzing and Optimizing Query Performance
Analyzing and optimizing query performance is a critical task for database administrators. SQL Server 2022 provides advanced tools and techniques for query tuning, including query plan analysis, index optimization, and statistics management. By understanding the query execution plans, identifying bottlenecks, and applying appropriate optimizations, administrators can significantly enhance query performance and overall system efficiency.
Monitoring and Tuning Indexes
Indexes play a vital role in query performance. SQL Server 2022 offers comprehensive indexing capabilities, including tools for index monitoring, analysis, and optimization. By monitoring index usage, identifying fragmented or redundant indexes, and optimizing index design, administrators can improve data retrieval speed, reduce disk I/O, and boost overall database performance.
Utilizing Query Store for Performance Insights
Query Store, introduced in previous versions of SQL Server, is a powerful tool for query performance monitoring and troubleshooting. SQL Server 2022 enhances the capabilities of Query Store, providing administrators with valuable insights into query performance, plan regressions, and historical trends. By utilizing Query Store, administrators can identify and rectify performance issues, track query behavior over time, and ensure optimal performance for their databases.
Conclusion
In conclusion, SQL Server 2022 introduces a wide range of performance enhancements that address common challenges in database management. From query optimizer improvements to in-memory enhancements, storage enhancements, security upgrades, scalability improvements, and integration with cloud technologies, these enhancements pave the way for faster query execution, improved resource utilization, enhanced data access, and superior overall performance.
By upgrading to SQL Server 2022 and leveraging its performance-enhancing features, organizations can unlock new possibilities, optimize their databases, and stay ahead in today’s data-driven landscape. It is crucial for businesses to explore and leverage these benefits to maximize their productivity, gain valuable insights, and maintain a competitive edge in their respective industries.
Thank You, Vivek Janakiraman
Disclaimer: The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.
