Configuring Transactional replication using Initialize from LSN

Environment

-> Below queries will be used to create the required database and objects on Publisher JBRESEARCH\IN2019,

create database JBDB
GO
use [JBDB]
GO
create table Table1(
SNO INT PRIMARY KEY ,
SNAME VARCHAR(255))
GO
create table Table2(
SNO INT PRIMARY KEY ,
SNAME VARCHAR(255))
GO
create table Table3(
SNO INT PRIMARY KEY ,
SNAME VARCHAR(255))
GO
insert into Table1 values (1,'A')
GO
insert into Table2 values (1,'A')
GO
insert into Table3 values (1,'A')
GO

-> Configure Distribution using below queries on Publisher JBRESEARCH\IN2019,

use master
exec sp_adddistributor @distributor = N'JBResearch\IN2019', @password = N'Pa$$W0rd'
GO
exec sp_adddistributiondb @database = N'distribution'
, @data_folder = N'C:\Program Files\Microsoft SQL Server\MSSQL15.IN2019\MSSQL\Data'
, @log_folder = N'C:\Program Files\Microsoft SQL Server\MSSQL15.IN2019\MSSQL\Data', @log_file_size = 2, @min_distretention = 0, @max_distretention = 72
, @history_retention = 48, @deletebatchsize_xact = 5000, @deletebatchsize_cmd = 2000, @security_mode = 1
GO
exec sp_adddistpublisher @publisher = N'JBResearch\IN2019', @distribution_db = N'distribution', @security_mode = 1
, @working_directory = N'\\JBRESEARCH\Backup', @trusted = N'false', @thirdparty_flag = 0, @publisher_type = N'MSSQLSERVER'
GO

-> Configure Publication using below query on Publisher JBRESEARCH\IN2019,

use [JBDB]
exec sp_replicationdboption @dbname = N'JBDB', @optname = N'publish', @value = N'true'
GO

use [JBDB]
EXEC sp_addpublication
@publication = N'JBDBREPL',
@status = N'active',
@allow_push = N'true',
@allow_pull = N'true',
@independent_agent = N'true',
@immediate_sync = N'true',
@allow_initialize_from_backup = N'true'
GO

EXEC sp_addarticle @publication = N'JBDBREPL',@article =N'Table1',@source_object =N'Table1'
GO
EXEC sp_addarticle @publication = N'JBDBREPL',@article =N'Table2',@source_object =N'Table2'
GO
EXEC sp_addarticle @publication = N'JBDBREPL',@article =N'Table3',@source_object =N'Table3'
GO

-> Checking the available data on objects Table1, Table2 and Table3 on Publisher JBRESEARCH\IN2019.

select * from JBDB..Table1
GO
select * from JBDB..Table2
GO
select * from JBDB..Table3

-> Backup JBDB database using below query on Publisher JBRESEARCH\IN2019,

use master
backup database JBDB to disk = 'c:\temp\JBDB_14OCT2020.bak' with STATS=1

-> Let us add some new data using below query after the database backup on Publisher JBRESEARCH\IN2019. We are adding new data to check if replication picks up these data too after replication configuration. In real world scenario there is high possibility that new data will be added when we are performing backup on the publisher and then restore it in subscriber.

use [JBDB]
GO
insert into Table1 values (2,'B')
GO
insert into Table2 values (2,'B')
GO
insert into Table3 values (2,'B')
GO
insert into Table1 values (3,'C')
GO
insert into Table2 values (3,'C')
GO
insert into Table3 values (3,'C')
GO

-> Checking the available data on objects Table1, Table2 and Table3 on Publisher JBRESEARCH\IN2019.

select * from JBDB..Table1
GO
select * from JBDB..Table2
GO
select * from JBDB..Table3
GO

-> Restore database JBDBSub on the Subscriber SQL Server Instance JBRESEARCH\IN2019_1 using the backup that was performed on Publisher server JBRESEARCH\IN2019.

USE [master]
RESTORE DATABASE [JBDBSub] FROM DISK = N'C:\Temp\JBDB_14OCT2020.bak' WITH FILE = 1
, MOVE N'JBDB' TO N'C:\Program Files\Microsoft SQL Server\MSSQL15.IN2019_1\MSSQL\DATA\JBDB.mdf'
, MOVE N'JBDB_log' TO N'C:\Program Files\Microsoft SQL Server\MSSQL15.IN2019_1\MSSQL\DATA\JBDB_log.ldf', NOUNLOAD, STATS = 1
GO

-> Execute below query to get LastLSN details from the database backup that was performed on Publisher server JBRESEARCH\IN2019.

declare @Backupfile nvarchar(1000)
set @Backupfile = 'c:\temp\JBDB_14OCT2020.bak'
declare @sql nvarchar(1000)
declare @LASTLSN numeric(25,0)
create table #BackupLSNDetails
(
BackupName nvarchar(128),
BackupDescription nvarchar(255),
BackupType smallint,
ExpirationDate datetime,
Compressed bit,
Position smallint,
DeviceType tinyint,
UserName nvarchar(128),
ServerName nvarchar(128),
DatabaseName nvarchar(128),
DatabaseVersion int,
DatabaseCreationDate datetime,
BackupSize numeric(20, 0),
FirstLSN numeric(25, 0),
LastLSN numeric(25, 0),
CheckpointLSN numeric(25, 0),
DatabaseBackupLSN numeric(25, 0),
BackupStartDate datetime,
BackupFinishDate datetime,
SortOrder smallint,
[CodePage] smallint,
UnicodeLocaleId int,
UnicodeComparisonStyle int,
CompatibilityLevel tinyint,
SoftwareVendorId int,
SoftwareVersionMajor int,
SoftwareVersionMinor int,
SoftwareVersionBuild int,
MachineName nvarchar(128),
Flags int,
BindingId uniqueidentifier,
RecoveryForkId uniqueidentifier,
Collation nvarchar(128),
FamilyGUID uniqueidentifier,
HasBulkLoggedData bit,
IsSnapshot bit,
IsReadOnly bit,
IsSingleUser bit,
HasBackupChecksums bit,
IsDamaged bit,
BeginsLogChain bit,
HasIncompleteMetaData bit,
IsForceOffline bit,
IsCopyOnly bit,
FirstRecoveryForkID uniqueidentifier,
ForkPointLSN numeric(25, 0),
RecoveryModel nvarchar(60),
DifferentialBaseLSN numeric(25, 0),
DifferentialBaseGUID uniqueidentifier,
BackupTypeDescription nvarchar(60),
BackupSetGUID uniqueidentifier,
CompressedBackupSize bigint,
Containment tinyint,
KeyAlgorithm nvarchar(32),
EncryptorThumbprint varbinary(20),
EncryptorType nvarchar(32)
)
set @sql= 'RESTORE HEADERONLY FROM DISK = '''+@Backupfile+''''
insert into #BackupLSNDetails
exec (@sql)
select @LASTLSN = LastLSN from #BackupLSNDetails
DROP table #BackupLSNDetails
declare @numericlsn numeric(25,0)
declare @high4bytelsncomponent bigint,@mid4bytelsncomponent bigint,
@low2bytelsncomponent int
--set the lsn here
set @numericlsn = @LASTLSN
select @high4bytelsncomponent = convert(bigint, floor(@numericlsn / 1000000000000000))
select @numericlsn = @numericlsn - convert(numeric(25,0), @high4bytelsncomponent) * 1000000000000000
select @mid4bytelsncomponent = convert(bigint,floor(@numericlsn / 100000))
select @numericlsn = @numericlsn - convert(numeric(25,0), @mid4bytelsncomponent) * 100000
select @low2bytelsncomponent = convert(int, @numericlsn)
SELECT convert(binary(4), @high4bytelsncomponent) + convert(binary(4), @mid4bytelsncomponent) + convert(binary(2), @low2bytelsncomponent) as 'LastLSN_To_Be_Used_In_@subscriptionlsn'

-> I see it to be 0x00000025000009300001 in my case.

-> Create the push subscription on Publisher server JBRESEARCH\IN2019 using below query.

USE [JBDB]
GO
EXEC sp_addsubscription
@publication = N'JBDBREPL',
@subscriber = N'JBResearch\IN2019_1',
@destination_db = N'JBDBSub',
@sync_type= N'initialize from lsn',
@backupdevicetype='DISK',
@backupdevicename=N'C:\Temp\JBDB_14OCT2020.bak',
@subscription_type = N'push',
@subscriptionlsn =0x00000025000009300001,
@update_mode = N'read only'
GO

exec sp_addpushsubscription_agent
@publication = N'JBDBREPL',
@subscriber = N'JBResearch\IN2019_1',
@subscriber_db = N'JBDBSub',
@subscriber_security_mode = 1,
@dts_package_location = N'Distributor'
GO

-> Verify replication monitor to see there are no failures. Query the involved objects on either side to check if data’s are in sync,

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

Replicated transactions are waiting for next Log backup or for mirroring partner to catch up

Environment

-> We had an environment which is same as above environment. Remote distributor, Publisher and subscriber servers were configured in Always ON availability group.

-> I worked on a disaster recovery drill. As part of the DR, All database server on primary datacentre were brought down. Below is the representation after the primary datacentre was brought down.

-> I failed over all database servers from Primary Datacentre to Secondary Datacentre.

-> Failover completed fine. Below is the view in SQL Server management studio,

-> I checked the Replication monitor and found below message in Log reader agent,

Replicated transactions are waiting for next Log backup or for mirroring partner to catch up.
Initializing
Validating publisher
Starting agent.

-> Message “Replicated transactions are waiting for next Log backup or for mirroring partner to catch up.” is as a result of Always ON availability secondary replica being down.

-> With the current setup transactional replication log reader agent will not be able to to move forward as the asynchronous secondary replicas have not acknowledged the reception of a change.

-> One option to solve this issue is to wait for the Always ON secondary replica to come online and that should fix the issue.

-> But in my case Always ON secondary replica will be online after 12 hours. So waiting for the Always ON secondary replica to come online is not an option.

-> Second option will be enable trace flag 1448. Please check article1 and article2 for more details on this trace flag.

-> Trace flag 1448 enables the replication log reader to move forward even if the asynchronous secondary replicas have not acknowledged the reception of a change. Even with this trace flag enabled, the log reader always waits for the synchronous secondary replicas. The log reader will not go beyond the min ack of the synchronous secondary replicas. This trace flag applies to the instance of SQL Server, not just to an availability group, an availability database, or a log reader instance. This trace flag takes effect immediately without a restart. It can be activated ahead of time or when an asynchronous secondary replica fails.

-> I enabled trace flag 1448 on the Publisher server JBPRIMARY-DR using below query,

dbcc traceon(1448,-1)

-> Once the trace flag 1448 is enabled. Log reader agent completed fine.

-> Please note that this trace flag can be enabled without a SQL Server restart.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

Agent message code 21021. The subscription does not exist.

Environment

-> I worked on a disaster recovery drill. As part of the DR, I failed over all database servers from Primary Datacentre to Secondary Datacentre.

-> We had an environment which is same as above environment. Remote distributor, Publisher and subscriber servers were configured in Always ON availability group.

-> After the failover, distribution agent failed with below error,

Error messages:
Agent 'JBPUB-PRIMARY-JBREPL-TranProducts-JBREPLSUB-1' is retrying after an error. 8 retries attempted. See agent job history in the Jobs folder for more details.

-> I then checked the distribution agent job in SQL Server Agent and found below error,

Date		9/4/2020 11:41:07 AM
Log		Job History (JBPUB-PRIMARY-JBREPL-TranProducts-JBREPLSUB-1)

Step ID		2
Server		JBDIST-DR
Job Name	JBPUB-PRIMARY-JBREPL-TranProducts-JBREPLSUB-1
Step Name		Run agent.
Duration		00:11:51
Sql Severity	0
Sql Message ID	0
Operator Emailed	
Operator Net sent	
Operator Paged	
Retries Attempted	0

Message
2020-09-04 11:52:54.399 Copyright (c) 2016 Microsoft Corporation
2020-09-04 11:52:54.399 Microsoft SQL Server Replication Agent: distrib
2020-09-04 11:52:54.399 
2020-09-04 11:52:54.399 The timestamps prepended to the output lines are expressed in terms of UTC time.
2020-09-04 11:52:54.399 User-specified agent parameter values:
			-Subscriber JBREPLSUB
			-SubscriberDB JBREPL_SUB
			-Publisher JBPUB-PRIMARY
			-Distributor JBDistributor
			-DistributorSecurityMode 1
			-Publication TranProducts
			-PublisherDB JBREPL
			-Continuous
			-XJOBID 0x3B3FD20E052A694491BD0660C526FA21
			-XJOBNAME JBPUB-PRIMARY-JBREPL-TranProducts-JBREPLSUB-1
			-XSTEPID 2
			-XSUBSYSTEM Distribution
			-XSERVER JBDIST-DR
			-XCMDLINE 0
			-XCancelEventHandle 0000000000001BA8
			-XParentProcessHandle 0000000000000358
2020-09-04 11:52:54.399 Startup Delay: 4172 (msecs)
2020-09-04 11:52:58.581 Connecting to Distributor 'JBDistributor'
2020-09-04 11:52:58.612 Parameter values obtained from agent profile:
			-bcpbatchsize 2147473647
			-commitbatchsize 100
			-commitbatchthreshold 1000
			-historyverboselevel 1
			-keepalivemessageinterval 300
			-logintimeout 15
			-maxbcpthreads 1
			-maxdeliveredtransactions 0
			-pollinginterval 5000
			-querytimeout 1800
			-skiperrors 
			-transactionsperhistory 100
2020-09-04 11:52:58.612 Connecting to Subscriber 'JBREPLSUB'
2020-09-04 11:52:58.659 Initializing
2020-09-04 11:52:58.706 Agent message code 21021. The subscription does not exist.

-> From the replication monitor, Right Click Distribution agent and click on “Agent Profiles”,

-> It seems like there is a custom distribution agent profile created and used in Database Server in Primary Datacentre (JBDIST-PRIMARY) that is not available in Database Server at Secondary Datacentre (JBDIST-DR).

-> Now that we failed over from Primary Datacentre to Secondary Datacentre, this missing “Agent Profile” is causing issue.

-> Execute below query on distribution database in Database Server JBDIST-PRIMARY which is currently a secondary replica after the failover and check what agent profile was used by the distribution agent previously,

select a.id, a.name, a.publisher_database_id,a.publisher_db, a.subscriber_db,
b.profile_id , b.profile_name from MSdistribution_agents a 
INNER JOIN msdb..MSagent_profiles b on a.profile_id = b.profile_id

-> In my case the the “Agent profile” missing is JBS-Load.

-> Now I will create this missing distribution agent profile in replication monitor. Right click Distribution agent and click on “Agent profile” and Click “New”.

-> I could find what were the parameters used as part of Agent profile “JBS-Load” and I created it as below,

-> Stop and start the distribution agent and it should work fine.

-> You can select “Default agent profile” if you are not aware of the custom profile created. But please understand it may not have the setting same as custom profile, which means there are possibilities of issues.

-> It is always a best practice to compare the agent profiles in replication monitor as part of DR planning if you dont want to run into issues like this.

Thank You,
Vivek Janakiraman

Disclaimer:
The views expressed on this blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.