Rajnish Noonia

Tag: SQL

  • Validate SSIS package on server

    SSIS server validation

    Untitled

     

    • Any change in database schema will break the associated SSIS package.
    • Generally each change is responsible to identify dependent components to be included in the change scope.
    • But what about any missed dependencies. E.g. Recent production issue.
    • Such broken SSIS can be identified in Pre-Prod only if pre-prod is well controlled and identical copy of prod environment.
    • Process is required to identify packages impacted by dependencies changes like database schema.
    • Proactive validation of SSIS package could help to identify the issue and reduce the system downtime.

    High level architecture

    Untitled2

     

    The solution is capable of validating all ssis packages for sql server version 2008 on-wards. Instead of publishing the full source code below is the pseudo code and sql scripts to validate the packages.

    1. Create C# console base project which takes the server name you want to validate. The app calls the web service to validate the SQL server (validate SSIS packages on that server).
    2. Create web api project and expose following methods (Job Controller)
      1. Validate – start the validation process by creating SQL agent job dynamically on target server
      2. ValidationFinished – this will be invoked by job when validation is finished.
      3. GetValidationStatus – this will provide the SQL agent job status currently validating server

    The Validate method works as follows..

    • Get the SQL Server version
    private Version GetServerVersion(string serverName)
     {
     var connectionString = string.Format("Server={0};Database=master;Trusted_Connection=True;", serverName);
     using (var connection = new SqlConnection(connectionString))
     {
     connection.Open();
     return new Version(connection.ServerVersion);
     }
     }
    
    • Create interface IProcessor and implement it for sql server 2008 and 2012 as they both have different way to validate package
    • Based on server version get the eligible concrete implementation.
    • The validation engine will create the validation job on target sql server. for each proxy account, the script will create job step to trigger a network share package running under proxy account. (for details see createValidationJob.sql)
    • Each validation step gets the package scheduled under the current account and validate the package, populate the results back on central server. and in the end extract the job history and delete the job.)
    • The network deployed ssis package uses same engine to validate 2008 and 2012 via IProcessor since validation is done differently for both servers.

    The common scripts used for both servers are here

    Common-Scripts

    The SQL server 2008 validation is done via DTEXEC with /validate command

    SQl 2008-Scripts

    The SQL 2012 validation is done via EXECUTE SSISDB.catalog.validate_project stored proc.

    SQL 2012 Scripts

    The centralized server which keeps the status of SSIS validation looks like

    Untitled3

    The font end to display the data was build on angular 2 API..

  • Export data from corrupted database

    Below is the sql script to import data from source database into target database, It is assumed that you have both the databases on single server.The source database is current database & few tables are corrupted whereas the traget database is created from old backup for target database.since their is corruption ,it is not possible to take backup of current database. This script imports back data (only) from source to target database. You need to only replace 3 lines of the script (12th line from bottom).
    Download script from here

     
    
    ------------- Create helper functions -----------------------------
    
    IF EXISTS (select * from dbo.sysobjects where id = object_id(N'[dbo].[Mig_ImportTable]') and OBJECTPROPERTY(id, N'IsProcedure') = 1)
    DROP PROCEDURE [dbo].[Mig_ImportTable]
    GO
    
    CREATE PROC dbo.Mig_ImportTable @Database SYSNAME, @Table SYSNAME AS
    	SET NOCOUNT ON
    
    	DECLARE @Column SYSNAME
    	DECLARE @SQL NVARCHAR(4000)
    	DECLARE @colSQL NVARCHAR(4000)
    	DECLARE @IsIdentity BIT
    	DECLARE @IsTableIdentity BIT
    
    	SET @colSQL = ''
    	SET @SQL=''
    	SET @IsTableIdentity = 0 --false
    	SET @IsIdentity = 0 --false
    
    	PRINT 'Table Migration Started for :' + @Table + ' in ' + @Database
    
    	DECLARE curMoveDown CURSOR
    	LOCAL FORWARD_ONLY
    	OPTIMISTIC FOR
    	SELECT column_name,COLUMNPROPERTY(object_id(TABLE_NAME), COLUMN_NAME, 'IsIdentity')as IsIdentity FROM information_schema.columns
    	WHERE UPPER(table_name ) = UPPER(@Table)
    
    	OPEN curMoveDown FETCH NEXT FROM curMoveDown INTO @Column,@IsIdentity
    	WHILE @@FETCH_STATUS = 0
    	BEGIN
    		SET @colSQL = @colSQL + '[' + @Column + '],'
    		SET @IsTableIdentity = @IsTableIdentity | @IsIdentity
    		IF (@IsTableIdentity = 1) BREAK
    		FETCH NEXT FROM curMoveDown INTO @Column,@IsIdentity
    	END
    	CLOSE curMoveDown
    	DEALLOCATE curMoveDown 
    
    	IF(LEN(@colSQL)>1)
    	BEGIN
    		SET @colSQL = LEFT(@colSQL,LEN(@colSQL) - 1)
    
    		IF(@IsTableIdentity = 1) SET @SQL = @SQL + 'SET IDENTITY_INSERT ' + @Table + ' ON '
    		SET @SQL = @SQL + '	ALTER TABLE ' + @Table + ' DISABLE TRIGGER ALL '
    		SET @SQL = @SQL + '	ALTER TABLE ' + @Table + ' NOCHECK CONSTRAINT ALL '
    		SET @SQL = @SQL + '	TRUNCATE TABLE ' + @Table + '  '
    		IF(@IsTableIdentity = 1)
    			SET @SQL = @SQL + '	INSERT INTO ' + @Table + ' (' +  @colSQL + ') SELECT '+  @colSQL + ' FROM ' + '[' + @Database + '].[dbo].[' + @Table + '] '
    		ELSE
    			SET @SQL = @SQL + '	INSERT INTO ' + @Table + ' SELECT * FROM ' + '[' + @Database + '].[dbo].[' + @Table + '] '
    		IF(@IsTableIdentity = 1) SET @SQL = @SQL + '	SET IDENTITY_INSERT ' + @Table + ' OFF '
    		SET @SQL = @SQL + '	ALTER TABLE ' + @Table + ' ENABLE TRIGGER ALL '
    		SET @SQL = @SQL + '	ALTER TABLE ' + @Table + ' CHECK CONSTRAINT ALL '
    
    		EXEC sp_executesql @SQL
    		PRINT 'Table migrated :' + @Table
    	END
    	ELSE
    		PRINT 'No Column found :' + @Table + ' (SQL = ' + @colSQL + ')'
    
    	PRINT 'Table Migration finished for :' + @Table
    GO
    
    IF EXISTS (select * from dbo.sysobjects where id = object_id(N'[dbo].[Mig_ImportDatabase]') and OBJECTPROPERTY(id, N'IsProcedure') = 1)
    DROP PROCEDURE [dbo].[Mig_ImportDatabase]
    GO
    
    CREATE PROC dbo.Mig_ImportDatabase @Source SYSNAME AS	
    
    	DECLARE @SERVER SYSNAME
    	DECLARE @INSTANCE SYSNAME
    	DECLARE @FULLNAME SYSNAME
    	DECLARE @DBNAME SYSNAME
    	DECLARE @cmd varchar(1000)
    	DECLARE @SQL Nvarchar(1000)
    	DECLARE @Table Nvarchar(100)
    	DECLARE @Result INT
    
    	SET NOCOUNT ON
    
    	CREATE TABLE [#Mig_FailedTables] ( [name] SYSNAME NOT NULL ) ON [PRIMARY]
    
    	SELECT @SERVER = CONVERT(SYSNAME, SERVERPROPERTY('servername'))
    	SELECT @INSTANCE = IsNull('',CONVERT(SYSNAME, SERVERPROPERTY('InstanceName')))
    	SELECT @DBNAME = DB_NAME()
    
    	IF(Len(@INSTANCE)>0) SET @FULLNAME = @SERVER + '' + @INSTANCE
    	ELSE SET @FULLNAME = @SERVER
    
    	Print 'Migration started from database [' + @SOURCE + '] to database [' + @DBNAME + '] on server  ' + @FULLNAME
    
    	--Cursor to loop throw all tables of Target database (current database)
    	DECLARE CurTables CURSOR LOCAL FORWARD_ONLY
    	OPTIMISTIC FOR
    	SELECT NAME from dbo.sysobjects where OBJECTPROPERTY(id, N'IsUserTable') = 1 order by Name
    
    	OPEN CurTables FETCH NEXT FROM CurTables INTO @Table
    	WHILE @@FETCH_STATUS = 0
    	BEGIN
    		Print 'Migrating table ' + @Table
    		SET @cmd = 'ECHO Exec [Mig_ImportTable] ''' + @Source + ''',''' + @Table + '''  > DbMig.sql'
    		EXEC @Result = master..xp_cmdshell @cmd, no_output
    
    		SET @cmd = 'ECHO GO  >> DbMig.sql'
    		EXEC @Result =  master..xp_cmdshell @cmd, no_output
    
    		SET @cmd = 'ECHO @ECHO OFF  > DbMig.cmd'
    		EXEC @Result =  master..xp_cmdshell @cmd, no_output
    
    		SET @cmd = 'ECHO osql -E -b -S "' + @FULLNAME +'" -d "' + @DBNAME +'" -i "DbMig.sql" >> DbMig.cmd'
    		EXEC @Result =  master..xp_cmdshell @cmd, no_output
    
    		SET @cmd = 'ECHO EXIT ERRORLEVEL >> DbMig.cmd'
    		EXEC @Result =  master..xp_cmdshell @cmd, no_output
    
    		SET @cmd = 'CMD /c "DbMig.cmd>>DbMig.Log"'
    		EXEC @Result = master..xp_cmdshell @cmd, no_output
    		IF (@Result <> 0)
    			Insert into #Mig_FailedTables (Name) values (@Table)
    		FETCH NEXT FROM CurTables INTO @Table
    
    	END
    	CLOSE CurTables
    	DEALLOCATE CurTables
    	Print 'Migration Finished..'
    	SELECT Name as [Failed Tables] FROM #Mig_FailedTables
    	DROP TABLE #Mig_FailedTables
    
    GO
    
    ------------- Migration of data starts from here ------------------
    
    ----------------------------------------------------------------------------------------------------------------------------------
    
    USE TargetDatabase						-- Target Database  	** Change This
    Exec Mig_ImportDatabase 'SourceDatabase'			-- Source Database  	** Change This - Import Complete database
    EXEC Mig_ImportTable 'SourceDatabase','SpecificTableName'	-- SpecificTableName  	** Change This - Import Single Table
    ----------------------------------------------------------------------------------------------------------------------------------
  • Compare DB – record counts

    Code snippet will loop through all tables of Database SOURCEDB and compare record counts with tables in TARGETDB database

    USE SOURCEDB

    EXEC SP_MSforeachtable ‘DECLARE @OriCount INT
         DECLARE @Count INT
         DECLARE @Name VARCHAR(400)
         SET @Count = (SELECT COUNT(*) FROM TARGETDB.?)
         SET @OriCount = (SELECT COUNT(*) FROM SOURCEDB.?)
         SET @Name=”?”

    IF(@Count <> @OriCount)
    BEGIN
             SELECT @Name,@Count as target,@OriCount as source

    END‘

  • Recovering a SQL Server Database from Suspect Mode

    USE Master
    GO

    EXEC sp_configure ‘allow updates’, 1
    RECONFIGURE WITH OVERRIDE
    GO

    BEGIN TRAN
    UPDATE master..sysdatabases SET status = status | 32768 WHERE name = ‘YourDBName’
    IF @@ROWCOUNT = 1
     BEGIN COMMIT TRAN
      RAISERROR(‘Emergency Mode Successfully Set’, 0, 1)
     END
    ELSE
     BEGIN ROLLBACK
     RAISERROR(‘Setting Emergency Mode Failed’, 16, 1)
    END
    GO
    –Stop mssql service

    –Rename the existing LOG file for YourDBName database.

    –Start mssql service

    DBCC REBUILD_LOG(YourDBName,’C:program files….dataYourDBName_log.ldf’)
    GO

    DBCC checkdb (YourDBName)
    GO

    –If DBCC return errors then fix it
    — BEGIN
     ALTER DATABASE YourDBName SET SINGLE_USER
     GO
     –Repair the consistency errors if found by above command
     DBCC checkdb (YourDBName,REPAIR_ALLOW_DATA_LOSS)
     GO
     –OR
     DBCC checkdb (YourDBName,REPAIR_FAST)
     GO
    — END Fix errors finished

    ALTER DATABASE YourDBName SET MULTI_USER

    Article applies to SQL Server 2000