Ralf Eisenreich

There are two options executing SSIS packages:

- xp_cmdshell command (not recommended for security reasons)
- sp_start_job command

The main difference between both options is the execution method. The xp_cmdshell command is a synchronous call and the sp_start_job command is an asynchronous call.

xp_cmdshell command (synchronous)

• enable xp_cmdshell mode in Surface Area Configuration Tool for SQL Server
• use following procedure to execute SSIS package:
  
DECLARE @returncode int
EXEC @returncode = xp_cmdshell 'dtexec /f "PackageNameWithFullPath.dtsx"'


sp_start_job (asynchronous)

• define job in SQL Server Agent with SSIS execution step
• use following procedure to start SQL Server Agent job with SSIS execution step:
  
[msdb].dbo.sp_start_job @job_name='JobName'

• check execution status of job, since the call is asynchronous:
  
SELECT
[server],
[start_execution_date],
[stop_execution_date],
[run_date],
[run_duration],
[run_status],
[message]
FROM
MSDB.DBO.SYSJOBS Z
INNER JOIN
MSDB.DBO.SYSJOBACTIVITY A
ON Z.JOB_ID = A.JOB_ID
INNER JOIN
(
SELECT
MAX(SESSION_ID) AS SESSION_ID
FROM
MSDB.DBO.SYSSESSIONS
) AS B
ON A.SESSION_ID = B.SESSION_ID
LEFT JOIN
MSDB.DBO.SYSJOBHISTORY C
ON A.JOB_HISTORY_ID = C.INSTANCE_ID
WHERE
Z.NAME = 'JobName'


There is following way to make the call of sp_start_job synchronous.

sp_start_job (synchronous)

• define job in SQL Server Agent with SSIS execution step
• use following procedure to start SQL Server Agent job with SSIS execution step:
  
ALTER PROCEDURE [dbo].[AGENT_JOB_CHECK2]
-- Add the parameters for the stored procedure here
DECLARE @job_name nvarchar(100)='',
DECLARE @maxwaitmins int = 5
AS
BEGIN
set NOCOUNT ON;
set XACT_ABORT ON;
BEGIN TRY
declare @running as int
declare @seccount as int
declare @maxseccount as int
set @maxseccount = 60*@maxwaitmins
set @seccount = 0
set @running = 0
declare @job_owner sysname
declare @job_id UNIQUEIDENTIFIER
set @job_owner = SUSER_SNAME()
-- get job id
select @job_id=job_id
from msdb.dbo.sysjobs sj
where sj.name=@job_name
-- invalid job name then exit with an error
if @job_id is null
RAISERROR (N'Unknown job: %s.', 16, 1, @job_name)
-- output from stored procedure xp_sqlagent_enum_jobs is captured in the following table
declare @xp_results TABLE ( job_id UNIQUEIDENTIFIER NOT NULL,
last_run_date INT NOT NULL,
last_run_time INT NOT NULL,
next_run_date INT NOT NULL,
next_run_time INT NOT NULL,
next_run_schedule_id INT NOT NULL,
requested_to_run INT NOT NULL, -- BOOL
request_source INT NOT NULL,
request_source_id sysname COLLATE database_default NULL,
running INT NOT NULL, -- BOOL
current_step INT NOT NULL,
current_retry_attempt INT NOT NULL,
job_state INT NOT NULL)
-- start the job
declare @r as int
exec @r = msdb..sp_start_job @job_name
-- quit if unable to start
if @r<>0
RAISERROR (N'Could not start job: %s.', 16, 2, @job_name)
-- start with an initial delay to allow the job to appear in the job list (maybe I am missing something ?)
WAITFOR DELAY '0:0:10';
set @seccount = 10
-- check job run state
insert into @xp_results
execute master.dbo.xp_sqlagent_enum_jobs 1, @job_owner, @job_id
set @running= (SELECT top 1 running from @xp_results)
while @running<>0 and @seccount < @maxseccount
begin
WAITFOR DELAY '0:0:10';
set @seccount = @seccount + 10
delete from @xp_results
insert into @xp_results
execute master.dbo.xp_sqlagent_enum_jobs 1, @job_owner, @job_id
set @running= (SELECT top 1 running from @xp_results)
end
-- result: query
SELECT
[server],
[start_execution_date],
[stop_execution_date],
[run_date],
[run_duration],
[run_status], -- 0: failed, 1: success, null: running
[message]
FROM
MSDB.DBO.SYSJOBS Z
INNER JOIN
MSDB.DBO.SYSJOBACTIVITY A
ON Z.JOB_ID = A.JOB_ID
INNER JOIN
(
SELECT
MAX(SESSION_ID) AS SESSION_ID
FROM
MSDB.DBO.SYSSESSIONS
) AS B
ON A.SESSION_ID = B.SESSION_ID
LEFT JOIN
MSDB.DBO.SYSJOBHISTORY C
ON A.JOB_HISTORY_ID = C.INSTANCE_ID
WHERE Z.NAME = @job_name
-- result: not ok (=1) if still running
--if @running <> 0
--return 0
--else
--return 1
END TRY
BEGIN CATCH
DECLARE
@ErrorMessage NVARCHAR(4000),
@ErrorNumber INT,
@ErrorSeverity INT,
@ErrorState INT,
@ErrorLine INT,
@ErrorProcedure NVARCHAR(200);
SELECT
@ErrorNumber = ERROR_NUMBER(),
@ErrorSeverity = ERROR_SEVERITY(),
@ErrorState = ERROR_STATE(),
@ErrorLine = ERROR_LINE(),
@ErrorProcedure = ISNULL(ERROR_PROCEDURE(), '-');
SELECT @ErrorMessage =
N'Error %d, Level %d, State %d, Procedure %s, Line %d, ' +
'Message: '+ ERROR_MESSAGE();
RAISERROR
(
@ErrorMessage,
@ErrorSeverity,
1,
@ErrorNumber, -- original error number.
@ErrorSeverity, -- original error severity.
@ErrorState, -- original error state.
@ErrorProcedure, -- original error procedure name.
@ErrorLine -- original error line number.
);
END CATCH
END


[Source: http://blog.boxedbits.com/archives/124]

If you experience following error “Executed as user: MyDomain\SQLServer. The package execution failed. The step failed.”
then check that your Service User for the SQL Server Agent has full file permissions to all packages, config files and log files!
Moreover this service user should be member of the SQL Server DTS Group.

Importing Date values e.g. from Flat Files it can happen that the format of the date is different to the Host System date formatting of the regional settings. Especial the date setting for English (UK: day/month/year) and English (US: month/day/year) are often very problematic.

In SSIS a solution for this issue can be realized with a ScriptComponent as follows.

1. Date column
   First of all you should import your Date column as nvarchar type. In this example let us assume we are going to import a Birthday column in Germany format on a host system with US-based regional setting.
2. Script Component: Columns
   We specify as import column the BIRTHDAY_STR column (nvarchar) from the Flat File and as export column we specify a column of type date named BIRTHDAY_DATE.
3. ScriptComponent: Code
   In the Script we are using the TryParse method which is stable and flexible enough. But in order to import the German date we have to create a culture object.

   
Dim dt As DateTime
Dim culture As Globalization.CultureInfo
culture = Globalization.CultureInfo.CreateSpecificCulture("de-DE")
Try
If DateTime.TryParse(CStr(Row.BIRTHDAY_STR), culture, Globalization.DateTimeStyles.None, dt) Then
Row.BIRTHDAY_DATE = dt
Else
Row.BIRTHDAY_DATE_IsNull = True
End If
Catch ex As Exception
Row.BIRTHDAY_DATE_IsNull = True
End Try


This example should help you to import the dates of different regional settings.

Here I would like to show you a solution for following problem.
One source row coming from Excel contains several values within one column. Each one of these values should be reflected in a separate row in the data destination.
So the basic problem is that there we have a “list column” whose contents should be result in separate rows.

To make the problem more complex: The number of values in the “list column” is variable and there is a second column which contains corresponding values.

Excel Source

Excel Sheet

solution: SSIS Script Component

To implement a solution for the problem we use the most adequate item – the Script Component.

1. We start with a data flow task in our package and add an OLE DB Excel Source component (that returns the source data above).
   We can use following SQL statement:
   
SELECT
F1,F2,F3
FROM
[sheet1$B1:D5]

   The imported columns can be named as KEY, PERSON, DEPARTMENT.
   
   

   Excel Source

2. Then we add a Derived Column Component to the data flow as well.
   
   

   Derived Column

   In order to replace the CRs with commas we replace the imported columns with following formula:
   
REPLACE(REPLACE(REPLACE(REPLACE(PERSON,"\n",",")," ",""),",,",","),",,",",")
REPLACE(REPLACE(REPLACE(REPLACE(DEPARTMENT,"\n",",")," ",""),",,",","),",,",",")

   

   

   Derived Column Settings

3. As result we receive following list of elements:
   • – -
   • row 1
   • PERSON: Mr. Wayne,Mr. Kent,Mr. Parker
   • DEPARTMENT: Heros/Batman,Heros/Superman,Heros/Spiderman
   • – -
   • row 2
   • PERSON: Mr. Kirk,Ms. Janeway
   • DEPARTMENT: StarTrek/Classic,StartTrek/Voyager
   • – -
   • row 3
   • PERSON: “Mr. Wrong”
   • DEPARTMENT: Wrong entry 1,Wrong entry 2
   • – -
4. Now we add a Script Component to the data flow.
   When we are prompted to select the type of the component we choose Transformation.
   

   Script Component

5. Then we right-click on the Script Component and choose Edit. In the Script Transformation Editor dialog box we select all three columns to make them available as input columns. Now these column are available in the Script code.
   
   

   Script Component Columns

6. Since the tranformation is going to produce more output rows than it receives input rows we have to mark the transformation as asynchronous. This is done by setting the SynchronousInputID property of the output to None.
   
   

   Script Component Settings

7. Next, we have to specify the output columns that the output buffer will contain. So we select the transformation’s output and click Add Column and set the properties (column name and data type).
   
   

   Script Component Input / Output rows

8. Now we can add the code by clicking on the Design Script button.
   
   

   Script Component - Design Script

9. Following Code is splitting our list of items into rows:
   
' Variables
Dim i As Integer = 0
Dim itemList_PERSON As String = Row.PERSON
Dim itemList_DEPARTMENT As String = Row.DEPARTMENT
Dim delimiter As String = ','
If Not (String.IsNullOrEmpty(itemList_PERSON) Or String.IsNullOrEmpty(itemList_DEPARTMENT)) Then
Dim inputListArray_Wer() As String = itemList_PERSON.Split(New String() {delimiter}, StringSplitOptions.RemoveEmptyEntries)
Dim inputListArray_Abt() As String = itemList_DEPARTMENT.Split(New String() {delimiter}, StringSplitOptions.RemoveEmptyEntries)
If (inputListArray_Wer.Length = inputListArray_Abt.Length) Then
' new rows
For Each item As String In inputListArray_Wer
With Output0Buffer
.AddRow()
.KEY = CInt(Row.KEY)
.PERSON = item
.DEPARTMENT = inputListArray_Abt(i).ToString()
End With
' increase counter
i = i + 1
Next
Else
' Filtered Rows
With Output1Buffer
.AddRow()
.KEY = CInt(Row.KEY)
.PERSON = Row.PERSON
.DEPARTMENT = Row.DEPARTMENT
End With
End If
End If

   The code shows that there are two outputs. Whenever a record set cannot be split or the columns PERSON and DEPARTMENT have a different number of itemss the rows is redirected to the second output (here Output1).
10. In order to receive the output from the Script Component we add a Union All Component. Additionally we add two Data Viewer Components, since we want to see the resulting outputs from the Script Component.
    
    

    Union ALL

11. Running the SSIS package will bring the following result.
    
    

    Data Flow completed

12. As we can see, the first DataViewer Component show us the valid rows produced by our Script Component.
    
    

    Data Viewer valid rows

13. The second DataViewer Component show us the invalid rows produced by our Script Component.
    
    

    Data Viewer invalid rows

download
Of course I provide you the solution as download.

When Excel data is not coming into SSIS right, then do the following:

1. use option “IMEX=1” in ConnectionString (Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\..\MyExcel.xls;Extended Properties="Excel 8.0;HDR=Yes;IMEX=1")
2. modify registry setting “TypeGuessRows=8” to a much higher value
3. use a sql statement to retrieve data (e.g. “SELECT * FROM [sheet1$A5:M2000]“)

NOTE: The setting IMEX=1 tells the Excel driver to use Import mode. This mode reads the registry setting “ImportMixedTypes=Text” which forces mixed data to be converted to text. To achieve a more reliable column type recognition, the registry setting “TypeGuessRows=8” should be increased. By default the ISAM driver analyzes  the first eight rows and determines the column datatypes from this sampling. If the first analysed rows contain only numeric values, then setting IMEX=1 will not convert this column datatype to Text – it will remain numeric.

ATTENTION: The IMEX setting has some other modes:

• 0 is Export mode
• 1 is Import mode
• 2 is Linked mode (full update capabilities)

Registry Setting Location:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Jet\X.X\Engines\Excel


supplement
If you experience following error the above described solution could fix it.
1. An OLE DB error has occurred. Error code: 0x80040E21.
2. Failed to retrieve long data for column “XYZ”.
3. There was an error with output column “XYZ” (55488) on output “Excel Source Output” (109). The column status returned was: “DBSTATUS_UNAVAILABLE”.
4. The “output column “XYZ” (55488)” failed because error code 0xC0209071 occurred, and the error row disposition on “output column “XYZ” (55488)” specifies failure on error. An error occurred on the specified object of the specified component.
5. [DTS.Pipeline] Error: The PrimeOutput method on component “Info Source” (101) returned error code 0xC0209029. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.
6. [DTS.Pipeline] Error: Thread “SourceThread0″ has exited with error code 0xC0047038.

In order to start SSIS packages from ASP.NET web applications you can use a Web Service.

Following page shows how to implement this: Running packages programmatically on the server by using a Web service or remote component.
What you should know about running SSIS packages from ASP.NET is that there is a problem with the threads SSIS processes use.

Even Impersonation does not work successfully. The impersonation applies to the calling thread only, but SSIS creates additional threads in order to be able to perform multiple steps simultanously. Unfortunately, the impersonation context is not passed to these additional threads, so any data base access occurs under process user context, not under impersonated context.

So it is better to execute SSIS package outside of ASP.NET process, e.g. using DTEXEC or Agent Job (Agent proxies are convinient if you want to execute under specific credentials).

Important:
With its default settings for authentication and authorization, a Web service generally does not have sufficient permissions to access SQL Server or the file system to load and execute packages. You may have to assign appropriate permissions to the Web service by configuring its authentication and authorization settings in the web.config file and assigning database and file system permissions as appropriate.

Solution:

IIS 5.0: only solution seems to be assigning Administrator rights to the ASP.NET User

IIS 6.0: create new Application Pool with Administrator user context and set the Web Service
using this context in order to initiate SSIS packages

Here is an example Application Configuration for a SSIS & ASP.NET project:

Following error messages can show that you’re running into the problem with the SSIS threads:
- package validation failed
- TaskHost validation failed
- cannot access file location
- component xyz validation failed

Falls aus Excel Daten importiert werden die sowohl textuell als auch numerisch sein können (pro Spalte), dann kann es passieren, dass numerische Werte als NULL-Values dargestellt werden.

Lösung:
Einfach bei Extended Connection Properties (des Excel Connection Managers) “IMEX=1″ (für import-Mode) anfügen.

[Quelle: http://support.microsoft.com/default.aspx?scid=kb;EN-US;194124]

Falls ein Fehler beim Importieren von SSIS-Paketen auftritt und der SQL Server nicht unter der Default-Instanz installiert ist, liegt das mit hoher Sicherheit an einem Konfigurationsfehler.

In der Datei steht standardmäßig der Eintrag ., welcher in SERVER\instance geändert werden muss.

Dann findet der Integration Service auch die richtige SQL-Instanz zum speichern von Paketen.

Die Datei befindet sich unter C:\Program Files\Microsoft SQL Server\90\DTS\Binn.

error 1053 integration services

Falls ein Fehlstart der SSIS nach der Installation des Service Pack 1 für den SQL-Server auftritt, dann kann es daran liegen, dass der direkte Internetzugang geblockt wird.

Entweder Firewall konfigurieren oder dem System die aktuellen Proxy-Einstellungen mitteilen.

Den Proxy konfiguriert man dann in der Eingabeaufforderung mit dem Tool proxycfg.exe.

[Quelle: MS Knowledge]

Um ein SSIS-Paket, dass im SSIS-Storage des SQL-Servers (nicht im FileSystem) abgelegt ist aus einer .NET-Anwendung zu starten kann folgender Code verwendet werden:

(more…)

Die SQL Server Integration Services erlauben das Zusammenfuehren von Daten aus heterogenen Datenquellen. Um den Ueberblick zu behalten wird der gesamte Datenfluss grafisch dargestellt und modelliert.

Das finde ich klasse, denn gerade wenn man ein Projekt an eine weitere Person uebergeben muss oder selbst an mehreren Projekten arbeitet, behaelt man durch diese kleine Hilfe den Ueberblick besser.

An manchen Stellen wird es trotzdem notwendig sein, Programmierfaehigkeiten mitzubringen. Dazu kann dann die Scriptkomponente benutzt werden. Mit folgendem Script kann man auf die Datenquellen in seinem Projekt zugreifen. Aber Achtung es muss eine OLEDB-Verbindung definiert werden!


' initialize connection
Me.myConnection = DirectCast(Dts.Connections("SERVER.Database.root.ADO").AcquireConnection(Dts.Transaction), SqlClient.SqlConnection)
' -- open connection to database --
Try
If Not (myConnection.State.Open = ConnectionState.Open) Then
myConnection.Open()
End If
Catch ex As Exception
MsgBox("Could not open connection to DataBase: " & ex.Message.ToString)
End Try
' -- define data tables --
Dim tblTabelle As New Data.DataTable
' -- define SQL-commands --
Dim cmdSQLTblTabelle As New SqlClient.SqlCommand
cmdSQLTblTabelle.CommandText = "SELECT * FROM tblTabelle;"
cmdSQLTblTabelle.Connection() = myConnection
' -- define DataAdapters --
Dim myDATblTabelle As SqlClient.SqlDataAdapter(cmdSQLTblTabelle)
' -- get data from tables --
myDATblTabelle.FillSchema(tblTabelle, SchemaType.Mapped)
myDATblTabelle.Fill(tblTabelle)
' generate InsertCommand automatically with Commandbuilder
Dim myCommandBuilder As New SqlClient.SqlCommandBuilder(myDATblTabelle)
' show rows of query
msgbox("rows: " & tblTabelle.Rows.Count)