Scenario: Download Script
In year 2013, I wrote a post Read Excel File After Skipping Few Rows by using OLE DB Source. Today we are going to use Script Task to perform the same task, How to skip Few Rows in Excel sheet and load rest of the records to SQL Server Table.
In this post, we will loop through multiple Excel files. Each file can have single or multiple Sheets. Also we don't care about sheet name. It can be anything.
Just keep in mind, the sheets on each file should be according to our criteria. I have few sample files. We are going to use variables to make our SSIS Package more dynamic. If our requirement change, we simple need to change the values of variables and package should work just fine.
Excel File TechBrothersIT.com has two sheets ( same meta data or column information)
Excel File TechBrothersIT.com_1 has only single sheet with same meta data as Excel Sheets from Excel file TechBrothersIt.com
Solution:
Take a detail look into your Excel Sheets. Notice that we don't need to load the first 7 Rows. We need to skip those rows and the data we have is only start from Column A and Ends at column C.
We need to use this information in our variables. If tomorrow we will get more columns, we can change our Start Column and End Column values. Also if rows start up or down, we can simple change the value for StartingReadingRow value for variable.
Let's create the table in which we are going to load the data. I am going to create Customer table with id, name and dob columns.
CREATETABLE dbo.Customer (
id INT
,name VARCHAR(100)
,dob DATE
)
GO
Step 1: Create variables to make your SSIS Package dynamic
Create an SSIS Package in SSDT ( SQL Server Data Tools). Create below variables in SSIS Package.
FolderPath : Provide the Folder path where your excel files will be dropped or placed.
SchemaName : Schema Name of the table in which you would like to load the data
TableName: TableName in which you would like to load the data
StartReadingFromRow : Provide the Row number from which you would like to read. in my case I would like to read from row 8 where I have header row.
StartColumn: Provide the Starting Column from Which you would like to Read. In my case it is A
EndColumn : Provide the End Column, In my case it is C.
Create Variables in SSIS Package - Skip Rows in Excel and Load Data Dynamically to SQL Server Tables
Step 2: Create ADO.NET Connection in SSIS Package to use in Script Task
Create ADO.NET Connection Manager so we can use in Script Task to Load data from Excel Sheets to SQL Server Table after skipping few rows by using Script Task in SSIS Package.
Create ADO.NET Connection in SSIS Package to use in Script Task to Load Excel File after skipping few rows
Step3: Add Variables to Script Task to use from SSIS Package
Bring the Script Task on Control Flow Pane in SSIS Package and open by double clicking Check-box in front of variable to add to Script Task.
How to Skip Rows in Excel Sheet and Load to SQL Server Table by using Script Task in SSIS Package
Step 4: Add Script to Script task Editor in SSIS To load Excel Data after skipping Rows in Excel Sheet
Click Edit Button and it will open Script Task Editor.
Under #region Namespaces, I have added below code
using System.IO;
using System.Data.OleDb;
using System.Data.SqlClient;
Under public void Main() {
I have added below code.
string FolderPath=Dts.Variables["User::FolderPath"].Value.ToString();
string TableName = Dts.Variables["User::TableName"].Value.ToString();
string SchemaName = Dts.Variables["User::SchemaName"].Value.ToString();
string StartingColumn = Dts.Variables["User::StartingColumn"].Value.ToString();
string EndingColumn = Dts.Variables["User::EndingColumn"].Value.ToString();
string StartReadingFromRow = Dts.Variables["User::StartReadingFromRow"].Value.ToString();
var directory = new DirectoryInfo(FolderPath);
FileInfo[] files = directory.GetFiles();
//Declare and initilize variables
string fileFullPath = "";
//Get one Book(Excel file at a time)
foreach (FileInfo file in files)
{
fileFullPath = FolderPath+"\\"+file.Name;
//Create Excel Connection
string ConStr;
string HDR;
HDR="YES";
ConStr = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + fileFullPath + ";Extended Properties=\"Excel 12.0;HDR=" + HDR + ";IMEX=0\"";
OleDbConnection cnn = new OleDbConnection(ConStr);
//MessageBox.Show(TableName);
//Get Sheet Name
cnn.Open();
DataTable dtSheet = cnn.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
string sheetname;
sheetname="";
foreach (DataRow drSheet in dtSheet.Rows)
{
if (drSheet["TABLE_NAME"].ToString().Contains("$"))
{
sheetname=drSheet["TABLE_NAME"].ToString();
//Load the DataTable with Sheet Data so we can get the column header
OleDbCommand oconn = new OleDbCommand("select top 1 * from [" + sheetname + StartingColumn + StartReadingFromRow + ":" + EndingColumn + "]", cnn);
OleDbDataAdapter adp = new OleDbDataAdapter(oconn);
DataTable dt = new DataTable();
adp.Fill(dt);
cnn.Close();
//Prepare Header columns list so we can run against Database to get matching columns for a table.
string ExcelHeaderColumn = "";
string SQLQueryToGetMatchingColumn = "";
for (int i = 0; i < dt.Columns.Count; i++)
{
if (i != dt.Columns.Count - 1)
ExcelHeaderColumn += "'" + dt.Columns[i].ColumnName + "'" + ",";
else
ExcelHeaderColumn += "'" + dt.Columns[i].ColumnName + "'";
}
SQLQueryToGetMatchingColumn = "select STUFF((Select ',['+Column_Name+']' from Information_schema.Columns where Table_Name='" +
TableName + "' and Table_SChema='" + SchemaName + "'" +
"and Column_Name in (" + @ExcelHeaderColumn + ") for xml path('')),1,1,'') AS ColumnList";
// MessageBox.Show(SQLQueryToGetMatchingColumn);
//MessageBox.Show(ExcelHeaderColumn);
//USE ADO.NET Connection
SqlConnection myADONETConnection = new SqlConnection();
myADONETConnection = (SqlConnection)(Dts.Connections["DBConn"].AcquireConnection(Dts.Transaction) as SqlConnection);
//Get Matching Column List from SQL Server
string SQLColumnList = "";
SqlCommand cmd = myADONETConnection.CreateCommand();
cmd.CommandText = SQLQueryToGetMatchingColumn;
SQLColumnList = (string)cmd.ExecuteScalar();
//MessageBox.Show(" Matching Columns: " + SQLColumnList);
//Use Actual Matching Columns to get data from Excel Sheet
OleDbConnection cnn1 = new OleDbConnection(ConStr);
cnn1.Open();
OleDbCommand oconn1 = new OleDbCommand("select " + SQLColumnList + " from [" + sheetname + StartingColumn + StartReadingFromRow + ":" + EndingColumn + "]", cnn1);
OleDbDataAdapter adp1 = new OleDbDataAdapter(oconn1);
DataTable dt1 = new DataTable();
adp1.Fill(dt1);
cnn1.Close();
//Load Data from DataTable to SQL Server Table.
using (SqlBulkCopy BC = new SqlBulkCopy(myADONETConnection))
{
BC.DestinationTableName = SchemaName + "." + TableName;
foreach (var column in dt1.Columns)
BC.ColumnMappings.Add(column.ToString(), column.ToString());
BC.WriteToServer(dt1);
}
}
}
}
Step 5: Save Script in Script Task Editor and Run SSIS Package to Skip Rows and Load Data
Save the Script in Script Task Editor, close the editor and run your SSIS Pakcage. It should skip the rows we have provided as variable values and load the data for columns given.
I executed for sample files and here are my results. It skipped first 7 rows and read the data from Column A to C( id,name and dob columns) and loaded to dbo.Customer table.
Skip Rows in Excel Sheet and Load the Data to SQL Server Table in SSIS Package by using Script Task C# script language
Things we learnt in this post
How to load Data from Excel Files to SQL Server Table by using C#
How to create variables in SSIS Package and use them in Script Task to make package more dynamic
How to Skip Rows in Excel Sheet and Load Data to SQL server Table in SSIS Packge by using Script Task
How to read Excel File Name in SSIS Package
How to Check matching Header Column Names to SQL Server Table by using C sharp
How to save scalar string value from SQL Server Query to Variable in C Sharp
How to use BulkCopy to load data from DataTable to SQL Server Table
How to Map Columns Dynamically in BulkCopy C Sharp